Jul 07 18:58:33.522591 ip-10-0-142-134 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Jul 07 18:58:33.522604 ip-10-0-142-134 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Jul 07 18:58:33.522611 ip-10-0-142-134 systemd[1]: kubelet.service: Failed with result 'resources'. Jul 07 18:58:33.522903 ip-10-0-142-134 systemd[1]: Failed to start Kubernetes Kubelet. Jul 07 18:58:43.577479 ip-10-0-142-134 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Jul 07 18:58:43.577495 ip-10-0-142-134 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 3698c91b246e468fb77cf800a002afcb -- Jul 07 19:00:15.547653 ip-10-0-142-134 systemd[1]: Starting Kubernetes Kubelet... Jul 07 19:00:15.995294 ip-10-0-142-134 kubenswrapper[2563]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 07 19:00:15.995294 ip-10-0-142-134 kubenswrapper[2563]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 07 19:00:15.995294 ip-10-0-142-134 kubenswrapper[2563]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 07 19:00:15.995294 ip-10-0-142-134 kubenswrapper[2563]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 07 19:00:15.995294 ip-10-0-142-134 kubenswrapper[2563]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 07 19:00:15.997463 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:15.997378 2563 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 07 19:00:16.003283 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.003263 2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 07 19:00:16.003338 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.003286 2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 07 19:00:16.003338 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.003293 2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 07 19:00:16.003338 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.003297 2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 07 19:00:16.003338 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.003302 2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 07 19:00:16.003338 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.003306 2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 07 19:00:16.003338 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.003311 2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 07 19:00:16.003338 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.003315 2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 07 19:00:16.003338 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.003319 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 07 19:00:16.003338 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.003323 2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 07 19:00:16.003338 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.003334 2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 07 19:00:16.003338 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.003338 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 07 19:00:16.003338 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.003342 2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 07 19:00:16.003740 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.003346 2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 07 19:00:16.003740 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.003363 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 07 19:00:16.003740 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.003368 2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 07 19:00:16.003740 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.003371 2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 07 19:00:16.003740 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.003375 2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 07 19:00:16.003740 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.003379 2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 07 19:00:16.003740 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.003383 2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 07 19:00:16.003740 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.003387 2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 07 19:00:16.003740 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.003390 2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 07 19:00:16.003740 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.003394 2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 07 19:00:16.003740 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.003398 2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 07 19:00:16.003740 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.003402 2563 feature_gate.go:328] unrecognized feature gate: Example Jul 07 19:00:16.003740 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.003405 2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 07 19:00:16.003740 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.003409 2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 07 19:00:16.003740 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.003413 2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 07 19:00:16.003740 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.003417 2563 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 07 19:00:16.003740 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.003421 2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 07 19:00:16.003740 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.003424 2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 07 19:00:16.003740 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.003430 2563 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 07 19:00:16.004179 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.003433 2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 07 19:00:16.004179 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.003437 2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 07 19:00:16.004179 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.003441 2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 07 19:00:16.004179 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.003445 2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 07 19:00:16.004179 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.003449 2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 07 19:00:16.004179 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.003453 2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 07 19:00:16.004179 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.003466 2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 07 19:00:16.004179 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.003470 2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 07 19:00:16.004179 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.003474 2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 07 19:00:16.004179 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.003478 2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 07 19:00:16.004179 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.003484 2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 07 19:00:16.004179 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.003490 2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 07 19:00:16.004179 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.003494 2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 07 19:00:16.004179 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.003498 2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 07 19:00:16.004179 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.003505 2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 07 19:00:16.004179 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.003509 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 07 19:00:16.004179 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.003513 2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 07 19:00:16.004179 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.003517 2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 07 19:00:16.004179 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.003522 2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 07 19:00:16.004179 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.003526 2563 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 07 19:00:16.004688 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.003529 2563 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 07 19:00:16.004688 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.003533 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 07 19:00:16.004688 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.003537 2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 07 19:00:16.004688 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.003541 2563 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 07 19:00:16.004688 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.003545 2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 07 19:00:16.004688 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.003549 2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 07 19:00:16.004688 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.003552 2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 07 19:00:16.004688 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.003556 2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 07 19:00:16.004688 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.003559 2563 feature_gate.go:328] unrecognized feature gate: Example2 Jul 07 19:00:16.004688 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.003563 2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 07 19:00:16.004688 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.003566 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 07 19:00:16.004688 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.003570 2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 07 19:00:16.004688 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.003575 2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 07 19:00:16.004688 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.003578 2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 07 19:00:16.004688 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.003582 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 07 19:00:16.004688 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.003586 2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 07 19:00:16.004688 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.003589 2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 07 19:00:16.004688 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.003593 2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 07 19:00:16.004688 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.003597 2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 07 19:00:16.004688 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.003600 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 07 19:00:16.005155 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.003604 2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 07 19:00:16.005155 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.003609 2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 07 19:00:16.005155 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.003612 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 07 19:00:16.005155 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.003616 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 07 19:00:16.005155 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.003620 2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 07 19:00:16.005155 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.003623 2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 07 19:00:16.005155 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.003628 2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 07 19:00:16.005155 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.003632 2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 07 19:00:16.005155 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.003636 2563 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 07 19:00:16.005155 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.003639 2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 07 19:00:16.005155 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.003643 2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 07 19:00:16.005155 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.003646 2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 07 19:00:16.005155 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.003650 2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 07 19:00:16.005155 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.003654 2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 07 19:00:16.005155 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.003657 2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 07 19:00:16.005155 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.003661 2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 07 19:00:16.005155 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.003665 2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 07 19:00:16.005155 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.003668 2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 07 19:00:16.005155 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.003672 2563 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 07 19:00:16.005155 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.003675 2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 07 19:00:16.005649 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.003681 2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 07 19:00:16.005649 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.003686 2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 07 19:00:16.005649 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.003690 2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 07 19:00:16.005649 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.003694 2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 07 19:00:16.005649 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.003698 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 07 19:00:16.005649 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.003702 2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 07 19:00:16.005649 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.003706 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 07 19:00:16.005649 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.004765 2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 07 19:00:16.005649 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.004774 2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 07 19:00:16.005649 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.004780 2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 07 19:00:16.005649 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.004784 2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 07 19:00:16.005649 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.004788 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 07 19:00:16.005649 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.004792 2563 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 07 19:00:16.005649 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.004797 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 07 19:00:16.005649 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.004801 2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 07 19:00:16.005649 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.004804 2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 07 19:00:16.005649 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.004808 2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 07 19:00:16.005649 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.004813 2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 07 19:00:16.005649 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.004817 2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 07 19:00:16.005649 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.004821 2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 07 19:00:16.006127 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.004825 2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 07 19:00:16.006127 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.004829 2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 07 19:00:16.006127 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.004833 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 07 19:00:16.006127 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.004837 2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 07 19:00:16.006127 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.004841 2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 07 19:00:16.006127 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.004845 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 07 19:00:16.006127 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.004849 2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 07 19:00:16.006127 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.004852 2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 07 19:00:16.006127 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.004856 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 07 19:00:16.006127 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.004860 2563 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 07 19:00:16.006127 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.004864 2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 07 19:00:16.006127 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.004867 2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 07 19:00:16.006127 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.004874 2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 07 19:00:16.006127 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.004879 2563 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 07 19:00:16.006127 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.004884 2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 07 19:00:16.006127 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.004890 2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 07 19:00:16.006127 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.004894 2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 07 19:00:16.006127 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.004898 2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 07 19:00:16.006127 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.004903 2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 07 19:00:16.006721 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.004907 2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 07 19:00:16.006721 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.004911 2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 07 19:00:16.006721 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.004914 2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 07 19:00:16.006721 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.004918 2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 07 19:00:16.006721 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.004921 2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 07 19:00:16.006721 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.004925 2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 07 19:00:16.006721 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.004930 2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 07 19:00:16.006721 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.004934 2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 07 19:00:16.006721 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.004938 2563 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 07 19:00:16.006721 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.004942 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 07 19:00:16.006721 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.004946 2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 07 19:00:16.006721 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.004949 2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 07 19:00:16.006721 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.004953 2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 07 19:00:16.006721 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.004957 2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 07 19:00:16.006721 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.004961 2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 07 19:00:16.006721 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.004965 2563 feature_gate.go:328] unrecognized feature gate: Example2 Jul 07 19:00:16.006721 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.004969 2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 07 19:00:16.006721 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.004974 2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 07 19:00:16.006721 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.004978 2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 07 19:00:16.006721 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.004981 2563 feature_gate.go:328] unrecognized feature gate: Example Jul 07 19:00:16.007397 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.004984 2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 07 19:00:16.007397 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.004988 2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 07 19:00:16.007397 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.004992 2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 07 19:00:16.007397 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.004996 2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 07 19:00:16.007397 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.005000 2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 07 19:00:16.007397 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.005003 2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 07 19:00:16.007397 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.005007 2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 07 19:00:16.007397 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.005012 2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 07 19:00:16.007397 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.005015 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 07 19:00:16.007397 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.005019 2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 07 19:00:16.007397 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.005023 2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 07 19:00:16.007397 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.005026 2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 07 19:00:16.007397 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.005031 2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 07 19:00:16.007397 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.005035 2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 07 19:00:16.007397 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.005039 2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 07 19:00:16.007397 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.005043 2563 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 07 19:00:16.007397 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.005046 2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 07 19:00:16.007397 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.005050 2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 07 19:00:16.007397 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.005055 2563 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 07 19:00:16.007881 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.005060 2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 07 19:00:16.007881 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.005065 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 07 19:00:16.007881 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.005069 2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 07 19:00:16.007881 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.005074 2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 07 19:00:16.007881 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.005079 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 07 19:00:16.007881 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.005102 2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 07 19:00:16.007881 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.005107 2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 07 19:00:16.007881 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.005111 2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 07 19:00:16.007881 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.005115 2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 07 19:00:16.007881 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.005120 2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 07 19:00:16.007881 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.005124 2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 07 19:00:16.007881 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.005128 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 07 19:00:16.007881 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.005131 2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 07 19:00:16.007881 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.005136 2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 07 19:00:16.007881 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.005140 2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 07 19:00:16.007881 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.005144 2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 07 19:00:16.007881 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.005147 2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 07 19:00:16.007881 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.005151 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 07 19:00:16.007881 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.005155 2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 07 19:00:16.008372 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.005159 2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 07 19:00:16.008372 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.005163 2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 07 19:00:16.008372 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.005167 2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 07 19:00:16.008372 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.005171 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 07 19:00:16.008372 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.005174 2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 07 19:00:16.008372 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.005178 2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 07 19:00:16.008372 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.005182 2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 07 19:00:16.008372 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.005186 2563 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 07 19:00:16.008372 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.005190 2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 07 19:00:16.008372 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.005755 2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 07 19:00:16.008372 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.005762 2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 07 19:00:16.008372 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.005766 2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 07 19:00:16.008372 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.005771 2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 07 19:00:16.008372 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.005775 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 07 19:00:16.008372 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.005779 2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 07 19:00:16.008372 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.005782 2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 07 19:00:16.008372 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.005787 2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 07 19:00:16.008372 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.005792 2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 07 19:00:16.008372 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.005796 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 07 19:00:16.008372 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.005800 2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 07 19:00:16.008894 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.005804 2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 07 19:00:16.008894 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.005808 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 07 19:00:16.008894 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.005812 2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 07 19:00:16.008894 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.005816 2563 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 07 19:00:16.008894 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.005820 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 07 19:00:16.008894 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.005824 2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 07 19:00:16.008894 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.005827 2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 07 19:00:16.008894 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.005831 2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 07 19:00:16.008894 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.005835 2563 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 07 19:00:16.008894 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.005839 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 07 19:00:16.008894 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.005845 2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 07 19:00:16.008894 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.005850 2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 07 19:00:16.008894 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.005855 2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 07 19:00:16.008894 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.005858 2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 07 19:00:16.008894 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.005862 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 07 19:00:16.008894 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.005866 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 07 19:00:16.008894 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.005869 2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 07 19:00:16.008894 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.005873 2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 07 19:00:16.008894 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.005877 2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 07 19:00:16.008894 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.005881 2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 07 19:00:16.009420 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.005885 2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 07 19:00:16.009420 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.005889 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 07 19:00:16.009420 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.005893 2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 07 19:00:16.009420 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.005896 2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 07 19:00:16.009420 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.005901 2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 07 19:00:16.009420 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.005904 2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 07 19:00:16.009420 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.005908 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 07 19:00:16.009420 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.005912 2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 07 19:00:16.009420 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.005916 2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 07 19:00:16.009420 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.005920 2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 07 19:00:16.009420 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.005923 2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 07 19:00:16.009420 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.005927 2563 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 07 19:00:16.009420 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.005931 2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 07 19:00:16.009420 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.005936 2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 07 19:00:16.009420 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.005940 2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 07 19:00:16.009420 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.005943 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 07 19:00:16.009420 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.005947 2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 07 19:00:16.009420 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.005951 2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 07 19:00:16.009420 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.005955 2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 07 19:00:16.009901 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.005961 2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 07 19:00:16.009901 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.005966 2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 07 19:00:16.009901 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.005970 2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 07 19:00:16.009901 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.005975 2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 07 19:00:16.009901 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.005979 2563 feature_gate.go:328] unrecognized feature gate: Example Jul 07 19:00:16.009901 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.005984 2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 07 19:00:16.009901 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.005988 2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 07 19:00:16.009901 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.005991 2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 07 19:00:16.009901 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.005995 2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 07 19:00:16.009901 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.005999 2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 07 19:00:16.009901 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.006004 2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 07 19:00:16.009901 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.006008 2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 07 19:00:16.009901 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.006011 2563 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 07 19:00:16.009901 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.006016 2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 07 19:00:16.009901 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.006020 2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 07 19:00:16.009901 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.006024 2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 07 19:00:16.009901 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.006028 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 07 19:00:16.009901 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.006033 2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 07 19:00:16.009901 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.006037 2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 07 19:00:16.010364 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.006040 2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 07 19:00:16.010364 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.006044 2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 07 19:00:16.010364 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.006048 2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 07 19:00:16.010364 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.006052 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 07 19:00:16.010364 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.006056 2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 07 19:00:16.010364 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.006059 2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 07 19:00:16.010364 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.006063 2563 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 07 19:00:16.010364 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.006067 2563 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 07 19:00:16.010364 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.006070 2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 07 19:00:16.010364 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.006074 2563 feature_gate.go:328] unrecognized feature gate: Example2 Jul 07 19:00:16.010364 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.006078 2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 07 19:00:16.010364 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.006084 2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 07 19:00:16.010364 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.006089 2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 07 19:00:16.010364 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.006093 2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 07 19:00:16.010364 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.006096 2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 07 19:00:16.010364 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.006100 2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 07 19:00:16.010364 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.006104 2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 07 19:00:16.010364 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.006107 2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 07 19:00:16.010364 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.006111 2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 07 19:00:16.010364 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.006115 2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 07 19:00:16.010858 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.006119 2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 07 19:00:16.010858 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.006122 2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 07 19:00:16.010858 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.006126 2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 07 19:00:16.010858 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.006130 2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 07 19:00:16.010858 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.006134 2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 07 19:00:16.010858 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.006138 2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 07 19:00:16.010858 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.006141 2563 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 07 19:00:16.010858 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.006146 2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 07 19:00:16.010858 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.006149 2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 07 19:00:16.010858 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.006153 2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 07 19:00:16.010858 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.006254 2563 flags.go:64] FLAG: --address="0.0.0.0" Jul 07 19:00:16.010858 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.006264 2563 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 07 19:00:16.010858 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.006274 2563 flags.go:64] FLAG: --anonymous-auth="true" Jul 07 19:00:16.010858 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.006280 2563 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 07 19:00:16.010858 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.006289 2563 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 07 19:00:16.010858 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.006293 2563 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 07 19:00:16.010858 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.006300 2563 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 07 19:00:16.010858 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.006305 2563 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 07 19:00:16.010858 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.006310 2563 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 07 19:00:16.010858 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.006315 2563 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 07 19:00:16.010858 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.006320 2563 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 07 19:00:16.010858 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.006324 2563 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 07 19:00:16.011436 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.006329 2563 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 07 19:00:16.011436 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.006334 2563 flags.go:64] FLAG: --cgroup-root="" Jul 07 19:00:16.011436 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.006339 2563 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 07 19:00:16.011436 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.006343 2563 flags.go:64] FLAG: --client-ca-file="" Jul 07 19:00:16.011436 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.006348 2563 flags.go:64] FLAG: --cloud-provider="external" Jul 07 19:00:16.011436 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.006366 2563 flags.go:64] FLAG: --cluster-dns="[]" Jul 07 19:00:16.011436 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.006372 2563 flags.go:64] FLAG: --cluster-domain="" Jul 07 19:00:16.011436 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.006376 2563 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 07 19:00:16.011436 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.006381 2563 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 07 19:00:16.011436 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.006385 2563 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 07 19:00:16.011436 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.006391 2563 flags.go:64] FLAG: --container-log-max-files="5" Jul 07 19:00:16.011436 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.006396 2563 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 07 19:00:16.011436 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.006401 2563 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 07 19:00:16.011436 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.006405 2563 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 07 19:00:16.011436 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.006410 2563 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 07 19:00:16.011436 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.006415 2563 flags.go:64] FLAG: --contention-profiling="false" Jul 07 19:00:16.011436 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.006419 2563 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 07 19:00:16.011436 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.006424 2563 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 07 19:00:16.011436 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.006430 2563 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 07 19:00:16.011436 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.006434 2563 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 07 19:00:16.011436 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.006440 2563 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 07 19:00:16.011436 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.006445 2563 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 07 19:00:16.011436 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.006450 2563 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 07 19:00:16.011436 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.006456 2563 flags.go:64] FLAG: --enable-load-reader="false" Jul 07 19:00:16.011436 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.006461 2563 flags.go:64] FLAG: --enable-server="true" Jul 07 19:00:16.012046 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.006465 2563 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 07 19:00:16.012046 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.006472 2563 flags.go:64] FLAG: --event-burst="100" Jul 07 19:00:16.012046 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.006476 2563 flags.go:64] FLAG: --event-qps="50" Jul 07 19:00:16.012046 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.006481 2563 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 07 19:00:16.012046 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.006485 2563 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 07 19:00:16.012046 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.006490 2563 flags.go:64] FLAG: --eviction-hard="" Jul 07 19:00:16.012046 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.006495 2563 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 07 19:00:16.012046 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.006500 2563 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 07 19:00:16.012046 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.006504 2563 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 07 19:00:16.012046 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.006508 2563 flags.go:64] FLAG: --eviction-soft="" Jul 07 19:00:16.012046 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.006512 2563 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 07 19:00:16.012046 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.006516 2563 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 07 19:00:16.012046 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.006521 2563 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 07 19:00:16.012046 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.006524 2563 flags.go:64] FLAG: --experimental-mounter-path="" Jul 07 19:00:16.012046 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.006529 2563 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 07 19:00:16.012046 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.006539 2563 flags.go:64] FLAG: --fail-swap-on="true" Jul 07 19:00:16.012046 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.006544 2563 flags.go:64] FLAG: --feature-gates="" Jul 07 19:00:16.012046 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.006550 2563 flags.go:64] FLAG: --file-check-frequency="20s" Jul 07 19:00:16.012046 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.006555 2563 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 07 19:00:16.012046 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.006559 2563 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 07 19:00:16.012046 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.006564 2563 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 07 19:00:16.012046 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.006568 2563 flags.go:64] FLAG: --healthz-port="10248" Jul 07 19:00:16.012046 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.006576 2563 flags.go:64] FLAG: --help="false" Jul 07 19:00:16.012046 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.006580 2563 flags.go:64] FLAG: --hostname-override="ip-10-0-142-134.ec2.internal" Jul 07 19:00:16.012046 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.006585 2563 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 07 19:00:16.012697 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.006589 2563 flags.go:64] FLAG: --http-check-frequency="20s" Jul 07 19:00:16.012697 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.006594 2563 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 07 19:00:16.012697 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.006599 2563 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 07 19:00:16.012697 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.006605 2563 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 07 19:00:16.012697 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.006609 2563 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 07 19:00:16.012697 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.006615 2563 flags.go:64] FLAG: --image-service-endpoint="" Jul 07 19:00:16.012697 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.006619 2563 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 07 19:00:16.012697 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.006623 2563 flags.go:64] FLAG: --kube-api-burst="100" Jul 07 19:00:16.012697 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.006627 2563 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 07 19:00:16.012697 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.006632 2563 flags.go:64] FLAG: --kube-api-qps="50" Jul 07 19:00:16.012697 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.006636 2563 flags.go:64] FLAG: --kube-reserved="" Jul 07 19:00:16.012697 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.006641 2563 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 07 19:00:16.012697 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.006645 2563 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 07 19:00:16.012697 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.006650 2563 flags.go:64] FLAG: --kubelet-cgroups="" Jul 07 19:00:16.012697 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.006654 2563 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 07 19:00:16.012697 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.006658 2563 flags.go:64] FLAG: --lock-file="" Jul 07 19:00:16.012697 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.006663 2563 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 07 19:00:16.012697 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.006667 2563 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 07 19:00:16.012697 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.006672 2563 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 07 19:00:16.012697 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.006680 2563 flags.go:64] FLAG: --log-json-split-stream="false" Jul 07 19:00:16.012697 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.006684 2563 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 07 19:00:16.012697 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.006688 2563 flags.go:64] FLAG: --log-text-split-stream="false" Jul 07 19:00:16.012697 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.006693 2563 flags.go:64] FLAG: --logging-format="text" Jul 07 19:00:16.013268 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.006697 2563 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 07 19:00:16.013268 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.006702 2563 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 07 19:00:16.013268 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.006707 2563 flags.go:64] FLAG: --manifest-url="" Jul 07 19:00:16.013268 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.006711 2563 flags.go:64] FLAG: --manifest-url-header="" Jul 07 19:00:16.013268 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.006717 2563 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 07 19:00:16.013268 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.006722 2563 flags.go:64] FLAG: --max-open-files="1000000" Jul 07 19:00:16.013268 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.006730 2563 flags.go:64] FLAG: --max-pods="110" Jul 07 19:00:16.013268 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.006735 2563 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 07 19:00:16.013268 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.006740 2563 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 07 19:00:16.013268 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.006744 2563 flags.go:64] FLAG: --memory-manager-policy="None" Jul 07 19:00:16.013268 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.006749 2563 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 07 19:00:16.013268 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.006754 2563 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 07 19:00:16.013268 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.006759 2563 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 07 19:00:16.013268 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.006763 2563 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 07 19:00:16.013268 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.006773 2563 flags.go:64] FLAG: --node-status-max-images="50" Jul 07 19:00:16.013268 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.006778 2563 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 07 19:00:16.013268 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.006782 2563 flags.go:64] FLAG: --oom-score-adj="-999" Jul 07 19:00:16.013268 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.006787 2563 flags.go:64] FLAG: --pod-cidr="" Jul 07 19:00:16.013268 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.006791 2563 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ce93f6b4197b9c62a26a3131f8057ffe9c9379c5079a4f62e5919b75151394e5" Jul 07 19:00:16.013268 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.006799 2563 flags.go:64] FLAG: --pod-manifest-path="" Jul 07 19:00:16.013268 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.006829 2563 flags.go:64] FLAG: --pod-max-pids="-1" Jul 07 19:00:16.013268 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.006852 2563 flags.go:64] FLAG: --pods-per-core="0" Jul 07 19:00:16.013268 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.006857 2563 flags.go:64] FLAG: --port="10250" Jul 07 19:00:16.013268 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.006861 2563 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 07 19:00:16.014177 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.006865 2563 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0578b302607aace22" Jul 07 19:00:16.014177 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.006869 2563 flags.go:64] FLAG: --qos-reserved="" Jul 07 19:00:16.014177 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.006873 2563 flags.go:64] FLAG: --read-only-port="10255" Jul 07 19:00:16.014177 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.006876 2563 flags.go:64] FLAG: --register-node="true" Jul 07 19:00:16.014177 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.006880 2563 flags.go:64] FLAG: --register-with-taints="" Jul 07 19:00:16.014177 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.006887 2563 flags.go:64] FLAG: --registry-burst="10" Jul 07 19:00:16.014177 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.006892 2563 flags.go:64] FLAG: --registry-qps="5" Jul 07 19:00:16.014177 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.006897 2563 flags.go:64] FLAG: --reserved-cpus="" Jul 07 19:00:16.014177 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.006902 2563 flags.go:64] FLAG: --reserved-memory="" Jul 07 19:00:16.014177 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.006906 2563 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 07 19:00:16.014177 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.006910 2563 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 07 19:00:16.014177 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.006913 2563 flags.go:64] FLAG: --rotate-certificates="false" Jul 07 19:00:16.014177 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.006916 2563 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 07 19:00:16.014177 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.006919 2563 flags.go:64] FLAG: --runonce="false" Jul 07 19:00:16.014177 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.006922 2563 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 07 19:00:16.014177 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.006925 2563 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 07 19:00:16.014177 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.006929 2563 flags.go:64] FLAG: --seccomp-default="false" Jul 07 19:00:16.014177 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.006932 2563 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 07 19:00:16.014177 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.006935 2563 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 07 19:00:16.014177 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.006939 2563 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 07 19:00:16.014177 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.006942 2563 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 07 19:00:16.014177 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.006945 2563 flags.go:64] FLAG: --storage-driver-password="root" Jul 07 19:00:16.014177 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.006949 2563 flags.go:64] FLAG: --storage-driver-secure="false" Jul 07 19:00:16.014177 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.006952 2563 flags.go:64] FLAG: --storage-driver-table="stats" Jul 07 19:00:16.014177 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.006955 2563 flags.go:64] FLAG: --storage-driver-user="root" Jul 07 19:00:16.015272 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.006958 2563 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 07 19:00:16.015272 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.006961 2563 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 07 19:00:16.015272 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.006966 2563 flags.go:64] FLAG: --system-cgroups="" Jul 07 19:00:16.015272 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.006970 2563 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 07 19:00:16.015272 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.006987 2563 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 07 19:00:16.015272 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.006991 2563 flags.go:64] FLAG: --tls-cert-file="" Jul 07 19:00:16.015272 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.006994 2563 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 07 19:00:16.015272 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.007000 2563 flags.go:64] FLAG: --tls-min-version="" Jul 07 19:00:16.015272 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.007003 2563 flags.go:64] FLAG: --tls-private-key-file="" Jul 07 19:00:16.015272 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.007007 2563 flags.go:64] FLAG: --topology-manager-policy="none" Jul 07 19:00:16.015272 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.007010 2563 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 07 19:00:16.015272 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.007013 2563 flags.go:64] FLAG: --topology-manager-scope="container" Jul 07 19:00:16.015272 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.007016 2563 flags.go:64] FLAG: --v="2" Jul 07 19:00:16.015272 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.007020 2563 flags.go:64] FLAG: --version="false" Jul 07 19:00:16.015272 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.007024 2563 flags.go:64] FLAG: --vmodule="" Jul 07 19:00:16.015272 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.007028 2563 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 07 19:00:16.015272 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.007032 2563 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 07 19:00:16.015272 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.007176 2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 07 19:00:16.015272 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.007181 2563 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 07 19:00:16.015272 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.007184 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 07 19:00:16.015272 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.007188 2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 07 19:00:16.015272 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.007193 2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 07 19:00:16.015272 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.007196 2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 07 19:00:16.015999 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.007199 2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 07 19:00:16.015999 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.007202 2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 07 19:00:16.015999 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.007206 2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 07 19:00:16.015999 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.007210 2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 07 19:00:16.015999 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.007215 2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 07 19:00:16.015999 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.007219 2563 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 07 19:00:16.015999 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.007224 2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 07 19:00:16.015999 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.007227 2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 07 19:00:16.015999 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.007230 2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 07 19:00:16.015999 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.007233 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 07 19:00:16.015999 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.007235 2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 07 19:00:16.015999 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.007238 2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 07 19:00:16.015999 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.007240 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 07 19:00:16.015999 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.007246 2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 07 19:00:16.015999 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.007250 2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 07 19:00:16.015999 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.007252 2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 07 19:00:16.015999 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.007255 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 07 19:00:16.015999 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.007257 2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 07 19:00:16.015999 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.007260 2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 07 19:00:16.015999 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.007262 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 07 19:00:16.016511 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.007265 2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 07 19:00:16.016511 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.007267 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 07 19:00:16.016511 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.007270 2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 07 19:00:16.016511 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.007273 2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 07 19:00:16.016511 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.007275 2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 07 19:00:16.016511 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.007279 2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 07 19:00:16.016511 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.007282 2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 07 19:00:16.016511 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.007286 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 07 19:00:16.016511 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.007291 2563 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 07 19:00:16.016511 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.007295 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 07 19:00:16.016511 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.007300 2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 07 19:00:16.016511 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.007305 2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 07 19:00:16.016511 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.007309 2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 07 19:00:16.016511 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.007311 2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 07 19:00:16.016511 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.007314 2563 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 07 19:00:16.016511 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.007316 2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 07 19:00:16.016511 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.007319 2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 07 19:00:16.016511 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.007321 2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 07 19:00:16.016511 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.007324 2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 07 19:00:16.016969 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.007326 2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 07 19:00:16.016969 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.007329 2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 07 19:00:16.016969 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.007331 2563 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 07 19:00:16.016969 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.007334 2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 07 19:00:16.016969 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.007336 2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 07 19:00:16.016969 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.007339 2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 07 19:00:16.016969 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.007344 2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 07 19:00:16.016969 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.007347 2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 07 19:00:16.016969 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.007373 2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 07 19:00:16.016969 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.007379 2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 07 19:00:16.016969 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.007383 2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 07 19:00:16.016969 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.007387 2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 07 19:00:16.016969 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.007391 2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 07 19:00:16.016969 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.007394 2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 07 19:00:16.016969 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.007397 2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 07 19:00:16.016969 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.007400 2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 07 19:00:16.016969 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.007403 2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 07 19:00:16.016969 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.007405 2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 07 19:00:16.016969 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.007408 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 07 19:00:16.017453 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.007411 2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 07 19:00:16.017453 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.007414 2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 07 19:00:16.017453 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.007417 2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 07 19:00:16.017453 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.007420 2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 07 19:00:16.017453 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.007422 2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 07 19:00:16.017453 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.007424 2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 07 19:00:16.017453 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.007427 2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 07 19:00:16.017453 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.007430 2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 07 19:00:16.017453 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.007432 2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 07 19:00:16.017453 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.007435 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 07 19:00:16.017453 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.007438 2563 feature_gate.go:328] unrecognized feature gate: Example Jul 07 19:00:16.017453 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.007441 2563 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 07 19:00:16.017453 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.007445 2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 07 19:00:16.017453 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.007449 2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 07 19:00:16.017453 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.007453 2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 07 19:00:16.017453 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.007462 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 07 19:00:16.017453 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.007467 2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 07 19:00:16.017453 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.007472 2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 07 19:00:16.017453 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.007477 2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 07 19:00:16.017916 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.007484 2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 07 19:00:16.017916 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.007487 2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 07 19:00:16.017916 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.007489 2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 07 19:00:16.017916 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.007493 2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 07 19:00:16.017916 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.007496 2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 07 19:00:16.017916 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.007498 2563 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 07 19:00:16.017916 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.007501 2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 07 19:00:16.017916 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.007504 2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 07 19:00:16.017916 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.007507 2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 07 19:00:16.017916 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.007510 2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 07 19:00:16.017916 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.007513 2563 feature_gate.go:328] unrecognized feature gate: Example2 Jul 07 19:00:16.017916 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.007516 2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 07 19:00:16.017916 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.007518 2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 07 19:00:16.017916 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.007521 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 07 19:00:16.017916 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.007524 2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 07 19:00:16.017916 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.007526 2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 07 19:00:16.018316 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.007532 2563 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 07 19:00:16.018316 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.016475 2563 server.go:529] "Kubelet version" kubeletVersion="v1.34.8" Jul 07 19:00:16.018316 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.016496 2563 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 07 19:00:16.018316 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.017554 2563 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 07 19:00:16.018316 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.017562 2563 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 07 19:00:16.018316 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.017626 2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 07 19:00:16.018316 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.017632 2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 07 19:00:16.018316 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.017635 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 07 19:00:16.018316 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.017639 2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 07 19:00:16.018316 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.017642 2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 07 19:00:16.018316 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.017645 2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 07 19:00:16.018316 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.017648 2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 07 19:00:16.018316 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.017651 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 07 19:00:16.018316 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.017653 2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 07 19:00:16.018787 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.017656 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 07 19:00:16.018787 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.017659 2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 07 19:00:16.018787 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.017662 2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 07 19:00:16.018787 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.017665 2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 07 19:00:16.018787 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.017668 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 07 19:00:16.018787 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.017670 2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 07 19:00:16.018787 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.017673 2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 07 19:00:16.018787 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.017676 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 07 19:00:16.018787 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.017679 2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 07 19:00:16.018787 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.017681 2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 07 19:00:16.018787 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.017684 2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 07 19:00:16.018787 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.017686 2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 07 19:00:16.018787 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.017690 2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 07 19:00:16.018787 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.017693 2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 07 19:00:16.018787 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.017697 2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 07 19:00:16.018787 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.017701 2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 07 19:00:16.018787 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.017705 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 07 19:00:16.018787 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.017708 2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 07 19:00:16.018787 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.017711 2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 07 19:00:16.018787 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.017714 2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 07 19:00:16.019268 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.017717 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 07 19:00:16.019268 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.017719 2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 07 19:00:16.019268 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.017722 2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 07 19:00:16.019268 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.017725 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 07 19:00:16.019268 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.017727 2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 07 19:00:16.019268 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.017730 2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 07 19:00:16.019268 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.017732 2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 07 19:00:16.019268 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.017735 2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 07 19:00:16.019268 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.017737 2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 07 19:00:16.019268 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.017740 2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 07 19:00:16.019268 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.017743 2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 07 19:00:16.019268 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.017745 2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 07 19:00:16.019268 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.017748 2563 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 07 19:00:16.019268 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.017751 2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 07 19:00:16.019268 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.017754 2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 07 19:00:16.019268 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.017756 2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 07 19:00:16.019268 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.017759 2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 07 19:00:16.019268 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.017763 2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 07 19:00:16.019268 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.017765 2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 07 19:00:16.019268 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.017768 2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 07 19:00:16.019781 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.017771 2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 07 19:00:16.019781 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.017774 2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 07 19:00:16.019781 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.017776 2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 07 19:00:16.019781 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.017779 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 07 19:00:16.019781 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.017782 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 07 19:00:16.019781 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.017785 2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 07 19:00:16.019781 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.017787 2563 feature_gate.go:328] unrecognized feature gate: Example Jul 07 19:00:16.019781 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.017790 2563 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 07 19:00:16.019781 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.017793 2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 07 19:00:16.019781 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.017795 2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 07 19:00:16.019781 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.017798 2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 07 19:00:16.019781 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.017800 2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 07 19:00:16.019781 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.017803 2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 07 19:00:16.019781 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.017806 2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 07 19:00:16.019781 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.017808 2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 07 19:00:16.019781 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.017811 2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 07 19:00:16.019781 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.017813 2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 07 19:00:16.019781 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.017816 2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 07 19:00:16.019781 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.017818 2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 07 19:00:16.019781 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.017821 2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 07 19:00:16.020269 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.017823 2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 07 19:00:16.020269 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.017826 2563 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 07 19:00:16.020269 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.017828 2563 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 07 19:00:16.020269 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.017831 2563 feature_gate.go:328] unrecognized feature gate: Example2 Jul 07 19:00:16.020269 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.017833 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 07 19:00:16.020269 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.017836 2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 07 19:00:16.020269 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.017838 2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 07 19:00:16.020269 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.017841 2563 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 07 19:00:16.020269 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.017843 2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 07 19:00:16.020269 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.017846 2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 07 19:00:16.020269 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.017850 2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 07 19:00:16.020269 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.017852 2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 07 19:00:16.020269 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.017856 2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 07 19:00:16.020269 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.017859 2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 07 19:00:16.020269 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.017863 2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 07 19:00:16.020269 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.017867 2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 07 19:00:16.020269 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.017871 2563 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 07 19:00:16.020269 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.017874 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 07 19:00:16.020269 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.017877 2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 07 19:00:16.020744 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.017879 2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 07 19:00:16.020744 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.017882 2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 07 19:00:16.020744 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.017884 2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 07 19:00:16.020744 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.017887 2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 07 19:00:16.020744 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.017890 2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 07 19:00:16.020744 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.017893 2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 07 19:00:16.020744 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.017896 2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 07 19:00:16.020744 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.017898 2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 07 19:00:16.020744 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.017901 2563 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 07 19:00:16.020744 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.017904 2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 07 19:00:16.020744 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.017906 2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 07 19:00:16.020744 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.017912 2563 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 07 19:00:16.020744 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.018039 2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 07 19:00:16.020744 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.018044 2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 07 19:00:16.020744 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.018046 2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 07 19:00:16.021156 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.018049 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 07 19:00:16.021156 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.018052 2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 07 19:00:16.021156 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.018054 2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 07 19:00:16.021156 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.018058 2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 07 19:00:16.021156 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.018060 2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 07 19:00:16.021156 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.018063 2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 07 19:00:16.021156 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.018065 2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 07 19:00:16.021156 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.018068 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 07 19:00:16.021156 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.018070 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 07 19:00:16.021156 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.018073 2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 07 19:00:16.021156 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.018076 2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 07 19:00:16.021156 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.018079 2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 07 19:00:16.021156 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.018081 2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 07 19:00:16.021156 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.018084 2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 07 19:00:16.021156 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.018087 2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 07 19:00:16.021156 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.018090 2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 07 19:00:16.021156 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.018092 2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 07 19:00:16.021156 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.018095 2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 07 19:00:16.021156 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.018097 2563 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 07 19:00:16.021156 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.018100 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 07 19:00:16.021673 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.018103 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 07 19:00:16.021673 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.018105 2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 07 19:00:16.021673 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.018108 2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 07 19:00:16.021673 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.018110 2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 07 19:00:16.021673 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.018113 2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 07 19:00:16.021673 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.018115 2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 07 19:00:16.021673 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.018120 2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 07 19:00:16.021673 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.018124 2563 feature_gate.go:328] unrecognized feature gate: Example2 Jul 07 19:00:16.021673 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.018127 2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 07 19:00:16.021673 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.018130 2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 07 19:00:16.021673 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.018133 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 07 19:00:16.021673 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.018136 2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 07 19:00:16.021673 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.018139 2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 07 19:00:16.021673 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.018142 2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 07 19:00:16.021673 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.018144 2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 07 19:00:16.021673 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.018147 2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 07 19:00:16.021673 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.018149 2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 07 19:00:16.021673 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.018152 2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 07 19:00:16.021673 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.018155 2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 07 19:00:16.022130 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.018157 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 07 19:00:16.022130 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.018160 2563 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 07 19:00:16.022130 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.018162 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 07 19:00:16.022130 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.018166 2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 07 19:00:16.022130 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.018169 2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 07 19:00:16.022130 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.018171 2563 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 07 19:00:16.022130 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.018174 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 07 19:00:16.022130 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.018177 2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 07 19:00:16.022130 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.018180 2563 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 07 19:00:16.022130 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.018183 2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 07 19:00:16.022130 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.018185 2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 07 19:00:16.022130 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.018188 2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 07 19:00:16.022130 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.018190 2563 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 07 19:00:16.022130 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.018193 2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 07 19:00:16.022130 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.018196 2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 07 19:00:16.022130 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.018198 2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 07 19:00:16.022130 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.018200 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 07 19:00:16.022130 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.018203 2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 07 19:00:16.022130 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.018206 2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 07 19:00:16.022130 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.018208 2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 07 19:00:16.022130 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.018211 2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 07 19:00:16.022673 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.018213 2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 07 19:00:16.022673 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.018216 2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 07 19:00:16.022673 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.018218 2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 07 19:00:16.022673 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.018221 2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 07 19:00:16.022673 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.018223 2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 07 19:00:16.022673 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.018225 2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 07 19:00:16.022673 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.018228 2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 07 19:00:16.022673 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.018231 2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 07 19:00:16.022673 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.018233 2563 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 07 19:00:16.022673 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.018236 2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 07 19:00:16.022673 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.018239 2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 07 19:00:16.022673 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.018241 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 07 19:00:16.022673 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.018244 2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 07 19:00:16.022673 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.018246 2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 07 19:00:16.022673 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.018253 2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 07 19:00:16.022673 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.018256 2563 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 07 19:00:16.022673 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.018260 2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 07 19:00:16.022673 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.018263 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 07 19:00:16.022673 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.018266 2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 07 19:00:16.023122 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.018269 2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 07 19:00:16.023122 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.018272 2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 07 19:00:16.023122 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.018274 2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 07 19:00:16.023122 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.018277 2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 07 19:00:16.023122 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.018279 2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 07 19:00:16.023122 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.018282 2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 07 19:00:16.023122 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.018284 2563 feature_gate.go:328] unrecognized feature gate: Example Jul 07 19:00:16.023122 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.018287 2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 07 19:00:16.023122 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.018289 2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 07 19:00:16.023122 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.018292 2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 07 19:00:16.023122 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.018294 2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 07 19:00:16.023122 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.018297 2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 07 19:00:16.023122 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.018300 2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 07 19:00:16.023122 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.018302 2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 07 19:00:16.023122 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.018305 2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 07 19:00:16.023122 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.018308 2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 07 19:00:16.023122 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:16.018311 2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 07 19:00:16.023549 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.018316 2563 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 07 19:00:16.023549 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.018454 2563 server.go:962] "Client rotation is on, will bootstrap in background" Jul 07 19:00:16.023549 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.021079 2563 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 07 19:00:16.023549 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.022746 2563 server.go:1019] "Starting client certificate rotation" Jul 07 19:00:16.023549 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.022850 2563 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 07 19:00:16.023549 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.022888 2563 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 07 19:00:16.026425 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.026407 2563 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 07 19:00:16.027693 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.027679 2563 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 07 19:00:16.041867 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.041802 2563 log.go:25] "Validated CRI v1 runtime API" Jul 07 19:00:16.044448 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.044430 2563 log.go:25] "Validated CRI v1 image API" Jul 07 19:00:16.045311 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.045285 2563 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 07 19:00:16.050338 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.050317 2563 fs.go:135] Filesystem UUIDs: map[5507cc30-2941-4185-906b-30f393c065ca:/dev/nvme0n1p4 585dbbe6-4465-4989-a844-1795b4b2b3e3:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2] Jul 07 19:00:16.050425 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.050336 2563 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 07 19:00:16.055690 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.055667 2563 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 07 19:00:16.055860 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.055704 2563 manager.go:216] Machine: {Timestamp:2026-07-07 19:00:16.055131839 +0000 UTC m=+0.376735507 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099617 MemoryCapacity:33164488704 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec28922a7043e2f961a153728a937227 SystemUUID:ec28922a-7043-e2f9-61a1-53728a937227 BootID:3698c91b-246e-468f-b77c-f800a002afcb Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582242304 Type:vfs Inodes:4048399 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:bb:11:2c:4c:c5 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:bb:11:2c:4c:c5 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:7e:4b:c4:9c:e2:63 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164488704 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 07 19:00:16.055860 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.055854 2563 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 07 19:00:16.055972 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.055939 2563 manager.go:232] Version: {KernelVersion:5.14.0-570.125.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260630-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 07 19:00:16.058731 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.058697 2563 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 07 19:00:16.058907 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.058730 2563 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-142-134.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 07 19:00:16.059651 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.059640 2563 topology_manager.go:138] "Creating topology manager with none policy" Jul 07 19:00:16.059687 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.059655 2563 container_manager_linux.go:309] "Creating device plugin manager" Jul 07 19:00:16.059687 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.059665 2563 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 07 19:00:16.059749 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.059693 2563 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 07 19:00:16.061708 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.061696 2563 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 07 19:00:16.066369 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.066343 2563 state_mem.go:36] "Initialized new in-memory state store" Jul 07 19:00:16.066516 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.066505 2563 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 07 19:00:16.066553 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.066541 2563 kubelet.go:486] "Attempting to sync node with API server" Jul 07 19:00:16.066581 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.066555 2563 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 07 19:00:16.066581 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.066573 2563 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 07 19:00:16.066688 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.066587 2563 kubelet.go:398] "Adding apiserver pod source" Jul 07 19:00:16.066688 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.066599 2563 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 07 19:00:16.067616 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.067596 2563 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 07 19:00:16.067708 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.067632 2563 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 07 19:00:16.072803 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.072785 2563 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.8-3.rhaos4.21.git91b4b0b.el9" apiVersion="v1" Jul 07 19:00:16.075399 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.075383 2563 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 07 19:00:16.075469 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.075422 2563 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 07 19:00:16.076900 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.076887 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 07 19:00:16.076951 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.076907 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 07 19:00:16.076951 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.076916 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 07 19:00:16.076951 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.076923 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 07 19:00:16.076951 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.076935 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 07 19:00:16.076951 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.076943 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 07 19:00:16.076951 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.076953 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 07 19:00:16.077111 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.076961 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 07 19:00:16.077111 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.076970 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 07 19:00:16.077111 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.076978 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 07 19:00:16.077111 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.076989 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 07 19:00:16.077111 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.076996 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 07 19:00:16.077644 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:00:16.077592 2563 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 07 19:00:16.077715 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:00:16.077682 2563 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-142-134.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 07 19:00:16.078067 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.078054 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 07 19:00:16.078067 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.078067 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 07 19:00:16.080544 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.080514 2563 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-142-134.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 07 19:00:16.081764 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.081752 2563 server.go:1268] "Started kubelet" Jul 07 19:00:16.081973 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.081901 2563 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 07 19:00:16.082010 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.081998 2563 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 07 19:00:16.082593 ip-10-0-142-134 systemd[1]: Started Kubernetes Kubelet. Jul 07 19:00:16.082729 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.082615 2563 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 07 19:00:16.084159 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.084134 2563 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 07 19:00:16.085454 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.085441 2563 server.go:310] "Adding debug handlers to kubelet server" Jul 07 19:00:16.087989 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:00:16.087960 2563 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 07 19:00:16.092186 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.092171 2563 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 07 19:00:16.092253 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.092193 2563 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 07 19:00:16.092503 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:00:16.092473 2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-134.ec2.internal\" not found" Jul 07 19:00:16.092967 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.092951 2563 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 07 19:00:16.093044 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.092975 2563 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 07 19:00:16.093092 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.093060 2563 reconstruct.go:97] "Volume reconstruction finished" Jul 07 19:00:16.093092 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.093068 2563 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 07 19:00:16.093092 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.093069 2563 reconciler.go:29] "Reconciler: start to sync state" Jul 07 19:00:16.094696 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.094611 2563 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 07 19:00:16.094696 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.094628 2563 factory.go:55] Registering systemd factory Jul 07 19:00:16.094696 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.094637 2563 factory.go:223] Registration of the systemd container factory successfully Jul 07 19:00:16.095509 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.095491 2563 factory.go:153] Registering CRI-O factory Jul 07 19:00:16.095509 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.095510 2563 factory.go:223] Registration of the crio container factory successfully Jul 07 19:00:16.095630 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.095532 2563 factory.go:103] Registering Raw factory Jul 07 19:00:16.095630 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.095546 2563 manager.go:1196] Started watching for new ooms in manager Jul 07 19:00:16.096263 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.096240 2563 manager.go:349] Starting recovery of all containers Jul 07 19:00:16.098329 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:00:16.098303 2563 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 07 19:00:16.099025 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:00:16.098995 2563 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-142-134.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 07 19:00:16.099674 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:00:16.098400 2563 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-134.ec2.internal.18c017116eee79c0 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-134.ec2.internal,UID:ip-10-0-142-134.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-142-134.ec2.internal,},FirstTimestamp:2026-07-07 19:00:16.081721792 +0000 UTC m=+0.403325461,LastTimestamp:2026-07-07 19:00:16.081721792 +0000 UTC m=+0.403325461,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-134.ec2.internal,}" Jul 07 19:00:16.109456 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.109433 2563 manager.go:354] Recovery completed Jul 07 19:00:16.113729 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.113716 2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 07 19:00:16.115562 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.115540 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-134.ec2.internal" event="NodeHasSufficientMemory" Jul 07 19:00:16.115640 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.115577 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-134.ec2.internal" event="NodeHasNoDiskPressure" Jul 07 19:00:16.115640 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.115588 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-134.ec2.internal" event="NodeHasSufficientPID" Jul 07 19:00:16.116175 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.116162 2563 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 07 19:00:16.116175 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.116174 2563 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 07 19:00:16.116283 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.116184 2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 07 19:00:16.116283 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.116190 2563 state_mem.go:36] "Initialized new in-memory state store" Jul 07 19:00:16.116900 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.116885 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-134.ec2.internal" event="NodeHasSufficientMemory" Jul 07 19:00:16.116971 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.116915 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-134.ec2.internal" event="NodeHasNoDiskPressure" Jul 07 19:00:16.116971 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.116929 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-134.ec2.internal" event="NodeHasSufficientPID" Jul 07 19:00:16.117495 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:00:16.117424 2563 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-134.ec2.internal.18c0171170f2d533 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-134.ec2.internal,UID:ip-10-0-142-134.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-142-134.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-142-134.ec2.internal,},FirstTimestamp:2026-07-07 19:00:16.115561779 +0000 UTC m=+0.437165454,LastTimestamp:2026-07-07 19:00:16.115561779 +0000 UTC m=+0.437165454,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-134.ec2.internal,}" Jul 07 19:00:16.118502 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.118491 2563 policy_none.go:49] "None policy: Start" Jul 07 19:00:16.118568 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.118506 2563 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 07 19:00:16.118568 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.118520 2563 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 07 19:00:16.120312 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.120291 2563 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-cxdrf" Jul 07 19:00:16.120537 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.120525 2563 policy_none.go:47] "Start" Jul 07 19:00:16.127301 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.127282 2563 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-cxdrf" Jul 07 19:00:16.130645 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:00:16.130553 2563 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-134.ec2.internal.18c0171170f3272a default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-134.ec2.internal,UID:ip-10-0-142-134.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-142-134.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-142-134.ec2.internal,},FirstTimestamp:2026-07-07 19:00:16.115582762 +0000 UTC m=+0.437186429,LastTimestamp:2026-07-07 19:00:16.115582762 +0000 UTC m=+0.437186429,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-134.ec2.internal,}" Jul 07 19:00:16.160208 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.160039 2563 manager.go:337] "Starting Device Plugin manager" Jul 07 19:00:16.174493 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:00:16.160230 2563 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 07 19:00:16.174493 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.160243 2563 server.go:87] "Starting device plugin registration server" Jul 07 19:00:16.174493 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.160579 2563 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 07 19:00:16.174493 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.160600 2563 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 07 19:00:16.174493 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.160728 2563 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 07 19:00:16.174493 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.160818 2563 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 07 19:00:16.174493 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.160843 2563 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 07 19:00:16.174493 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:00:16.161537 2563 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 07 19:00:16.174493 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:00:16.161575 2563 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-142-134.ec2.internal\" not found" Jul 07 19:00:16.174493 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.162646 2563 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 07 19:00:16.174493 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.164254 2563 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 07 19:00:16.174493 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.164303 2563 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 07 19:00:16.174493 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.164328 2563 kubelet.go:2448] "Starting kubelet main sync loop" Jul 07 19:00:16.174493 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:00:16.164381 2563 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 07 19:00:16.174493 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.167127 2563 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 07 19:00:16.261367 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.261320 2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 07 19:00:16.262522 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.262495 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-134.ec2.internal" event="NodeHasSufficientMemory" Jul 07 19:00:16.262629 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.262532 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-134.ec2.internal" event="NodeHasNoDiskPressure" Jul 07 19:00:16.262629 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.262558 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-134.ec2.internal" event="NodeHasSufficientPID" Jul 07 19:00:16.262629 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.262594 2563 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-142-134.ec2.internal" Jul 07 19:00:16.264650 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.264621 2563 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-134.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-142-134.ec2.internal"] Jul 07 19:00:16.264737 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.264691 2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 07 19:00:16.265478 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.265463 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-134.ec2.internal" event="NodeHasSufficientMemory" Jul 07 19:00:16.265576 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.265496 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-134.ec2.internal" event="NodeHasNoDiskPressure" Jul 07 19:00:16.265576 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.265509 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-134.ec2.internal" event="NodeHasSufficientPID" Jul 07 19:00:16.267968 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.267952 2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 07 19:00:16.268085 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.268069 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-134.ec2.internal" Jul 07 19:00:16.268133 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.268102 2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 07 19:00:16.268669 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.268653 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-134.ec2.internal" event="NodeHasSufficientMemory" Jul 07 19:00:16.268728 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.268684 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-134.ec2.internal" event="NodeHasNoDiskPressure" Jul 07 19:00:16.268728 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.268695 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-134.ec2.internal" event="NodeHasSufficientPID" Jul 07 19:00:16.268793 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.268653 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-134.ec2.internal" event="NodeHasSufficientMemory" Jul 07 19:00:16.268793 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.268753 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-134.ec2.internal" event="NodeHasNoDiskPressure" Jul 07 19:00:16.268793 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.268766 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-134.ec2.internal" event="NodeHasSufficientPID" Jul 07 19:00:16.271053 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.271037 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-134.ec2.internal" Jul 07 19:00:16.271138 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.271061 2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 07 19:00:16.271754 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.271740 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-134.ec2.internal" event="NodeHasSufficientMemory" Jul 07 19:00:16.271823 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.271765 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-134.ec2.internal" event="NodeHasNoDiskPressure" Jul 07 19:00:16.271823 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.271778 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-134.ec2.internal" event="NodeHasSufficientPID" Jul 07 19:00:16.277185 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.277170 2563 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-142-134.ec2.internal" Jul 07 19:00:16.277249 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:00:16.277191 2563 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-142-134.ec2.internal\": node \"ip-10-0-142-134.ec2.internal\" not found" Jul 07 19:00:16.294286 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.294205 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/067754c774fbc99720cce604846fd8f3-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-134.ec2.internal\" (UID: \"067754c774fbc99720cce604846fd8f3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-134.ec2.internal" Jul 07 19:00:16.294286 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.294238 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/067754c774fbc99720cce604846fd8f3-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-134.ec2.internal\" (UID: \"067754c774fbc99720cce604846fd8f3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-134.ec2.internal" Jul 07 19:00:16.294286 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.294260 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d64b863190df7f2a681aceffe233fdc3-config\") pod \"kube-apiserver-proxy-ip-10-0-142-134.ec2.internal\" (UID: \"d64b863190df7f2a681aceffe233fdc3\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-134.ec2.internal" Jul 07 19:00:16.296228 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:00:16.296213 2563 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-134.ec2.internal\" not found" node="ip-10-0-142-134.ec2.internal" Jul 07 19:00:16.300738 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:00:16.300719 2563 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-134.ec2.internal\" not found" node="ip-10-0-142-134.ec2.internal" Jul 07 19:00:16.311119 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:00:16.311094 2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-134.ec2.internal\" not found" Jul 07 19:00:16.395020 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.394984 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/067754c774fbc99720cce604846fd8f3-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-134.ec2.internal\" (UID: \"067754c774fbc99720cce604846fd8f3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-134.ec2.internal" Jul 07 19:00:16.395020 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.395021 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/067754c774fbc99720cce604846fd8f3-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-134.ec2.internal\" (UID: \"067754c774fbc99720cce604846fd8f3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-134.ec2.internal" Jul 07 19:00:16.395192 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.395040 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d64b863190df7f2a681aceffe233fdc3-config\") pod \"kube-apiserver-proxy-ip-10-0-142-134.ec2.internal\" (UID: \"d64b863190df7f2a681aceffe233fdc3\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-134.ec2.internal" Jul 07 19:00:16.395192 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.395084 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/067754c774fbc99720cce604846fd8f3-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-134.ec2.internal\" (UID: \"067754c774fbc99720cce604846fd8f3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-134.ec2.internal" Jul 07 19:00:16.395192 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.395140 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/067754c774fbc99720cce604846fd8f3-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-134.ec2.internal\" (UID: \"067754c774fbc99720cce604846fd8f3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-134.ec2.internal" Jul 07 19:00:16.395192 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.395169 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d64b863190df7f2a681aceffe233fdc3-config\") pod \"kube-apiserver-proxy-ip-10-0-142-134.ec2.internal\" (UID: \"d64b863190df7f2a681aceffe233fdc3\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-134.ec2.internal" Jul 07 19:00:16.411323 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:00:16.411292 2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-134.ec2.internal\" not found" Jul 07 19:00:16.511442 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:00:16.511406 2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-134.ec2.internal\" not found" Jul 07 19:00:16.600322 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.600266 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-134.ec2.internal" Jul 07 19:00:16.605811 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:16.605620 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-134.ec2.internal" Jul 07 19:00:16.611917 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:00:16.611901 2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-134.ec2.internal\" not found" Jul 07 19:00:16.712386 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:00:16.712331 2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-134.ec2.internal\" not found" Jul 07 19:00:16.812857 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:00:16.812828 2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-134.ec2.internal\" not found" Jul 07 19:00:18.219118 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:00:18.219036 2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-134.ec2.internal\" not found" Jul 07 19:00:18.319590 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:00:18.319547 2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-134.ec2.internal\" not found" Jul 07 19:00:18.327706 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.327688 2563 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 07 19:00:18.327867 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.327850 2563 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 07 19:00:18.369125 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.369100 2563 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 07 19:00:18.372753 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.372733 2563 apiserver.go:52] "Watching apiserver" Jul 07 19:00:18.381796 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.381774 2563 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 07 19:00:18.384448 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.384423 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-x7d7k","openshift-multus/network-metrics-daemon-h8m9b","openshift-dns/node-resolver-whw2v","openshift-image-registry/node-ca-s272w","openshift-network-diagnostics/network-check-target-d8t2d","openshift-network-operator/iptables-alerter-8ksxw","openshift-ovn-kubernetes/ovnkube-node-zzwbq","kube-system/global-pull-secret-syncer-lk7md","kube-system/konnectivity-agent-dsvgs","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qkmgh","openshift-cluster-node-tuning-operator/tuned-bjzmq","openshift-multus/multus-additional-cni-plugins-jls8k"] Jul 07 19:00:18.387055 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.387039 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-x7d7k" Jul 07 19:00:18.389279 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.389260 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h8m9b" Jul 07 19:00:18.389365 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:00:18.389346 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h8m9b" podUID="396810c7-8961-4755-b0f6-02a9d7e02ae3" Jul 07 19:00:18.391002 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.390980 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 07 19:00:18.391097 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.391030 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 07 19:00:18.391160 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.391113 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-pqr72\"" Jul 07 19:00:18.391160 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.391112 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 07 19:00:18.391276 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.391188 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 07 19:00:18.391423 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.391408 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-whw2v" Jul 07 19:00:18.393088 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.393071 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 07 19:00:18.393157 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.393140 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-95q2m\"" Jul 07 19:00:18.393200 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.393141 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 07 19:00:18.393600 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.393586 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-s272w" Jul 07 19:00:18.395477 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.395462 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 07 19:00:18.395572 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.395536 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 07 19:00:18.395772 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.395759 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d8t2d" Jul 07 19:00:18.395857 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:00:18.395838 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d8t2d" podUID="1d84b355-0ac7-4d14-83b6-7c18077e9319" Jul 07 19:00:18.395921 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.395868 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-g5td5\"" Jul 07 19:00:18.395921 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.395908 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 07 19:00:18.397841 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.397828 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-8ksxw" Jul 07 19:00:18.397985 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.397968 2563 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 07 19:00:18.398076 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.398063 2563 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-134.ec2.internal" Jul 07 19:00:18.399902 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.399886 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-xzqls\"" Jul 07 19:00:18.400184 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.400171 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-zzwbq" Jul 07 19:00:18.400435 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.400304 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 07 19:00:18.400435 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.400315 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 07 19:00:18.400553 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.400496 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 07 19:00:18.402443 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.402427 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lk7md" Jul 07 19:00:18.402515 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:00:18.402494 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lk7md" podUID="59f0c8fc-bc9a-4273-aac2-a8ee23bf6b63" Jul 07 19:00:18.402856 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.402838 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 07 19:00:18.402856 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.402850 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 07 19:00:18.403125 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.402881 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 07 19:00:18.403125 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.402937 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 07 19:00:18.403125 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.403048 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-mqmcg\"" Jul 07 19:00:18.403330 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.403316 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 07 19:00:18.403393 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.403350 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 07 19:00:18.403491 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.403474 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4f0c5d5c-147f-426a-965d-8886e5e79984-host-slash\") pod \"ovnkube-node-zzwbq\" (UID: \"4f0c5d5c-147f-426a-965d-8886e5e79984\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzwbq" Jul 07 19:00:18.403536 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.403497 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4f0c5d5c-147f-426a-965d-8886e5e79984-run-openvswitch\") pod \"ovnkube-node-zzwbq\" (UID: \"4f0c5d5c-147f-426a-965d-8886e5e79984\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzwbq" Jul 07 19:00:18.403536 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.403516 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4f0c5d5c-147f-426a-965d-8886e5e79984-host-run-ovn-kubernetes\") pod \"ovnkube-node-zzwbq\" (UID: \"4f0c5d5c-147f-426a-965d-8886e5e79984\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzwbq" Jul 07 19:00:18.403630 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.403537 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4f0c5d5c-147f-426a-965d-8886e5e79984-ovn-node-metrics-cert\") pod \"ovnkube-node-zzwbq\" (UID: \"4f0c5d5c-147f-426a-965d-8886e5e79984\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzwbq" Jul 07 19:00:18.403630 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.403559 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a1221769-6ea8-43b7-aa11-e4b2d3699a7b-system-cni-dir\") pod \"multus-x7d7k\" (UID: \"a1221769-6ea8-43b7-aa11-e4b2d3699a7b\") " pod="openshift-multus/multus-x7d7k" Jul 07 19:00:18.403630 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.403582 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a1221769-6ea8-43b7-aa11-e4b2d3699a7b-multus-conf-dir\") pod \"multus-x7d7k\" (UID: \"a1221769-6ea8-43b7-aa11-e4b2d3699a7b\") " pod="openshift-multus/multus-x7d7k" Jul 07 19:00:18.403736 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.403626 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/3953dadf-98ed-4933-a330-1191906b6cff-hosts-file\") pod \"node-resolver-whw2v\" (UID: \"3953dadf-98ed-4933-a330-1191906b6cff\") " pod="openshift-dns/node-resolver-whw2v" Jul 07 19:00:18.403736 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.403672 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j28vs\" (UniqueName: \"kubernetes.io/projected/3953dadf-98ed-4933-a330-1191906b6cff-kube-api-access-j28vs\") pod \"node-resolver-whw2v\" (UID: \"3953dadf-98ed-4933-a330-1191906b6cff\") " pod="openshift-dns/node-resolver-whw2v" Jul 07 19:00:18.403736 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.403700 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4f0c5d5c-147f-426a-965d-8886e5e79984-etc-openvswitch\") pod \"ovnkube-node-zzwbq\" (UID: \"4f0c5d5c-147f-426a-965d-8886e5e79984\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzwbq" Jul 07 19:00:18.403736 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.403725 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4f0c5d5c-147f-426a-965d-8886e5e79984-host-cni-bin\") pod \"ovnkube-node-zzwbq\" (UID: \"4f0c5d5c-147f-426a-965d-8886e5e79984\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzwbq" Jul 07 19:00:18.403923 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.403739 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bgqkm\" (UniqueName: \"kubernetes.io/projected/1d84b355-0ac7-4d14-83b6-7c18077e9319-kube-api-access-bgqkm\") pod \"network-check-target-d8t2d\" (UID: \"1d84b355-0ac7-4d14-83b6-7c18077e9319\") " pod="openshift-network-diagnostics/network-check-target-d8t2d" Jul 07 19:00:18.403923 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.403770 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4f0c5d5c-147f-426a-965d-8886e5e79984-host-kubelet\") pod \"ovnkube-node-zzwbq\" (UID: \"4f0c5d5c-147f-426a-965d-8886e5e79984\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzwbq" Jul 07 19:00:18.403923 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.403801 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4f0c5d5c-147f-426a-965d-8886e5e79984-var-lib-openvswitch\") pod \"ovnkube-node-zzwbq\" (UID: \"4f0c5d5c-147f-426a-965d-8886e5e79984\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzwbq" Jul 07 19:00:18.403923 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.403826 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4f0c5d5c-147f-426a-965d-8886e5e79984-run-ovn\") pod \"ovnkube-node-zzwbq\" (UID: \"4f0c5d5c-147f-426a-965d-8886e5e79984\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzwbq" Jul 07 19:00:18.403923 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.403847 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a6abe188-2b1b-427a-ba0c-e93930187c7e-host\") pod \"node-ca-s272w\" (UID: \"a6abe188-2b1b-427a-ba0c-e93930187c7e\") " pod="openshift-image-registry/node-ca-s272w" Jul 07 19:00:18.403923 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.403871 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/396810c7-8961-4755-b0f6-02a9d7e02ae3-metrics-certs\") pod \"network-metrics-daemon-h8m9b\" (UID: \"396810c7-8961-4755-b0f6-02a9d7e02ae3\") " pod="openshift-multus/network-metrics-daemon-h8m9b" Jul 07 19:00:18.403923 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.403901 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4f0c5d5c-147f-426a-965d-8886e5e79984-host-run-netns\") pod \"ovnkube-node-zzwbq\" (UID: \"4f0c5d5c-147f-426a-965d-8886e5e79984\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzwbq" Jul 07 19:00:18.404311 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.403934 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4f0c5d5c-147f-426a-965d-8886e5e79984-node-log\") pod \"ovnkube-node-zzwbq\" (UID: \"4f0c5d5c-147f-426a-965d-8886e5e79984\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzwbq" Jul 07 19:00:18.404311 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.403975 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4f0c5d5c-147f-426a-965d-8886e5e79984-host-cni-netd\") pod \"ovnkube-node-zzwbq\" (UID: \"4f0c5d5c-147f-426a-965d-8886e5e79984\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzwbq" Jul 07 19:00:18.404311 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.404000 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a1221769-6ea8-43b7-aa11-e4b2d3699a7b-cnibin\") pod \"multus-x7d7k\" (UID: \"a1221769-6ea8-43b7-aa11-e4b2d3699a7b\") " pod="openshift-multus/multus-x7d7k" Jul 07 19:00:18.404311 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.404023 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a1221769-6ea8-43b7-aa11-e4b2d3699a7b-host-run-netns\") pod \"multus-x7d7k\" (UID: \"a1221769-6ea8-43b7-aa11-e4b2d3699a7b\") " pod="openshift-multus/multus-x7d7k" Jul 07 19:00:18.404311 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.404048 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a1221769-6ea8-43b7-aa11-e4b2d3699a7b-multus-daemon-config\") pod \"multus-x7d7k\" (UID: \"a1221769-6ea8-43b7-aa11-e4b2d3699a7b\") " pod="openshift-multus/multus-x7d7k" Jul 07 19:00:18.404311 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.404070 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a1221769-6ea8-43b7-aa11-e4b2d3699a7b-etc-kubernetes\") pod \"multus-x7d7k\" (UID: \"a1221769-6ea8-43b7-aa11-e4b2d3699a7b\") " pod="openshift-multus/multus-x7d7k" Jul 07 19:00:18.404311 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.404093 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4f0c5d5c-147f-426a-965d-8886e5e79984-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-zzwbq\" (UID: \"4f0c5d5c-147f-426a-965d-8886e5e79984\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzwbq" Jul 07 19:00:18.404311 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.404131 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4f0c5d5c-147f-426a-965d-8886e5e79984-ovnkube-script-lib\") pod \"ovnkube-node-zzwbq\" (UID: \"4f0c5d5c-147f-426a-965d-8886e5e79984\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzwbq" Jul 07 19:00:18.404311 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.404150 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a1221769-6ea8-43b7-aa11-e4b2d3699a7b-multus-socket-dir-parent\") pod \"multus-x7d7k\" (UID: \"a1221769-6ea8-43b7-aa11-e4b2d3699a7b\") " pod="openshift-multus/multus-x7d7k" Jul 07 19:00:18.404311 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.404165 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a1221769-6ea8-43b7-aa11-e4b2d3699a7b-host-var-lib-cni-bin\") pod \"multus-x7d7k\" (UID: \"a1221769-6ea8-43b7-aa11-e4b2d3699a7b\") " pod="openshift-multus/multus-x7d7k" Jul 07 19:00:18.404311 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.404179 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a1221769-6ea8-43b7-aa11-e4b2d3699a7b-host-var-lib-kubelet\") pod \"multus-x7d7k\" (UID: \"a1221769-6ea8-43b7-aa11-e4b2d3699a7b\") " pod="openshift-multus/multus-x7d7k" Jul 07 19:00:18.404311 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.404194 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a1221769-6ea8-43b7-aa11-e4b2d3699a7b-host-run-multus-certs\") pod \"multus-x7d7k\" (UID: \"a1221769-6ea8-43b7-aa11-e4b2d3699a7b\") " pod="openshift-multus/multus-x7d7k" Jul 07 19:00:18.404311 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.404219 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v5k55\" (UniqueName: \"kubernetes.io/projected/a1221769-6ea8-43b7-aa11-e4b2d3699a7b-kube-api-access-v5k55\") pod \"multus-x7d7k\" (UID: \"a1221769-6ea8-43b7-aa11-e4b2d3699a7b\") " pod="openshift-multus/multus-x7d7k" Jul 07 19:00:18.404311 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.404246 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdc8d\" (UniqueName: \"kubernetes.io/projected/a6abe188-2b1b-427a-ba0c-e93930187c7e-kube-api-access-fdc8d\") pod \"node-ca-s272w\" (UID: \"a6abe188-2b1b-427a-ba0c-e93930187c7e\") " pod="openshift-image-registry/node-ca-s272w" Jul 07 19:00:18.404311 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.404268 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4f0c5d5c-147f-426a-965d-8886e5e79984-env-overrides\") pod \"ovnkube-node-zzwbq\" (UID: \"4f0c5d5c-147f-426a-965d-8886e5e79984\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzwbq" Jul 07 19:00:18.404311 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.404294 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4f0c5d5c-147f-426a-965d-8886e5e79984-log-socket\") pod \"ovnkube-node-zzwbq\" (UID: \"4f0c5d5c-147f-426a-965d-8886e5e79984\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzwbq" Jul 07 19:00:18.404311 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.404317 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4f0c5d5c-147f-426a-965d-8886e5e79984-ovnkube-config\") pod \"ovnkube-node-zzwbq\" (UID: \"4f0c5d5c-147f-426a-965d-8886e5e79984\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzwbq" Jul 07 19:00:18.404946 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.404339 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8rnk\" (UniqueName: \"kubernetes.io/projected/4f0c5d5c-147f-426a-965d-8886e5e79984-kube-api-access-k8rnk\") pod \"ovnkube-node-zzwbq\" (UID: \"4f0c5d5c-147f-426a-965d-8886e5e79984\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzwbq" Jul 07 19:00:18.404946 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.404361 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a1221769-6ea8-43b7-aa11-e4b2d3699a7b-os-release\") pod \"multus-x7d7k\" (UID: \"a1221769-6ea8-43b7-aa11-e4b2d3699a7b\") " pod="openshift-multus/multus-x7d7k" Jul 07 19:00:18.404946 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.404384 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a1221769-6ea8-43b7-aa11-e4b2d3699a7b-host-run-k8s-cni-cncf-io\") pod \"multus-x7d7k\" (UID: \"a1221769-6ea8-43b7-aa11-e4b2d3699a7b\") " pod="openshift-multus/multus-x7d7k" Jul 07 19:00:18.404946 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.404424 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a1221769-6ea8-43b7-aa11-e4b2d3699a7b-host-var-lib-cni-multus\") pod \"multus-x7d7k\" (UID: \"a1221769-6ea8-43b7-aa11-e4b2d3699a7b\") " pod="openshift-multus/multus-x7d7k" Jul 07 19:00:18.404946 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.404451 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mqwpr\" (UniqueName: \"kubernetes.io/projected/396810c7-8961-4755-b0f6-02a9d7e02ae3-kube-api-access-mqwpr\") pod \"network-metrics-daemon-h8m9b\" (UID: \"396810c7-8961-4755-b0f6-02a9d7e02ae3\") " pod="openshift-multus/network-metrics-daemon-h8m9b" Jul 07 19:00:18.404946 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.404475 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4f0c5d5c-147f-426a-965d-8886e5e79984-systemd-units\") pod \"ovnkube-node-zzwbq\" (UID: \"4f0c5d5c-147f-426a-965d-8886e5e79984\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzwbq" Jul 07 19:00:18.404946 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.404500 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a1221769-6ea8-43b7-aa11-e4b2d3699a7b-multus-cni-dir\") pod \"multus-x7d7k\" (UID: \"a1221769-6ea8-43b7-aa11-e4b2d3699a7b\") " pod="openshift-multus/multus-x7d7k" Jul 07 19:00:18.404946 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.404521 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a1221769-6ea8-43b7-aa11-e4b2d3699a7b-hostroot\") pod \"multus-x7d7k\" (UID: \"a1221769-6ea8-43b7-aa11-e4b2d3699a7b\") " pod="openshift-multus/multus-x7d7k" Jul 07 19:00:18.404946 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.404547 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a6abe188-2b1b-427a-ba0c-e93930187c7e-serviceca\") pod \"node-ca-s272w\" (UID: \"a6abe188-2b1b-427a-ba0c-e93930187c7e\") " pod="openshift-image-registry/node-ca-s272w" Jul 07 19:00:18.404946 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.404587 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/63c4ef25-f98e-4224-ad6c-1eb0267d23df-iptables-alerter-script\") pod \"iptables-alerter-8ksxw\" (UID: \"63c4ef25-f98e-4224-ad6c-1eb0267d23df\") " pod="openshift-network-operator/iptables-alerter-8ksxw" Jul 07 19:00:18.404946 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.404627 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/63c4ef25-f98e-4224-ad6c-1eb0267d23df-host-slash\") pod \"iptables-alerter-8ksxw\" (UID: \"63c4ef25-f98e-4224-ad6c-1eb0267d23df\") " pod="openshift-network-operator/iptables-alerter-8ksxw" Jul 07 19:00:18.404946 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.404652 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3953dadf-98ed-4933-a330-1191906b6cff-tmp-dir\") pod \"node-resolver-whw2v\" (UID: \"3953dadf-98ed-4933-a330-1191906b6cff\") " pod="openshift-dns/node-resolver-whw2v" Jul 07 19:00:18.404946 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.404704 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4f0c5d5c-147f-426a-965d-8886e5e79984-run-systemd\") pod \"ovnkube-node-zzwbq\" (UID: \"4f0c5d5c-147f-426a-965d-8886e5e79984\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzwbq" Jul 07 19:00:18.404946 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.404732 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-dsvgs" Jul 07 19:00:18.404946 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.404736 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a1221769-6ea8-43b7-aa11-e4b2d3699a7b-cni-binary-copy\") pod \"multus-x7d7k\" (UID: \"a1221769-6ea8-43b7-aa11-e4b2d3699a7b\") " pod="openshift-multus/multus-x7d7k" Jul 07 19:00:18.404946 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.404764 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7s8ck\" (UniqueName: \"kubernetes.io/projected/63c4ef25-f98e-4224-ad6c-1eb0267d23df-kube-api-access-7s8ck\") pod \"iptables-alerter-8ksxw\" (UID: \"63c4ef25-f98e-4224-ad6c-1eb0267d23df\") " pod="openshift-network-operator/iptables-alerter-8ksxw" Jul 07 19:00:18.406469 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.406452 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-nfmlp\"" Jul 07 19:00:18.406552 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.406506 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 07 19:00:18.406651 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.406637 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 07 19:00:18.407027 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.407010 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qkmgh" Jul 07 19:00:18.408746 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.408727 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 07 19:00:18.408746 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.408732 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 07 19:00:18.408871 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.408727 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-6v5zg\"" Jul 07 19:00:18.409366 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.409067 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 07 19:00:18.409366 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.409122 2563 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 07 19:00:18.409366 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.409189 2563 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-134.ec2.internal" Jul 07 19:00:18.409366 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.409364 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-bjzmq" Jul 07 19:00:18.411875 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.411836 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-142-134.ec2.internal"] Jul 07 19:00:18.412152 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.412132 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-jls8k" Jul 07 19:00:18.412558 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.412532 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 07 19:00:18.412662 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.412581 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-spjkw\"" Jul 07 19:00:18.413005 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.412141 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 07 19:00:18.415661 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.415645 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-hcwcs\"" Jul 07 19:00:18.415775 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.415759 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 07 19:00:18.416094 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.416081 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 07 19:00:18.418329 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.418311 2563 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 07 19:00:18.428898 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.428880 2563 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 07 19:00:18.429042 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.429027 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-134.ec2.internal"] Jul 07 19:00:18.436093 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.436048 2563 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-06 18:55:17 +0000 UTC" deadline="2028-04-06 00:33:40.748050282 +0000 UTC" Jul 07 19:00:18.436093 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.436092 2563 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15317h33m22.311960431s" Jul 07 19:00:18.447139 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.447113 2563 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-kc6cn" Jul 07 19:00:18.459099 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.459079 2563 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-kc6cn" Jul 07 19:00:18.500099 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.500074 2563 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 07 19:00:18.505147 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.505103 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4f0c5d5c-147f-426a-965d-8886e5e79984-host-kubelet\") pod \"ovnkube-node-zzwbq\" (UID: \"4f0c5d5c-147f-426a-965d-8886e5e79984\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzwbq" Jul 07 19:00:18.505271 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.505151 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4f0c5d5c-147f-426a-965d-8886e5e79984-var-lib-openvswitch\") pod \"ovnkube-node-zzwbq\" (UID: \"4f0c5d5c-147f-426a-965d-8886e5e79984\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzwbq" Jul 07 19:00:18.505271 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.505178 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4f0c5d5c-147f-426a-965d-8886e5e79984-run-ovn\") pod \"ovnkube-node-zzwbq\" (UID: \"4f0c5d5c-147f-426a-965d-8886e5e79984\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzwbq" Jul 07 19:00:18.505271 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.505204 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/cbf5b271-d24c-47eb-9971-3bab86804424-etc-sysctl-d\") pod \"tuned-bjzmq\" (UID: \"cbf5b271-d24c-47eb-9971-3bab86804424\") " pod="openshift-cluster-node-tuning-operator/tuned-bjzmq" Jul 07 19:00:18.505271 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.505221 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/794c654b-f09b-4936-b01f-eccd5e7a2a95-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-jls8k\" (UID: \"794c654b-f09b-4936-b01f-eccd5e7a2a95\") " pod="openshift-multus/multus-additional-cni-plugins-jls8k" Jul 07 19:00:18.505271 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.505225 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4f0c5d5c-147f-426a-965d-8886e5e79984-host-kubelet\") pod \"ovnkube-node-zzwbq\" (UID: \"4f0c5d5c-147f-426a-965d-8886e5e79984\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzwbq" Jul 07 19:00:18.505607 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.505279 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4f0c5d5c-147f-426a-965d-8886e5e79984-run-ovn\") pod \"ovnkube-node-zzwbq\" (UID: \"4f0c5d5c-147f-426a-965d-8886e5e79984\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzwbq" Jul 07 19:00:18.505607 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.505286 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4f0c5d5c-147f-426a-965d-8886e5e79984-var-lib-openvswitch\") pod \"ovnkube-node-zzwbq\" (UID: \"4f0c5d5c-147f-426a-965d-8886e5e79984\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzwbq" Jul 07 19:00:18.505607 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.505238 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4f0c5d5c-147f-426a-965d-8886e5e79984-host-run-netns\") pod \"ovnkube-node-zzwbq\" (UID: \"4f0c5d5c-147f-426a-965d-8886e5e79984\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzwbq" Jul 07 19:00:18.505607 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.505322 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4f0c5d5c-147f-426a-965d-8886e5e79984-host-run-netns\") pod \"ovnkube-node-zzwbq\" (UID: \"4f0c5d5c-147f-426a-965d-8886e5e79984\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzwbq" Jul 07 19:00:18.505607 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.505349 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a1221769-6ea8-43b7-aa11-e4b2d3699a7b-host-run-netns\") pod \"multus-x7d7k\" (UID: \"a1221769-6ea8-43b7-aa11-e4b2d3699a7b\") " pod="openshift-multus/multus-x7d7k" Jul 07 19:00:18.505607 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.505373 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a1221769-6ea8-43b7-aa11-e4b2d3699a7b-multus-daemon-config\") pod \"multus-x7d7k\" (UID: \"a1221769-6ea8-43b7-aa11-e4b2d3699a7b\") " pod="openshift-multus/multus-x7d7k" Jul 07 19:00:18.505607 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.505388 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a1221769-6ea8-43b7-aa11-e4b2d3699a7b-etc-kubernetes\") pod \"multus-x7d7k\" (UID: \"a1221769-6ea8-43b7-aa11-e4b2d3699a7b\") " pod="openshift-multus/multus-x7d7k" Jul 07 19:00:18.505607 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.505420 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/cbf5b271-d24c-47eb-9971-3bab86804424-etc-systemd\") pod \"tuned-bjzmq\" (UID: \"cbf5b271-d24c-47eb-9971-3bab86804424\") " pod="openshift-cluster-node-tuning-operator/tuned-bjzmq" Jul 07 19:00:18.505607 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.505441 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/cbf5b271-d24c-47eb-9971-3bab86804424-run\") pod \"tuned-bjzmq\" (UID: \"cbf5b271-d24c-47eb-9971-3bab86804424\") " pod="openshift-cluster-node-tuning-operator/tuned-bjzmq" Jul 07 19:00:18.505607 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.505460 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/59f0c8fc-bc9a-4273-aac2-a8ee23bf6b63-original-pull-secret\") pod \"global-pull-secret-syncer-lk7md\" (UID: \"59f0c8fc-bc9a-4273-aac2-a8ee23bf6b63\") " pod="kube-system/global-pull-secret-syncer-lk7md" Jul 07 19:00:18.505607 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.505488 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4f0c5d5c-147f-426a-965d-8886e5e79984-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-zzwbq\" (UID: \"4f0c5d5c-147f-426a-965d-8886e5e79984\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzwbq" Jul 07 19:00:18.505607 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.505539 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4f0c5d5c-147f-426a-965d-8886e5e79984-ovnkube-script-lib\") pod \"ovnkube-node-zzwbq\" (UID: \"4f0c5d5c-147f-426a-965d-8886e5e79984\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzwbq" Jul 07 19:00:18.505607 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.505565 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a1221769-6ea8-43b7-aa11-e4b2d3699a7b-multus-socket-dir-parent\") pod \"multus-x7d7k\" (UID: \"a1221769-6ea8-43b7-aa11-e4b2d3699a7b\") " pod="openshift-multus/multus-x7d7k" Jul 07 19:00:18.505607 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.505594 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a1221769-6ea8-43b7-aa11-e4b2d3699a7b-host-var-lib-kubelet\") pod \"multus-x7d7k\" (UID: \"a1221769-6ea8-43b7-aa11-e4b2d3699a7b\") " pod="openshift-multus/multus-x7d7k" Jul 07 19:00:18.505607 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.505599 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a1221769-6ea8-43b7-aa11-e4b2d3699a7b-etc-kubernetes\") pod \"multus-x7d7k\" (UID: \"a1221769-6ea8-43b7-aa11-e4b2d3699a7b\") " pod="openshift-multus/multus-x7d7k" Jul 07 19:00:18.506418 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.505636 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a1221769-6ea8-43b7-aa11-e4b2d3699a7b-host-run-netns\") pod \"multus-x7d7k\" (UID: \"a1221769-6ea8-43b7-aa11-e4b2d3699a7b\") " pod="openshift-multus/multus-x7d7k" Jul 07 19:00:18.506418 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.505642 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4f0c5d5c-147f-426a-965d-8886e5e79984-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-zzwbq\" (UID: \"4f0c5d5c-147f-426a-965d-8886e5e79984\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzwbq" Jul 07 19:00:18.506418 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.505769 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a1221769-6ea8-43b7-aa11-e4b2d3699a7b-host-var-lib-kubelet\") pod \"multus-x7d7k\" (UID: \"a1221769-6ea8-43b7-aa11-e4b2d3699a7b\") " pod="openshift-multus/multus-x7d7k" Jul 07 19:00:18.506418 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.505827 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a1221769-6ea8-43b7-aa11-e4b2d3699a7b-host-run-multus-certs\") pod \"multus-x7d7k\" (UID: \"a1221769-6ea8-43b7-aa11-e4b2d3699a7b\") " pod="openshift-multus/multus-x7d7k" Jul 07 19:00:18.506418 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.505859 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdc8d\" (UniqueName: \"kubernetes.io/projected/a6abe188-2b1b-427a-ba0c-e93930187c7e-kube-api-access-fdc8d\") pod \"node-ca-s272w\" (UID: \"a6abe188-2b1b-427a-ba0c-e93930187c7e\") " pod="openshift-image-registry/node-ca-s272w" Jul 07 19:00:18.506418 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.505884 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cbf5b271-d24c-47eb-9971-3bab86804424-host\") pod \"tuned-bjzmq\" (UID: \"cbf5b271-d24c-47eb-9971-3bab86804424\") " pod="openshift-cluster-node-tuning-operator/tuned-bjzmq" Jul 07 19:00:18.506418 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.505913 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a6abe188-2b1b-427a-ba0c-e93930187c7e-serviceca\") pod \"node-ca-s272w\" (UID: \"a6abe188-2b1b-427a-ba0c-e93930187c7e\") " pod="openshift-image-registry/node-ca-s272w" Jul 07 19:00:18.506418 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.505937 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/63c4ef25-f98e-4224-ad6c-1eb0267d23df-host-slash\") pod \"iptables-alerter-8ksxw\" (UID: \"63c4ef25-f98e-4224-ad6c-1eb0267d23df\") " pod="openshift-network-operator/iptables-alerter-8ksxw" Jul 07 19:00:18.506418 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.505961 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a1221769-6ea8-43b7-aa11-e4b2d3699a7b-multus-socket-dir-parent\") pod \"multus-x7d7k\" (UID: \"a1221769-6ea8-43b7-aa11-e4b2d3699a7b\") " pod="openshift-multus/multus-x7d7k" Jul 07 19:00:18.506418 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.506047 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/63c4ef25-f98e-4224-ad6c-1eb0267d23df-host-slash\") pod \"iptables-alerter-8ksxw\" (UID: \"63c4ef25-f98e-4224-ad6c-1eb0267d23df\") " pod="openshift-network-operator/iptables-alerter-8ksxw" Jul 07 19:00:18.506418 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.506046 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4f0c5d5c-147f-426a-965d-8886e5e79984-log-socket\") pod \"ovnkube-node-zzwbq\" (UID: \"4f0c5d5c-147f-426a-965d-8886e5e79984\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzwbq" Jul 07 19:00:18.506418 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.506101 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a1221769-6ea8-43b7-aa11-e4b2d3699a7b-host-run-multus-certs\") pod \"multus-x7d7k\" (UID: \"a1221769-6ea8-43b7-aa11-e4b2d3699a7b\") " pod="openshift-multus/multus-x7d7k" Jul 07 19:00:18.506418 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.505991 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4f0c5d5c-147f-426a-965d-8886e5e79984-log-socket\") pod \"ovnkube-node-zzwbq\" (UID: \"4f0c5d5c-147f-426a-965d-8886e5e79984\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzwbq" Jul 07 19:00:18.506418 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.506138 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4f0c5d5c-147f-426a-965d-8886e5e79984-ovnkube-config\") pod \"ovnkube-node-zzwbq\" (UID: \"4f0c5d5c-147f-426a-965d-8886e5e79984\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzwbq" Jul 07 19:00:18.506418 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.506163 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8rnk\" (UniqueName: \"kubernetes.io/projected/4f0c5d5c-147f-426a-965d-8886e5e79984-kube-api-access-k8rnk\") pod \"ovnkube-node-zzwbq\" (UID: \"4f0c5d5c-147f-426a-965d-8886e5e79984\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzwbq" Jul 07 19:00:18.506418 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.506169 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a1221769-6ea8-43b7-aa11-e4b2d3699a7b-multus-daemon-config\") pod \"multus-x7d7k\" (UID: \"a1221769-6ea8-43b7-aa11-e4b2d3699a7b\") " pod="openshift-multus/multus-x7d7k" Jul 07 19:00:18.506418 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.506225 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cbf5b271-d24c-47eb-9971-3bab86804424-var-lib-kubelet\") pod \"tuned-bjzmq\" (UID: \"cbf5b271-d24c-47eb-9971-3bab86804424\") " pod="openshift-cluster-node-tuning-operator/tuned-bjzmq" Jul 07 19:00:18.507246 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.506255 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/4f25fdbd-3251-4ef6-ab69-e87cc915a469-konnectivity-ca\") pod \"konnectivity-agent-dsvgs\" (UID: \"4f25fdbd-3251-4ef6-ab69-e87cc915a469\") " pod="kube-system/konnectivity-agent-dsvgs" Jul 07 19:00:18.507246 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.506258 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4f0c5d5c-147f-426a-965d-8886e5e79984-ovnkube-script-lib\") pod \"ovnkube-node-zzwbq\" (UID: \"4f0c5d5c-147f-426a-965d-8886e5e79984\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzwbq" Jul 07 19:00:18.507246 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.506281 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4f0c5d5c-147f-426a-965d-8886e5e79984-systemd-units\") pod \"ovnkube-node-zzwbq\" (UID: \"4f0c5d5c-147f-426a-965d-8886e5e79984\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzwbq" Jul 07 19:00:18.507246 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.506308 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a1221769-6ea8-43b7-aa11-e4b2d3699a7b-multus-cni-dir\") pod \"multus-x7d7k\" (UID: \"a1221769-6ea8-43b7-aa11-e4b2d3699a7b\") " pod="openshift-multus/multus-x7d7k" Jul 07 19:00:18.507246 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.506330 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4f0c5d5c-147f-426a-965d-8886e5e79984-systemd-units\") pod \"ovnkube-node-zzwbq\" (UID: \"4f0c5d5c-147f-426a-965d-8886e5e79984\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzwbq" Jul 07 19:00:18.507246 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.506332 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/63c4ef25-f98e-4224-ad6c-1eb0267d23df-iptables-alerter-script\") pod \"iptables-alerter-8ksxw\" (UID: \"63c4ef25-f98e-4224-ad6c-1eb0267d23df\") " pod="openshift-network-operator/iptables-alerter-8ksxw" Jul 07 19:00:18.507246 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.506366 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cbf5b271-d24c-47eb-9971-3bab86804424-tmp\") pod \"tuned-bjzmq\" (UID: \"cbf5b271-d24c-47eb-9971-3bab86804424\") " pod="openshift-cluster-node-tuning-operator/tuned-bjzmq" Jul 07 19:00:18.507246 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.506389 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/794c654b-f09b-4936-b01f-eccd5e7a2a95-cni-binary-copy\") pod \"multus-additional-cni-plugins-jls8k\" (UID: \"794c654b-f09b-4936-b01f-eccd5e7a2a95\") " pod="openshift-multus/multus-additional-cni-plugins-jls8k" Jul 07 19:00:18.507246 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.506416 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/794c654b-f09b-4936-b01f-eccd5e7a2a95-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jls8k\" (UID: \"794c654b-f09b-4936-b01f-eccd5e7a2a95\") " pod="openshift-multus/multus-additional-cni-plugins-jls8k" Jul 07 19:00:18.507246 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.506441 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4f0c5d5c-147f-426a-965d-8886e5e79984-run-systemd\") pod \"ovnkube-node-zzwbq\" (UID: \"4f0c5d5c-147f-426a-965d-8886e5e79984\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzwbq" Jul 07 19:00:18.507246 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.506449 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a6abe188-2b1b-427a-ba0c-e93930187c7e-serviceca\") pod \"node-ca-s272w\" (UID: \"a6abe188-2b1b-427a-ba0c-e93930187c7e\") " pod="openshift-image-registry/node-ca-s272w" Jul 07 19:00:18.507246 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.506461 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d6c3675f-6173-4275-8da8-3af895093d8a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qkmgh\" (UID: \"d6c3675f-6173-4275-8da8-3af895093d8a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qkmgh" Jul 07 19:00:18.507246 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.506482 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/794c654b-f09b-4936-b01f-eccd5e7a2a95-cnibin\") pod \"multus-additional-cni-plugins-jls8k\" (UID: \"794c654b-f09b-4936-b01f-eccd5e7a2a95\") " pod="openshift-multus/multus-additional-cni-plugins-jls8k" Jul 07 19:00:18.507246 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.506504 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/794c654b-f09b-4936-b01f-eccd5e7a2a95-os-release\") pod \"multus-additional-cni-plugins-jls8k\" (UID: \"794c654b-f09b-4936-b01f-eccd5e7a2a95\") " pod="openshift-multus/multus-additional-cni-plugins-jls8k" Jul 07 19:00:18.507246 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.506530 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a1221769-6ea8-43b7-aa11-e4b2d3699a7b-multus-cni-dir\") pod \"multus-x7d7k\" (UID: \"a1221769-6ea8-43b7-aa11-e4b2d3699a7b\") " pod="openshift-multus/multus-x7d7k" Jul 07 19:00:18.507246 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.506535 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4f0c5d5c-147f-426a-965d-8886e5e79984-run-systemd\") pod \"ovnkube-node-zzwbq\" (UID: \"4f0c5d5c-147f-426a-965d-8886e5e79984\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzwbq" Jul 07 19:00:18.507246 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.506531 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/794c654b-f09b-4936-b01f-eccd5e7a2a95-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-jls8k\" (UID: \"794c654b-f09b-4936-b01f-eccd5e7a2a95\") " pod="openshift-multus/multus-additional-cni-plugins-jls8k" Jul 07 19:00:18.508039 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.506571 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4f0c5d5c-147f-426a-965d-8886e5e79984-run-openvswitch\") pod \"ovnkube-node-zzwbq\" (UID: \"4f0c5d5c-147f-426a-965d-8886e5e79984\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzwbq" Jul 07 19:00:18.508039 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.506613 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4f0c5d5c-147f-426a-965d-8886e5e79984-ovnkube-config\") pod \"ovnkube-node-zzwbq\" (UID: \"4f0c5d5c-147f-426a-965d-8886e5e79984\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzwbq" Jul 07 19:00:18.508039 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.506596 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4f0c5d5c-147f-426a-965d-8886e5e79984-host-run-ovn-kubernetes\") pod \"ovnkube-node-zzwbq\" (UID: \"4f0c5d5c-147f-426a-965d-8886e5e79984\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzwbq" Jul 07 19:00:18.508039 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.506657 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4f0c5d5c-147f-426a-965d-8886e5e79984-host-run-ovn-kubernetes\") pod \"ovnkube-node-zzwbq\" (UID: \"4f0c5d5c-147f-426a-965d-8886e5e79984\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzwbq" Jul 07 19:00:18.508039 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.506660 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a1221769-6ea8-43b7-aa11-e4b2d3699a7b-multus-conf-dir\") pod \"multus-x7d7k\" (UID: \"a1221769-6ea8-43b7-aa11-e4b2d3699a7b\") " pod="openshift-multus/multus-x7d7k" Jul 07 19:00:18.508039 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.506665 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4f0c5d5c-147f-426a-965d-8886e5e79984-run-openvswitch\") pod \"ovnkube-node-zzwbq\" (UID: \"4f0c5d5c-147f-426a-965d-8886e5e79984\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzwbq" Jul 07 19:00:18.508039 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.506697 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/59f0c8fc-bc9a-4273-aac2-a8ee23bf6b63-dbus\") pod \"global-pull-secret-syncer-lk7md\" (UID: \"59f0c8fc-bc9a-4273-aac2-a8ee23bf6b63\") " pod="kube-system/global-pull-secret-syncer-lk7md" Jul 07 19:00:18.508039 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.506740 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a1221769-6ea8-43b7-aa11-e4b2d3699a7b-multus-conf-dir\") pod \"multus-x7d7k\" (UID: \"a1221769-6ea8-43b7-aa11-e4b2d3699a7b\") " pod="openshift-multus/multus-x7d7k" Jul 07 19:00:18.508039 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.506753 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cbf5b271-d24c-47eb-9971-3bab86804424-sys\") pod \"tuned-bjzmq\" (UID: \"cbf5b271-d24c-47eb-9971-3bab86804424\") " pod="openshift-cluster-node-tuning-operator/tuned-bjzmq" Jul 07 19:00:18.508039 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.506788 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/63c4ef25-f98e-4224-ad6c-1eb0267d23df-iptables-alerter-script\") pod \"iptables-alerter-8ksxw\" (UID: \"63c4ef25-f98e-4224-ad6c-1eb0267d23df\") " pod="openshift-network-operator/iptables-alerter-8ksxw" Jul 07 19:00:18.508039 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.506796 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j28vs\" (UniqueName: \"kubernetes.io/projected/3953dadf-98ed-4933-a330-1191906b6cff-kube-api-access-j28vs\") pod \"node-resolver-whw2v\" (UID: \"3953dadf-98ed-4933-a330-1191906b6cff\") " pod="openshift-dns/node-resolver-whw2v" Jul 07 19:00:18.508039 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.506829 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4f0c5d5c-147f-426a-965d-8886e5e79984-etc-openvswitch\") pod \"ovnkube-node-zzwbq\" (UID: \"4f0c5d5c-147f-426a-965d-8886e5e79984\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzwbq" Jul 07 19:00:18.508039 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.506855 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4f0c5d5c-147f-426a-965d-8886e5e79984-etc-openvswitch\") pod \"ovnkube-node-zzwbq\" (UID: \"4f0c5d5c-147f-426a-965d-8886e5e79984\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzwbq" Jul 07 19:00:18.508039 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.506873 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4f0c5d5c-147f-426a-965d-8886e5e79984-host-cni-bin\") pod \"ovnkube-node-zzwbq\" (UID: \"4f0c5d5c-147f-426a-965d-8886e5e79984\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzwbq" Jul 07 19:00:18.508039 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.506906 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/cbf5b271-d24c-47eb-9971-3bab86804424-lib-modules\") pod \"tuned-bjzmq\" (UID: \"cbf5b271-d24c-47eb-9971-3bab86804424\") " pod="openshift-cluster-node-tuning-operator/tuned-bjzmq" Jul 07 19:00:18.508039 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.506931 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jfqd4\" (UniqueName: \"kubernetes.io/projected/794c654b-f09b-4936-b01f-eccd5e7a2a95-kube-api-access-jfqd4\") pod \"multus-additional-cni-plugins-jls8k\" (UID: \"794c654b-f09b-4936-b01f-eccd5e7a2a95\") " pod="openshift-multus/multus-additional-cni-plugins-jls8k" Jul 07 19:00:18.508039 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.506908 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4f0c5d5c-147f-426a-965d-8886e5e79984-host-cni-bin\") pod \"ovnkube-node-zzwbq\" (UID: \"4f0c5d5c-147f-426a-965d-8886e5e79984\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzwbq" Jul 07 19:00:18.508855 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.507030 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgqkm\" (UniqueName: \"kubernetes.io/projected/1d84b355-0ac7-4d14-83b6-7c18077e9319-kube-api-access-bgqkm\") pod \"network-check-target-d8t2d\" (UID: \"1d84b355-0ac7-4d14-83b6-7c18077e9319\") " pod="openshift-network-diagnostics/network-check-target-d8t2d" Jul 07 19:00:18.508855 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.507058 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a6abe188-2b1b-427a-ba0c-e93930187c7e-host\") pod \"node-ca-s272w\" (UID: \"a6abe188-2b1b-427a-ba0c-e93930187c7e\") " pod="openshift-image-registry/node-ca-s272w" Jul 07 19:00:18.508855 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.507083 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/cbf5b271-d24c-47eb-9971-3bab86804424-etc-sysconfig\") pod \"tuned-bjzmq\" (UID: \"cbf5b271-d24c-47eb-9971-3bab86804424\") " pod="openshift-cluster-node-tuning-operator/tuned-bjzmq" Jul 07 19:00:18.508855 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.507106 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/cbf5b271-d24c-47eb-9971-3bab86804424-etc-tuned\") pod \"tuned-bjzmq\" (UID: \"cbf5b271-d24c-47eb-9971-3bab86804424\") " pod="openshift-cluster-node-tuning-operator/tuned-bjzmq" Jul 07 19:00:18.508855 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.507138 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/4f25fdbd-3251-4ef6-ab69-e87cc915a469-agent-certs\") pod \"konnectivity-agent-dsvgs\" (UID: \"4f25fdbd-3251-4ef6-ab69-e87cc915a469\") " pod="kube-system/konnectivity-agent-dsvgs" Jul 07 19:00:18.508855 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.507140 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a6abe188-2b1b-427a-ba0c-e93930187c7e-host\") pod \"node-ca-s272w\" (UID: \"a6abe188-2b1b-427a-ba0c-e93930187c7e\") " pod="openshift-image-registry/node-ca-s272w" Jul 07 19:00:18.508855 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.507170 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/396810c7-8961-4755-b0f6-02a9d7e02ae3-metrics-certs\") pod \"network-metrics-daemon-h8m9b\" (UID: \"396810c7-8961-4755-b0f6-02a9d7e02ae3\") " pod="openshift-multus/network-metrics-daemon-h8m9b" Jul 07 19:00:18.508855 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.507195 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4f0c5d5c-147f-426a-965d-8886e5e79984-node-log\") pod \"ovnkube-node-zzwbq\" (UID: \"4f0c5d5c-147f-426a-965d-8886e5e79984\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzwbq" Jul 07 19:00:18.508855 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.507213 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4f0c5d5c-147f-426a-965d-8886e5e79984-host-cni-netd\") pod \"ovnkube-node-zzwbq\" (UID: \"4f0c5d5c-147f-426a-965d-8886e5e79984\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzwbq" Jul 07 19:00:18.508855 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.507258 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4f0c5d5c-147f-426a-965d-8886e5e79984-host-cni-netd\") pod \"ovnkube-node-zzwbq\" (UID: \"4f0c5d5c-147f-426a-965d-8886e5e79984\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzwbq" Jul 07 19:00:18.508855 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:00:18.507302 2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 19:00:18.508855 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.507346 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a1221769-6ea8-43b7-aa11-e4b2d3699a7b-cnibin\") pod \"multus-x7d7k\" (UID: \"a1221769-6ea8-43b7-aa11-e4b2d3699a7b\") " pod="openshift-multus/multus-x7d7k" Jul 07 19:00:18.508855 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:00:18.507381 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/396810c7-8961-4755-b0f6-02a9d7e02ae3-metrics-certs podName:396810c7-8961-4755-b0f6-02a9d7e02ae3 nodeName:}" failed. No retries permitted until 2026-07-07 19:00:19.00735218 +0000 UTC m=+2.023356354 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/396810c7-8961-4755-b0f6-02a9d7e02ae3-metrics-certs") pod "network-metrics-daemon-h8m9b" (UID: "396810c7-8961-4755-b0f6-02a9d7e02ae3") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 19:00:18.508855 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.507405 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a1221769-6ea8-43b7-aa11-e4b2d3699a7b-host-var-lib-cni-bin\") pod \"multus-x7d7k\" (UID: \"a1221769-6ea8-43b7-aa11-e4b2d3699a7b\") " pod="openshift-multus/multus-x7d7k" Jul 07 19:00:18.508855 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.507430 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5k55\" (UniqueName: \"kubernetes.io/projected/a1221769-6ea8-43b7-aa11-e4b2d3699a7b-kube-api-access-v5k55\") pod \"multus-x7d7k\" (UID: \"a1221769-6ea8-43b7-aa11-e4b2d3699a7b\") " pod="openshift-multus/multus-x7d7k" Jul 07 19:00:18.508855 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.507458 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cbf5b271-d24c-47eb-9971-3bab86804424-etc-kubernetes\") pod \"tuned-bjzmq\" (UID: \"cbf5b271-d24c-47eb-9971-3bab86804424\") " pod="openshift-cluster-node-tuning-operator/tuned-bjzmq" Jul 07 19:00:18.508855 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.507481 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/cbf5b271-d24c-47eb-9971-3bab86804424-etc-modprobe-d\") pod \"tuned-bjzmq\" (UID: \"cbf5b271-d24c-47eb-9971-3bab86804424\") " pod="openshift-cluster-node-tuning-operator/tuned-bjzmq" Jul 07 19:00:18.509654 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.507501 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4f0c5d5c-147f-426a-965d-8886e5e79984-env-overrides\") pod \"ovnkube-node-zzwbq\" (UID: \"4f0c5d5c-147f-426a-965d-8886e5e79984\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzwbq" Jul 07 19:00:18.509654 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.507520 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a1221769-6ea8-43b7-aa11-e4b2d3699a7b-os-release\") pod \"multus-x7d7k\" (UID: \"a1221769-6ea8-43b7-aa11-e4b2d3699a7b\") " pod="openshift-multus/multus-x7d7k" Jul 07 19:00:18.509654 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.507540 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a1221769-6ea8-43b7-aa11-e4b2d3699a7b-host-run-k8s-cni-cncf-io\") pod \"multus-x7d7k\" (UID: \"a1221769-6ea8-43b7-aa11-e4b2d3699a7b\") " pod="openshift-multus/multus-x7d7k" Jul 07 19:00:18.509654 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.507565 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a1221769-6ea8-43b7-aa11-e4b2d3699a7b-host-var-lib-cni-multus\") pod \"multus-x7d7k\" (UID: \"a1221769-6ea8-43b7-aa11-e4b2d3699a7b\") " pod="openshift-multus/multus-x7d7k" Jul 07 19:00:18.509654 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.507587 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xv8xh\" (UniqueName: \"kubernetes.io/projected/cbf5b271-d24c-47eb-9971-3bab86804424-kube-api-access-xv8xh\") pod \"tuned-bjzmq\" (UID: \"cbf5b271-d24c-47eb-9971-3bab86804424\") " pod="openshift-cluster-node-tuning-operator/tuned-bjzmq" Jul 07 19:00:18.509654 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.507608 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d6c3675f-6173-4275-8da8-3af895093d8a-etc-selinux\") pod \"aws-ebs-csi-driver-node-qkmgh\" (UID: \"d6c3675f-6173-4275-8da8-3af895093d8a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qkmgh" Jul 07 19:00:18.509654 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.507631 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mqwpr\" (UniqueName: \"kubernetes.io/projected/396810c7-8961-4755-b0f6-02a9d7e02ae3-kube-api-access-mqwpr\") pod \"network-metrics-daemon-h8m9b\" (UID: \"396810c7-8961-4755-b0f6-02a9d7e02ae3\") " pod="openshift-multus/network-metrics-daemon-h8m9b" Jul 07 19:00:18.509654 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.507637 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a1221769-6ea8-43b7-aa11-e4b2d3699a7b-host-var-lib-cni-bin\") pod \"multus-x7d7k\" (UID: \"a1221769-6ea8-43b7-aa11-e4b2d3699a7b\") " pod="openshift-multus/multus-x7d7k" Jul 07 19:00:18.509654 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.507653 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a1221769-6ea8-43b7-aa11-e4b2d3699a7b-hostroot\") pod \"multus-x7d7k\" (UID: \"a1221769-6ea8-43b7-aa11-e4b2d3699a7b\") " pod="openshift-multus/multus-x7d7k" Jul 07 19:00:18.509654 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.507675 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qb4q8\" (UniqueName: \"kubernetes.io/projected/d6c3675f-6173-4275-8da8-3af895093d8a-kube-api-access-qb4q8\") pod \"aws-ebs-csi-driver-node-qkmgh\" (UID: \"d6c3675f-6173-4275-8da8-3af895093d8a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qkmgh" Jul 07 19:00:18.509654 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.507697 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/794c654b-f09b-4936-b01f-eccd5e7a2a95-system-cni-dir\") pod \"multus-additional-cni-plugins-jls8k\" (UID: \"794c654b-f09b-4936-b01f-eccd5e7a2a95\") " pod="openshift-multus/multus-additional-cni-plugins-jls8k" Jul 07 19:00:18.509654 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.507719 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3953dadf-98ed-4933-a330-1191906b6cff-tmp-dir\") pod \"node-resolver-whw2v\" (UID: \"3953dadf-98ed-4933-a330-1191906b6cff\") " pod="openshift-dns/node-resolver-whw2v" Jul 07 19:00:18.509654 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.507739 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a1221769-6ea8-43b7-aa11-e4b2d3699a7b-cni-binary-copy\") pod \"multus-x7d7k\" (UID: \"a1221769-6ea8-43b7-aa11-e4b2d3699a7b\") " pod="openshift-multus/multus-x7d7k" Jul 07 19:00:18.509654 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.507760 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7s8ck\" (UniqueName: \"kubernetes.io/projected/63c4ef25-f98e-4224-ad6c-1eb0267d23df-kube-api-access-7s8ck\") pod \"iptables-alerter-8ksxw\" (UID: \"63c4ef25-f98e-4224-ad6c-1eb0267d23df\") " pod="openshift-network-operator/iptables-alerter-8ksxw" Jul 07 19:00:18.509654 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.507411 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a1221769-6ea8-43b7-aa11-e4b2d3699a7b-cnibin\") pod \"multus-x7d7k\" (UID: \"a1221769-6ea8-43b7-aa11-e4b2d3699a7b\") " pod="openshift-multus/multus-x7d7k" Jul 07 19:00:18.509654 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.507781 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d6c3675f-6173-4275-8da8-3af895093d8a-device-dir\") pod \"aws-ebs-csi-driver-node-qkmgh\" (UID: \"d6c3675f-6173-4275-8da8-3af895093d8a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qkmgh" Jul 07 19:00:18.509654 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.507808 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4f0c5d5c-147f-426a-965d-8886e5e79984-host-slash\") pod \"ovnkube-node-zzwbq\" (UID: \"4f0c5d5c-147f-426a-965d-8886e5e79984\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzwbq" Jul 07 19:00:18.510397 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.507829 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4f0c5d5c-147f-426a-965d-8886e5e79984-ovn-node-metrics-cert\") pod \"ovnkube-node-zzwbq\" (UID: \"4f0c5d5c-147f-426a-965d-8886e5e79984\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzwbq" Jul 07 19:00:18.510397 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.507844 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a1221769-6ea8-43b7-aa11-e4b2d3699a7b-system-cni-dir\") pod \"multus-x7d7k\" (UID: \"a1221769-6ea8-43b7-aa11-e4b2d3699a7b\") " pod="openshift-multus/multus-x7d7k" Jul 07 19:00:18.510397 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.507881 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/59f0c8fc-bc9a-4273-aac2-a8ee23bf6b63-kubelet-config\") pod \"global-pull-secret-syncer-lk7md\" (UID: \"59f0c8fc-bc9a-4273-aac2-a8ee23bf6b63\") " pod="kube-system/global-pull-secret-syncer-lk7md" Jul 07 19:00:18.510397 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.507897 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/cbf5b271-d24c-47eb-9971-3bab86804424-etc-sysctl-conf\") pod \"tuned-bjzmq\" (UID: \"cbf5b271-d24c-47eb-9971-3bab86804424\") " pod="openshift-cluster-node-tuning-operator/tuned-bjzmq" Jul 07 19:00:18.510397 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.507924 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/3953dadf-98ed-4933-a330-1191906b6cff-hosts-file\") pod \"node-resolver-whw2v\" (UID: \"3953dadf-98ed-4933-a330-1191906b6cff\") " pod="openshift-dns/node-resolver-whw2v" Jul 07 19:00:18.510397 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.507945 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d6c3675f-6173-4275-8da8-3af895093d8a-socket-dir\") pod \"aws-ebs-csi-driver-node-qkmgh\" (UID: \"d6c3675f-6173-4275-8da8-3af895093d8a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qkmgh" Jul 07 19:00:18.510397 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.507979 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d6c3675f-6173-4275-8da8-3af895093d8a-registration-dir\") pod \"aws-ebs-csi-driver-node-qkmgh\" (UID: \"d6c3675f-6173-4275-8da8-3af895093d8a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qkmgh" Jul 07 19:00:18.510397 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.507994 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d6c3675f-6173-4275-8da8-3af895093d8a-sys-fs\") pod \"aws-ebs-csi-driver-node-qkmgh\" (UID: \"d6c3675f-6173-4275-8da8-3af895093d8a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qkmgh" Jul 07 19:00:18.510397 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.508073 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4f0c5d5c-147f-426a-965d-8886e5e79984-host-slash\") pod \"ovnkube-node-zzwbq\" (UID: \"4f0c5d5c-147f-426a-965d-8886e5e79984\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzwbq" Jul 07 19:00:18.510397 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.508110 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a1221769-6ea8-43b7-aa11-e4b2d3699a7b-hostroot\") pod \"multus-x7d7k\" (UID: \"a1221769-6ea8-43b7-aa11-e4b2d3699a7b\") " pod="openshift-multus/multus-x7d7k" Jul 07 19:00:18.510397 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.508133 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4f0c5d5c-147f-426a-965d-8886e5e79984-env-overrides\") pod \"ovnkube-node-zzwbq\" (UID: \"4f0c5d5c-147f-426a-965d-8886e5e79984\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzwbq" Jul 07 19:00:18.510397 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.508209 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a1221769-6ea8-43b7-aa11-e4b2d3699a7b-os-release\") pod \"multus-x7d7k\" (UID: \"a1221769-6ea8-43b7-aa11-e4b2d3699a7b\") " pod="openshift-multus/multus-x7d7k" Jul 07 19:00:18.510397 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.508259 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a1221769-6ea8-43b7-aa11-e4b2d3699a7b-host-run-k8s-cni-cncf-io\") pod \"multus-x7d7k\" (UID: \"a1221769-6ea8-43b7-aa11-e4b2d3699a7b\") " pod="openshift-multus/multus-x7d7k" Jul 07 19:00:18.510397 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.508310 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a1221769-6ea8-43b7-aa11-e4b2d3699a7b-host-var-lib-cni-multus\") pod \"multus-x7d7k\" (UID: \"a1221769-6ea8-43b7-aa11-e4b2d3699a7b\") " pod="openshift-multus/multus-x7d7k" Jul 07 19:00:18.510397 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.508396 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a1221769-6ea8-43b7-aa11-e4b2d3699a7b-system-cni-dir\") pod \"multus-x7d7k\" (UID: \"a1221769-6ea8-43b7-aa11-e4b2d3699a7b\") " pod="openshift-multus/multus-x7d7k" Jul 07 19:00:18.510397 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.508375 2563 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 07 19:00:18.510397 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.508441 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/3953dadf-98ed-4933-a330-1191906b6cff-hosts-file\") pod \"node-resolver-whw2v\" (UID: \"3953dadf-98ed-4933-a330-1191906b6cff\") " pod="openshift-dns/node-resolver-whw2v" Jul 07 19:00:18.510397 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.508483 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4f0c5d5c-147f-426a-965d-8886e5e79984-node-log\") pod \"ovnkube-node-zzwbq\" (UID: \"4f0c5d5c-147f-426a-965d-8886e5e79984\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzwbq" Jul 07 19:00:18.511338 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.508567 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3953dadf-98ed-4933-a330-1191906b6cff-tmp-dir\") pod \"node-resolver-whw2v\" (UID: \"3953dadf-98ed-4933-a330-1191906b6cff\") " pod="openshift-dns/node-resolver-whw2v" Jul 07 19:00:18.511338 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.508699 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a1221769-6ea8-43b7-aa11-e4b2d3699a7b-cni-binary-copy\") pod \"multus-x7d7k\" (UID: \"a1221769-6ea8-43b7-aa11-e4b2d3699a7b\") " pod="openshift-multus/multus-x7d7k" Jul 07 19:00:18.511633 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.511614 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4f0c5d5c-147f-426a-965d-8886e5e79984-ovn-node-metrics-cert\") pod \"ovnkube-node-zzwbq\" (UID: \"4f0c5d5c-147f-426a-965d-8886e5e79984\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzwbq" Jul 07 19:00:18.514514 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.514485 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8rnk\" (UniqueName: \"kubernetes.io/projected/4f0c5d5c-147f-426a-965d-8886e5e79984-kube-api-access-k8rnk\") pod \"ovnkube-node-zzwbq\" (UID: \"4f0c5d5c-147f-426a-965d-8886e5e79984\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzwbq" Jul 07 19:00:18.515419 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.515399 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdc8d\" (UniqueName: \"kubernetes.io/projected/a6abe188-2b1b-427a-ba0c-e93930187c7e-kube-api-access-fdc8d\") pod \"node-ca-s272w\" (UID: \"a6abe188-2b1b-427a-ba0c-e93930187c7e\") " pod="openshift-image-registry/node-ca-s272w" Jul 07 19:00:18.515635 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:00:18.515617 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 07 19:00:18.515635 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:00:18.515636 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 07 19:00:18.515766 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:00:18.515645 2563 projected.go:196] Error preparing data for projected volume kube-api-access-bgqkm for pod openshift-network-diagnostics/network-check-target-d8t2d: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 19:00:18.515766 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:00:18.515721 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1d84b355-0ac7-4d14-83b6-7c18077e9319-kube-api-access-bgqkm podName:1d84b355-0ac7-4d14-83b6-7c18077e9319 nodeName:}" failed. No retries permitted until 2026-07-07 19:00:19.015708175 +0000 UTC m=+2.031712333 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-bgqkm" (UniqueName: "kubernetes.io/projected/1d84b355-0ac7-4d14-83b6-7c18077e9319-kube-api-access-bgqkm") pod "network-check-target-d8t2d" (UID: "1d84b355-0ac7-4d14-83b6-7c18077e9319") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 19:00:18.518088 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.518065 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v5k55\" (UniqueName: \"kubernetes.io/projected/a1221769-6ea8-43b7-aa11-e4b2d3699a7b-kube-api-access-v5k55\") pod \"multus-x7d7k\" (UID: \"a1221769-6ea8-43b7-aa11-e4b2d3699a7b\") " pod="openshift-multus/multus-x7d7k" Jul 07 19:00:18.518532 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.518508 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j28vs\" (UniqueName: \"kubernetes.io/projected/3953dadf-98ed-4933-a330-1191906b6cff-kube-api-access-j28vs\") pod \"node-resolver-whw2v\" (UID: \"3953dadf-98ed-4933-a330-1191906b6cff\") " pod="openshift-dns/node-resolver-whw2v" Jul 07 19:00:18.518532 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.518524 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7s8ck\" (UniqueName: \"kubernetes.io/projected/63c4ef25-f98e-4224-ad6c-1eb0267d23df-kube-api-access-7s8ck\") pod \"iptables-alerter-8ksxw\" (UID: \"63c4ef25-f98e-4224-ad6c-1eb0267d23df\") " pod="openshift-network-operator/iptables-alerter-8ksxw" Jul 07 19:00:18.518919 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.518902 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mqwpr\" (UniqueName: \"kubernetes.io/projected/396810c7-8961-4755-b0f6-02a9d7e02ae3-kube-api-access-mqwpr\") pod \"network-metrics-daemon-h8m9b\" (UID: \"396810c7-8961-4755-b0f6-02a9d7e02ae3\") " pod="openshift-multus/network-metrics-daemon-h8m9b" Jul 07 19:00:18.560486 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.560453 2563 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 07 19:00:18.608070 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:18.608028 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod067754c774fbc99720cce604846fd8f3.slice/crio-121aaaed5cfe6db86cef7a6d01166bb30cd083ad7f98dfa26cb024eb1e55a42c WatchSource:0}: Error finding container 121aaaed5cfe6db86cef7a6d01166bb30cd083ad7f98dfa26cb024eb1e55a42c: Status 404 returned error can't find the container with id 121aaaed5cfe6db86cef7a6d01166bb30cd083ad7f98dfa26cb024eb1e55a42c Jul 07 19:00:18.608278 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.608260 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/59f0c8fc-bc9a-4273-aac2-a8ee23bf6b63-kubelet-config\") pod \"global-pull-secret-syncer-lk7md\" (UID: \"59f0c8fc-bc9a-4273-aac2-a8ee23bf6b63\") " pod="kube-system/global-pull-secret-syncer-lk7md" Jul 07 19:00:18.608352 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.608293 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/cbf5b271-d24c-47eb-9971-3bab86804424-etc-sysctl-conf\") pod \"tuned-bjzmq\" (UID: \"cbf5b271-d24c-47eb-9971-3bab86804424\") " pod="openshift-cluster-node-tuning-operator/tuned-bjzmq" Jul 07 19:00:18.608352 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.608318 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d6c3675f-6173-4275-8da8-3af895093d8a-socket-dir\") pod \"aws-ebs-csi-driver-node-qkmgh\" (UID: \"d6c3675f-6173-4275-8da8-3af895093d8a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qkmgh" Jul 07 19:00:18.608352 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.608342 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d6c3675f-6173-4275-8da8-3af895093d8a-registration-dir\") pod \"aws-ebs-csi-driver-node-qkmgh\" (UID: \"d6c3675f-6173-4275-8da8-3af895093d8a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qkmgh" Jul 07 19:00:18.608505 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.608398 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/59f0c8fc-bc9a-4273-aac2-a8ee23bf6b63-kubelet-config\") pod \"global-pull-secret-syncer-lk7md\" (UID: \"59f0c8fc-bc9a-4273-aac2-a8ee23bf6b63\") " pod="kube-system/global-pull-secret-syncer-lk7md" Jul 07 19:00:18.608505 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.608424 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d6c3675f-6173-4275-8da8-3af895093d8a-sys-fs\") pod \"aws-ebs-csi-driver-node-qkmgh\" (UID: \"d6c3675f-6173-4275-8da8-3af895093d8a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qkmgh" Jul 07 19:00:18.608505 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.608435 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/cbf5b271-d24c-47eb-9971-3bab86804424-etc-sysctl-conf\") pod \"tuned-bjzmq\" (UID: \"cbf5b271-d24c-47eb-9971-3bab86804424\") " pod="openshift-cluster-node-tuning-operator/tuned-bjzmq" Jul 07 19:00:18.608505 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.608452 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/cbf5b271-d24c-47eb-9971-3bab86804424-etc-sysctl-d\") pod \"tuned-bjzmq\" (UID: \"cbf5b271-d24c-47eb-9971-3bab86804424\") " pod="openshift-cluster-node-tuning-operator/tuned-bjzmq" Jul 07 19:00:18.608505 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.608405 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d6c3675f-6173-4275-8da8-3af895093d8a-registration-dir\") pod \"aws-ebs-csi-driver-node-qkmgh\" (UID: \"d6c3675f-6173-4275-8da8-3af895093d8a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qkmgh" Jul 07 19:00:18.608505 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.608455 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d6c3675f-6173-4275-8da8-3af895093d8a-socket-dir\") pod \"aws-ebs-csi-driver-node-qkmgh\" (UID: \"d6c3675f-6173-4275-8da8-3af895093d8a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qkmgh" Jul 07 19:00:18.608505 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.608494 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d6c3675f-6173-4275-8da8-3af895093d8a-sys-fs\") pod \"aws-ebs-csi-driver-node-qkmgh\" (UID: \"d6c3675f-6173-4275-8da8-3af895093d8a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qkmgh" Jul 07 19:00:18.608505 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.608501 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/794c654b-f09b-4936-b01f-eccd5e7a2a95-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-jls8k\" (UID: \"794c654b-f09b-4936-b01f-eccd5e7a2a95\") " pod="openshift-multus/multus-additional-cni-plugins-jls8k" Jul 07 19:00:18.608828 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.608530 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/cbf5b271-d24c-47eb-9971-3bab86804424-etc-systemd\") pod \"tuned-bjzmq\" (UID: \"cbf5b271-d24c-47eb-9971-3bab86804424\") " pod="openshift-cluster-node-tuning-operator/tuned-bjzmq" Jul 07 19:00:18.608828 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.608548 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/cbf5b271-d24c-47eb-9971-3bab86804424-run\") pod \"tuned-bjzmq\" (UID: \"cbf5b271-d24c-47eb-9971-3bab86804424\") " pod="openshift-cluster-node-tuning-operator/tuned-bjzmq" Jul 07 19:00:18.608828 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.608571 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/59f0c8fc-bc9a-4273-aac2-a8ee23bf6b63-original-pull-secret\") pod \"global-pull-secret-syncer-lk7md\" (UID: \"59f0c8fc-bc9a-4273-aac2-a8ee23bf6b63\") " pod="kube-system/global-pull-secret-syncer-lk7md" Jul 07 19:00:18.608828 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.608583 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/cbf5b271-d24c-47eb-9971-3bab86804424-etc-sysctl-d\") pod \"tuned-bjzmq\" (UID: \"cbf5b271-d24c-47eb-9971-3bab86804424\") " pod="openshift-cluster-node-tuning-operator/tuned-bjzmq" Jul 07 19:00:18.608828 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.608595 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/cbf5b271-d24c-47eb-9971-3bab86804424-etc-systemd\") pod \"tuned-bjzmq\" (UID: \"cbf5b271-d24c-47eb-9971-3bab86804424\") " pod="openshift-cluster-node-tuning-operator/tuned-bjzmq" Jul 07 19:00:18.608828 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.608604 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/cbf5b271-d24c-47eb-9971-3bab86804424-run\") pod \"tuned-bjzmq\" (UID: \"cbf5b271-d24c-47eb-9971-3bab86804424\") " pod="openshift-cluster-node-tuning-operator/tuned-bjzmq" Jul 07 19:00:18.608828 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.608599 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cbf5b271-d24c-47eb-9971-3bab86804424-host\") pod \"tuned-bjzmq\" (UID: \"cbf5b271-d24c-47eb-9971-3bab86804424\") " pod="openshift-cluster-node-tuning-operator/tuned-bjzmq" Jul 07 19:00:18.608828 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.608637 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cbf5b271-d24c-47eb-9971-3bab86804424-host\") pod \"tuned-bjzmq\" (UID: \"cbf5b271-d24c-47eb-9971-3bab86804424\") " pod="openshift-cluster-node-tuning-operator/tuned-bjzmq" Jul 07 19:00:18.608828 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.608646 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cbf5b271-d24c-47eb-9971-3bab86804424-var-lib-kubelet\") pod \"tuned-bjzmq\" (UID: \"cbf5b271-d24c-47eb-9971-3bab86804424\") " pod="openshift-cluster-node-tuning-operator/tuned-bjzmq" Jul 07 19:00:18.608828 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.608672 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/4f25fdbd-3251-4ef6-ab69-e87cc915a469-konnectivity-ca\") pod \"konnectivity-agent-dsvgs\" (UID: \"4f25fdbd-3251-4ef6-ab69-e87cc915a469\") " pod="kube-system/konnectivity-agent-dsvgs" Jul 07 19:00:18.608828 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:00:18.608683 2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 07 19:00:18.608828 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.608701 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cbf5b271-d24c-47eb-9971-3bab86804424-tmp\") pod \"tuned-bjzmq\" (UID: \"cbf5b271-d24c-47eb-9971-3bab86804424\") " pod="openshift-cluster-node-tuning-operator/tuned-bjzmq" Jul 07 19:00:18.608828 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.608724 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/794c654b-f09b-4936-b01f-eccd5e7a2a95-cni-binary-copy\") pod \"multus-additional-cni-plugins-jls8k\" (UID: \"794c654b-f09b-4936-b01f-eccd5e7a2a95\") " pod="openshift-multus/multus-additional-cni-plugins-jls8k" Jul 07 19:00:18.608828 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:00:18.608746 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/59f0c8fc-bc9a-4273-aac2-a8ee23bf6b63-original-pull-secret podName:59f0c8fc-bc9a-4273-aac2-a8ee23bf6b63 nodeName:}" failed. No retries permitted until 2026-07-07 19:00:19.108729476 +0000 UTC m=+2.124733637 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/59f0c8fc-bc9a-4273-aac2-a8ee23bf6b63-original-pull-secret") pod "global-pull-secret-syncer-lk7md" (UID: "59f0c8fc-bc9a-4273-aac2-a8ee23bf6b63") : object "kube-system"/"original-pull-secret" not registered Jul 07 19:00:18.608828 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.608683 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cbf5b271-d24c-47eb-9971-3bab86804424-var-lib-kubelet\") pod \"tuned-bjzmq\" (UID: \"cbf5b271-d24c-47eb-9971-3bab86804424\") " pod="openshift-cluster-node-tuning-operator/tuned-bjzmq" Jul 07 19:00:18.608828 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.608773 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/794c654b-f09b-4936-b01f-eccd5e7a2a95-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jls8k\" (UID: \"794c654b-f09b-4936-b01f-eccd5e7a2a95\") " pod="openshift-multus/multus-additional-cni-plugins-jls8k" Jul 07 19:00:18.608828 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.608802 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d6c3675f-6173-4275-8da8-3af895093d8a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qkmgh\" (UID: \"d6c3675f-6173-4275-8da8-3af895093d8a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qkmgh" Jul 07 19:00:18.609736 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.608826 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/794c654b-f09b-4936-b01f-eccd5e7a2a95-cnibin\") pod \"multus-additional-cni-plugins-jls8k\" (UID: \"794c654b-f09b-4936-b01f-eccd5e7a2a95\") " pod="openshift-multus/multus-additional-cni-plugins-jls8k" Jul 07 19:00:18.609736 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.608847 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/794c654b-f09b-4936-b01f-eccd5e7a2a95-os-release\") pod \"multus-additional-cni-plugins-jls8k\" (UID: \"794c654b-f09b-4936-b01f-eccd5e7a2a95\") " pod="openshift-multus/multus-additional-cni-plugins-jls8k" Jul 07 19:00:18.609736 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.608872 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/794c654b-f09b-4936-b01f-eccd5e7a2a95-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-jls8k\" (UID: \"794c654b-f09b-4936-b01f-eccd5e7a2a95\") " pod="openshift-multus/multus-additional-cni-plugins-jls8k" Jul 07 19:00:18.609736 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.608897 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/59f0c8fc-bc9a-4273-aac2-a8ee23bf6b63-dbus\") pod \"global-pull-secret-syncer-lk7md\" (UID: \"59f0c8fc-bc9a-4273-aac2-a8ee23bf6b63\") " pod="kube-system/global-pull-secret-syncer-lk7md" Jul 07 19:00:18.609736 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.608920 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cbf5b271-d24c-47eb-9971-3bab86804424-sys\") pod \"tuned-bjzmq\" (UID: \"cbf5b271-d24c-47eb-9971-3bab86804424\") " pod="openshift-cluster-node-tuning-operator/tuned-bjzmq" Jul 07 19:00:18.609736 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.608945 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/cbf5b271-d24c-47eb-9971-3bab86804424-lib-modules\") pod \"tuned-bjzmq\" (UID: \"cbf5b271-d24c-47eb-9971-3bab86804424\") " pod="openshift-cluster-node-tuning-operator/tuned-bjzmq" Jul 07 19:00:18.609736 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.608982 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfqd4\" (UniqueName: \"kubernetes.io/projected/794c654b-f09b-4936-b01f-eccd5e7a2a95-kube-api-access-jfqd4\") pod \"multus-additional-cni-plugins-jls8k\" (UID: \"794c654b-f09b-4936-b01f-eccd5e7a2a95\") " pod="openshift-multus/multus-additional-cni-plugins-jls8k" Jul 07 19:00:18.609736 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.609015 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/cbf5b271-d24c-47eb-9971-3bab86804424-etc-sysconfig\") pod \"tuned-bjzmq\" (UID: \"cbf5b271-d24c-47eb-9971-3bab86804424\") " pod="openshift-cluster-node-tuning-operator/tuned-bjzmq" Jul 07 19:00:18.609736 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.609033 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/cbf5b271-d24c-47eb-9971-3bab86804424-etc-tuned\") pod \"tuned-bjzmq\" (UID: \"cbf5b271-d24c-47eb-9971-3bab86804424\") " pod="openshift-cluster-node-tuning-operator/tuned-bjzmq" Jul 07 19:00:18.609736 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.609051 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/4f25fdbd-3251-4ef6-ab69-e87cc915a469-agent-certs\") pod \"konnectivity-agent-dsvgs\" (UID: \"4f25fdbd-3251-4ef6-ab69-e87cc915a469\") " pod="kube-system/konnectivity-agent-dsvgs" Jul 07 19:00:18.609736 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.609123 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cbf5b271-d24c-47eb-9971-3bab86804424-etc-kubernetes\") pod \"tuned-bjzmq\" (UID: \"cbf5b271-d24c-47eb-9971-3bab86804424\") " pod="openshift-cluster-node-tuning-operator/tuned-bjzmq" Jul 07 19:00:18.609736 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.609138 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/794c654b-f09b-4936-b01f-eccd5e7a2a95-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-jls8k\" (UID: \"794c654b-f09b-4936-b01f-eccd5e7a2a95\") " pod="openshift-multus/multus-additional-cni-plugins-jls8k" Jul 07 19:00:18.609736 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.609185 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/794c654b-f09b-4936-b01f-eccd5e7a2a95-cnibin\") pod \"multus-additional-cni-plugins-jls8k\" (UID: \"794c654b-f09b-4936-b01f-eccd5e7a2a95\") " pod="openshift-multus/multus-additional-cni-plugins-jls8k" Jul 07 19:00:18.609736 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.609225 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/794c654b-f09b-4936-b01f-eccd5e7a2a95-cni-binary-copy\") pod \"multus-additional-cni-plugins-jls8k\" (UID: \"794c654b-f09b-4936-b01f-eccd5e7a2a95\") " pod="openshift-multus/multus-additional-cni-plugins-jls8k" Jul 07 19:00:18.609736 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.609238 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/794c654b-f09b-4936-b01f-eccd5e7a2a95-os-release\") pod \"multus-additional-cni-plugins-jls8k\" (UID: \"794c654b-f09b-4936-b01f-eccd5e7a2a95\") " pod="openshift-multus/multus-additional-cni-plugins-jls8k" Jul 07 19:00:18.609736 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.609253 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/cbf5b271-d24c-47eb-9971-3bab86804424-etc-modprobe-d\") pod \"tuned-bjzmq\" (UID: \"cbf5b271-d24c-47eb-9971-3bab86804424\") " pod="openshift-cluster-node-tuning-operator/tuned-bjzmq" Jul 07 19:00:18.609736 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.609143 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/cbf5b271-d24c-47eb-9971-3bab86804424-etc-modprobe-d\") pod \"tuned-bjzmq\" (UID: \"cbf5b271-d24c-47eb-9971-3bab86804424\") " pod="openshift-cluster-node-tuning-operator/tuned-bjzmq" Jul 07 19:00:18.610548 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.609285 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/4f25fdbd-3251-4ef6-ab69-e87cc915a469-konnectivity-ca\") pod \"konnectivity-agent-dsvgs\" (UID: \"4f25fdbd-3251-4ef6-ab69-e87cc915a469\") " pod="kube-system/konnectivity-agent-dsvgs" Jul 07 19:00:18.610548 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.609307 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xv8xh\" (UniqueName: \"kubernetes.io/projected/cbf5b271-d24c-47eb-9971-3bab86804424-kube-api-access-xv8xh\") pod \"tuned-bjzmq\" (UID: \"cbf5b271-d24c-47eb-9971-3bab86804424\") " pod="openshift-cluster-node-tuning-operator/tuned-bjzmq" Jul 07 19:00:18.610548 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.609334 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d6c3675f-6173-4275-8da8-3af895093d8a-etc-selinux\") pod \"aws-ebs-csi-driver-node-qkmgh\" (UID: \"d6c3675f-6173-4275-8da8-3af895093d8a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qkmgh" Jul 07 19:00:18.610548 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.609367 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qb4q8\" (UniqueName: \"kubernetes.io/projected/d6c3675f-6173-4275-8da8-3af895093d8a-kube-api-access-qb4q8\") pod \"aws-ebs-csi-driver-node-qkmgh\" (UID: \"d6c3675f-6173-4275-8da8-3af895093d8a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qkmgh" Jul 07 19:00:18.610548 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.609384 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/794c654b-f09b-4936-b01f-eccd5e7a2a95-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jls8k\" (UID: \"794c654b-f09b-4936-b01f-eccd5e7a2a95\") " pod="openshift-multus/multus-additional-cni-plugins-jls8k" Jul 07 19:00:18.610548 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.609392 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/794c654b-f09b-4936-b01f-eccd5e7a2a95-system-cni-dir\") pod \"multus-additional-cni-plugins-jls8k\" (UID: \"794c654b-f09b-4936-b01f-eccd5e7a2a95\") " pod="openshift-multus/multus-additional-cni-plugins-jls8k" Jul 07 19:00:18.610548 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.609422 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d6c3675f-6173-4275-8da8-3af895093d8a-device-dir\") pod \"aws-ebs-csi-driver-node-qkmgh\" (UID: \"d6c3675f-6173-4275-8da8-3af895093d8a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qkmgh" Jul 07 19:00:18.610548 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.609438 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d6c3675f-6173-4275-8da8-3af895093d8a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qkmgh\" (UID: \"d6c3675f-6173-4275-8da8-3af895093d8a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qkmgh" Jul 07 19:00:18.610548 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.609581 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/59f0c8fc-bc9a-4273-aac2-a8ee23bf6b63-dbus\") pod \"global-pull-secret-syncer-lk7md\" (UID: \"59f0c8fc-bc9a-4273-aac2-a8ee23bf6b63\") " pod="kube-system/global-pull-secret-syncer-lk7md" Jul 07 19:00:18.610548 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.609633 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/cbf5b271-d24c-47eb-9971-3bab86804424-etc-sysconfig\") pod \"tuned-bjzmq\" (UID: \"cbf5b271-d24c-47eb-9971-3bab86804424\") " pod="openshift-cluster-node-tuning-operator/tuned-bjzmq" Jul 07 19:00:18.610548 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.609641 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cbf5b271-d24c-47eb-9971-3bab86804424-sys\") pod \"tuned-bjzmq\" (UID: \"cbf5b271-d24c-47eb-9971-3bab86804424\") " pod="openshift-cluster-node-tuning-operator/tuned-bjzmq" Jul 07 19:00:18.610548 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.609668 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/794c654b-f09b-4936-b01f-eccd5e7a2a95-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-jls8k\" (UID: \"794c654b-f09b-4936-b01f-eccd5e7a2a95\") " pod="openshift-multus/multus-additional-cni-plugins-jls8k" Jul 07 19:00:18.610548 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.609704 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cbf5b271-d24c-47eb-9971-3bab86804424-etc-kubernetes\") pod \"tuned-bjzmq\" (UID: \"cbf5b271-d24c-47eb-9971-3bab86804424\") " pod="openshift-cluster-node-tuning-operator/tuned-bjzmq" Jul 07 19:00:18.610548 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.609729 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/cbf5b271-d24c-47eb-9971-3bab86804424-lib-modules\") pod \"tuned-bjzmq\" (UID: \"cbf5b271-d24c-47eb-9971-3bab86804424\") " pod="openshift-cluster-node-tuning-operator/tuned-bjzmq" Jul 07 19:00:18.610548 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.609737 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d6c3675f-6173-4275-8da8-3af895093d8a-device-dir\") pod \"aws-ebs-csi-driver-node-qkmgh\" (UID: \"d6c3675f-6173-4275-8da8-3af895093d8a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qkmgh" Jul 07 19:00:18.610548 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.609876 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d6c3675f-6173-4275-8da8-3af895093d8a-etc-selinux\") pod \"aws-ebs-csi-driver-node-qkmgh\" (UID: \"d6c3675f-6173-4275-8da8-3af895093d8a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qkmgh" Jul 07 19:00:18.610548 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.609885 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/794c654b-f09b-4936-b01f-eccd5e7a2a95-system-cni-dir\") pod \"multus-additional-cni-plugins-jls8k\" (UID: \"794c654b-f09b-4936-b01f-eccd5e7a2a95\") " pod="openshift-multus/multus-additional-cni-plugins-jls8k" Jul 07 19:00:18.611558 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.611540 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cbf5b271-d24c-47eb-9971-3bab86804424-tmp\") pod \"tuned-bjzmq\" (UID: \"cbf5b271-d24c-47eb-9971-3bab86804424\") " pod="openshift-cluster-node-tuning-operator/tuned-bjzmq" Jul 07 19:00:18.611751 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.611727 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/cbf5b271-d24c-47eb-9971-3bab86804424-etc-tuned\") pod \"tuned-bjzmq\" (UID: \"cbf5b271-d24c-47eb-9971-3bab86804424\") " pod="openshift-cluster-node-tuning-operator/tuned-bjzmq" Jul 07 19:00:18.611876 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.611860 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/4f25fdbd-3251-4ef6-ab69-e87cc915a469-agent-certs\") pod \"konnectivity-agent-dsvgs\" (UID: \"4f25fdbd-3251-4ef6-ab69-e87cc915a469\") " pod="kube-system/konnectivity-agent-dsvgs" Jul 07 19:00:18.613556 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.613543 2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 07 19:00:18.618440 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.618422 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jfqd4\" (UniqueName: \"kubernetes.io/projected/794c654b-f09b-4936-b01f-eccd5e7a2a95-kube-api-access-jfqd4\") pod \"multus-additional-cni-plugins-jls8k\" (UID: \"794c654b-f09b-4936-b01f-eccd5e7a2a95\") " pod="openshift-multus/multus-additional-cni-plugins-jls8k" Jul 07 19:00:18.620634 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.620614 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xv8xh\" (UniqueName: \"kubernetes.io/projected/cbf5b271-d24c-47eb-9971-3bab86804424-kube-api-access-xv8xh\") pod \"tuned-bjzmq\" (UID: \"cbf5b271-d24c-47eb-9971-3bab86804424\") " pod="openshift-cluster-node-tuning-operator/tuned-bjzmq" Jul 07 19:00:18.624918 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.624899 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qb4q8\" (UniqueName: \"kubernetes.io/projected/d6c3675f-6173-4275-8da8-3af895093d8a-kube-api-access-qb4q8\") pod \"aws-ebs-csi-driver-node-qkmgh\" (UID: \"d6c3675f-6173-4275-8da8-3af895093d8a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qkmgh" Jul 07 19:00:18.629535 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.629518 2563 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 07 19:00:18.678621 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:18.678586 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd64b863190df7f2a681aceffe233fdc3.slice/crio-39d630e12d0f4811e65fd0a5d2e6d30540bf65db794ded28bbe1f0945376ddc3 WatchSource:0}: Error finding container 39d630e12d0f4811e65fd0a5d2e6d30540bf65db794ded28bbe1f0945376ddc3: Status 404 returned error can't find the container with id 39d630e12d0f4811e65fd0a5d2e6d30540bf65db794ded28bbe1f0945376ddc3 Jul 07 19:00:18.716070 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.716034 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-x7d7k" Jul 07 19:00:18.724570 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:18.724541 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1221769_6ea8_43b7_aa11_e4b2d3699a7b.slice/crio-373f553fd0df62f9daee9ba1995fc86df568dd7b674ad9d0d15f79e683d0ec81 WatchSource:0}: Error finding container 373f553fd0df62f9daee9ba1995fc86df568dd7b674ad9d0d15f79e683d0ec81: Status 404 returned error can't find the container with id 373f553fd0df62f9daee9ba1995fc86df568dd7b674ad9d0d15f79e683d0ec81 Jul 07 19:00:18.732826 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.732807 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-whw2v" Jul 07 19:00:18.740748 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:18.740722 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3953dadf_98ed_4933_a330_1191906b6cff.slice/crio-015ed6475c1ac5ff4c18954d8eaabfe1c8b32c3aeb49d0a133f2c0d934fe4484 WatchSource:0}: Error finding container 015ed6475c1ac5ff4c18954d8eaabfe1c8b32c3aeb49d0a133f2c0d934fe4484: Status 404 returned error can't find the container with id 015ed6475c1ac5ff4c18954d8eaabfe1c8b32c3aeb49d0a133f2c0d934fe4484 Jul 07 19:00:18.755150 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.755130 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-s272w" Jul 07 19:00:18.763408 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:18.763385 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6abe188_2b1b_427a_ba0c_e93930187c7e.slice/crio-88bcac580adadc1856cbc6d20b43b202ae59d98dc0b3e0b2692a177b054bde02 WatchSource:0}: Error finding container 88bcac580adadc1856cbc6d20b43b202ae59d98dc0b3e0b2692a177b054bde02: Status 404 returned error can't find the container with id 88bcac580adadc1856cbc6d20b43b202ae59d98dc0b3e0b2692a177b054bde02 Jul 07 19:00:18.768673 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.768653 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-8ksxw" Jul 07 19:00:18.776474 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:18.776453 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63c4ef25_f98e_4224_ad6c_1eb0267d23df.slice/crio-b1c36f8f5a67f5a5df7d614f42c119e16bbb69a1e8772c6c6e18fa7f1bb181dc WatchSource:0}: Error finding container b1c36f8f5a67f5a5df7d614f42c119e16bbb69a1e8772c6c6e18fa7f1bb181dc: Status 404 returned error can't find the container with id b1c36f8f5a67f5a5df7d614f42c119e16bbb69a1e8772c6c6e18fa7f1bb181dc Jul 07 19:00:18.783522 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.783503 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-zzwbq" Jul 07 19:00:18.791514 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:18.791490 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f0c5d5c_147f_426a_965d_8886e5e79984.slice/crio-d8c199b10ab0881facb3398b95a828f9759bca2b518f653c3d25ecbcc541f86e WatchSource:0}: Error finding container d8c199b10ab0881facb3398b95a828f9759bca2b518f653c3d25ecbcc541f86e: Status 404 returned error can't find the container with id d8c199b10ab0881facb3398b95a828f9759bca2b518f653c3d25ecbcc541f86e Jul 07 19:00:18.806544 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.806525 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-dsvgs" Jul 07 19:00:18.814388 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:18.814360 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f25fdbd_3251_4ef6_ab69_e87cc915a469.slice/crio-64835e5c5906c021facd5f52f2b4a0784d33ef7ce759772f117f59cab4cb47f2 WatchSource:0}: Error finding container 64835e5c5906c021facd5f52f2b4a0784d33ef7ce759772f117f59cab4cb47f2: Status 404 returned error can't find the container with id 64835e5c5906c021facd5f52f2b4a0784d33ef7ce759772f117f59cab4cb47f2 Jul 07 19:00:18.829026 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.829003 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qkmgh" Jul 07 19:00:18.838201 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:18.838176 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6c3675f_6173_4275_8da8_3af895093d8a.slice/crio-27ec7602f89f8066db6f35a6584437a2be0c6d7fb1d9a6dc779d8fa728224de3 WatchSource:0}: Error finding container 27ec7602f89f8066db6f35a6584437a2be0c6d7fb1d9a6dc779d8fa728224de3: Status 404 returned error can't find the container with id 27ec7602f89f8066db6f35a6584437a2be0c6d7fb1d9a6dc779d8fa728224de3 Jul 07 19:00:18.844517 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.844497 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-bjzmq" Jul 07 19:00:18.849510 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:18.849494 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-jls8k" Jul 07 19:00:18.852671 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:18.852648 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbf5b271_d24c_47eb_9971_3bab86804424.slice/crio-032d0426cce380aef1181a250683933e1a2741c55009481cf77d588894b56e41 WatchSource:0}: Error finding container 032d0426cce380aef1181a250683933e1a2741c55009481cf77d588894b56e41: Status 404 returned error can't find the container with id 032d0426cce380aef1181a250683933e1a2741c55009481cf77d588894b56e41 Jul 07 19:00:18.859360 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:18.859336 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod794c654b_f09b_4936_b01f_eccd5e7a2a95.slice/crio-12c2350fd4f78ddef17337b4529bb84700a546105c1cee21206526a337c4a4a1 WatchSource:0}: Error finding container 12c2350fd4f78ddef17337b4529bb84700a546105c1cee21206526a337c4a4a1: Status 404 returned error can't find the container with id 12c2350fd4f78ddef17337b4529bb84700a546105c1cee21206526a337c4a4a1 Jul 07 19:00:19.012238 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:19.012157 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/396810c7-8961-4755-b0f6-02a9d7e02ae3-metrics-certs\") pod \"network-metrics-daemon-h8m9b\" (UID: \"396810c7-8961-4755-b0f6-02a9d7e02ae3\") " pod="openshift-multus/network-metrics-daemon-h8m9b" Jul 07 19:00:19.012391 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:00:19.012279 2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 19:00:19.012391 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:00:19.012344 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/396810c7-8961-4755-b0f6-02a9d7e02ae3-metrics-certs podName:396810c7-8961-4755-b0f6-02a9d7e02ae3 nodeName:}" failed. No retries permitted until 2026-07-07 19:00:20.012327042 +0000 UTC m=+3.028331201 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/396810c7-8961-4755-b0f6-02a9d7e02ae3-metrics-certs") pod "network-metrics-daemon-h8m9b" (UID: "396810c7-8961-4755-b0f6-02a9d7e02ae3") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 19:00:19.112710 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:19.112676 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgqkm\" (UniqueName: \"kubernetes.io/projected/1d84b355-0ac7-4d14-83b6-7c18077e9319-kube-api-access-bgqkm\") pod \"network-check-target-d8t2d\" (UID: \"1d84b355-0ac7-4d14-83b6-7c18077e9319\") " pod="openshift-network-diagnostics/network-check-target-d8t2d" Jul 07 19:00:19.112894 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:19.112741 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/59f0c8fc-bc9a-4273-aac2-a8ee23bf6b63-original-pull-secret\") pod \"global-pull-secret-syncer-lk7md\" (UID: \"59f0c8fc-bc9a-4273-aac2-a8ee23bf6b63\") " pod="kube-system/global-pull-secret-syncer-lk7md" Jul 07 19:00:19.112894 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:00:19.112835 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 07 19:00:19.112894 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:00:19.112854 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 07 19:00:19.112894 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:00:19.112858 2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 07 19:00:19.112894 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:00:19.112877 2563 projected.go:196] Error preparing data for projected volume kube-api-access-bgqkm for pod openshift-network-diagnostics/network-check-target-d8t2d: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 19:00:19.113173 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:00:19.112919 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/59f0c8fc-bc9a-4273-aac2-a8ee23bf6b63-original-pull-secret podName:59f0c8fc-bc9a-4273-aac2-a8ee23bf6b63 nodeName:}" failed. No retries permitted until 2026-07-07 19:00:20.112902235 +0000 UTC m=+3.128906406 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/59f0c8fc-bc9a-4273-aac2-a8ee23bf6b63-original-pull-secret") pod "global-pull-secret-syncer-lk7md" (UID: "59f0c8fc-bc9a-4273-aac2-a8ee23bf6b63") : object "kube-system"/"original-pull-secret" not registered Jul 07 19:00:19.113173 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:00:19.112939 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1d84b355-0ac7-4d14-83b6-7c18077e9319-kube-api-access-bgqkm podName:1d84b355-0ac7-4d14-83b6-7c18077e9319 nodeName:}" failed. No retries permitted until 2026-07-07 19:00:20.112929089 +0000 UTC m=+3.128933251 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-bgqkm" (UniqueName: "kubernetes.io/projected/1d84b355-0ac7-4d14-83b6-7c18077e9319-kube-api-access-bgqkm") pod "network-check-target-d8t2d" (UID: "1d84b355-0ac7-4d14-83b6-7c18077e9319") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 19:00:19.462530 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:19.462427 2563 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-06 18:55:18 +0000 UTC" deadline="2028-04-17 07:57:54.684679968 +0000 UTC" Jul 07 19:00:19.462530 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:19.462468 2563 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15588h57m35.222216209s" Jul 07 19:00:19.486113 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:19.486014 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jls8k" event={"ID":"794c654b-f09b-4936-b01f-eccd5e7a2a95","Type":"ContainerStarted","Data":"12c2350fd4f78ddef17337b4529bb84700a546105c1cee21206526a337c4a4a1"} Jul 07 19:00:19.487917 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:19.487891 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-bjzmq" event={"ID":"cbf5b271-d24c-47eb-9971-3bab86804424","Type":"ContainerStarted","Data":"032d0426cce380aef1181a250683933e1a2741c55009481cf77d588894b56e41"} Jul 07 19:00:19.490916 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:19.490888 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qkmgh" event={"ID":"d6c3675f-6173-4275-8da8-3af895093d8a","Type":"ContainerStarted","Data":"27ec7602f89f8066db6f35a6584437a2be0c6d7fb1d9a6dc779d8fa728224de3"} Jul 07 19:00:19.492688 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:19.492658 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-s272w" event={"ID":"a6abe188-2b1b-427a-ba0c-e93930187c7e","Type":"ContainerStarted","Data":"88bcac580adadc1856cbc6d20b43b202ae59d98dc0b3e0b2692a177b054bde02"} Jul 07 19:00:19.493319 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:19.493299 2563 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 07 19:00:19.494325 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:19.494283 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-whw2v" event={"ID":"3953dadf-98ed-4933-a330-1191906b6cff","Type":"ContainerStarted","Data":"015ed6475c1ac5ff4c18954d8eaabfe1c8b32c3aeb49d0a133f2c0d934fe4484"} Jul 07 19:00:19.496390 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:19.496367 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-x7d7k" event={"ID":"a1221769-6ea8-43b7-aa11-e4b2d3699a7b","Type":"ContainerStarted","Data":"373f553fd0df62f9daee9ba1995fc86df568dd7b674ad9d0d15f79e683d0ec81"} Jul 07 19:00:19.499206 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:19.499164 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-dsvgs" event={"ID":"4f25fdbd-3251-4ef6-ab69-e87cc915a469","Type":"ContainerStarted","Data":"64835e5c5906c021facd5f52f2b4a0784d33ef7ce759772f117f59cab4cb47f2"} Jul 07 19:00:19.501697 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:19.501674 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zzwbq" event={"ID":"4f0c5d5c-147f-426a-965d-8886e5e79984","Type":"ContainerStarted","Data":"d8c199b10ab0881facb3398b95a828f9759bca2b518f653c3d25ecbcc541f86e"} Jul 07 19:00:19.503385 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:19.503364 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-8ksxw" event={"ID":"63c4ef25-f98e-4224-ad6c-1eb0267d23df","Type":"ContainerStarted","Data":"b1c36f8f5a67f5a5df7d614f42c119e16bbb69a1e8772c6c6e18fa7f1bb181dc"} Jul 07 19:00:19.505739 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:19.505718 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-134.ec2.internal" event={"ID":"d64b863190df7f2a681aceffe233fdc3","Type":"ContainerStarted","Data":"39d630e12d0f4811e65fd0a5d2e6d30540bf65db794ded28bbe1f0945376ddc3"} Jul 07 19:00:19.508329 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:19.508269 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-134.ec2.internal" event={"ID":"067754c774fbc99720cce604846fd8f3","Type":"ContainerStarted","Data":"121aaaed5cfe6db86cef7a6d01166bb30cd083ad7f98dfa26cb024eb1e55a42c"} Jul 07 19:00:20.020862 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:20.020823 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/396810c7-8961-4755-b0f6-02a9d7e02ae3-metrics-certs\") pod \"network-metrics-daemon-h8m9b\" (UID: \"396810c7-8961-4755-b0f6-02a9d7e02ae3\") " pod="openshift-multus/network-metrics-daemon-h8m9b" Jul 07 19:00:20.021073 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:00:20.021025 2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 19:00:20.021147 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:00:20.021102 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/396810c7-8961-4755-b0f6-02a9d7e02ae3-metrics-certs podName:396810c7-8961-4755-b0f6-02a9d7e02ae3 nodeName:}" failed. No retries permitted until 2026-07-07 19:00:22.021083599 +0000 UTC m=+5.037087758 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/396810c7-8961-4755-b0f6-02a9d7e02ae3-metrics-certs") pod "network-metrics-daemon-h8m9b" (UID: "396810c7-8961-4755-b0f6-02a9d7e02ae3") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 19:00:20.122261 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:20.121444 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/59f0c8fc-bc9a-4273-aac2-a8ee23bf6b63-original-pull-secret\") pod \"global-pull-secret-syncer-lk7md\" (UID: \"59f0c8fc-bc9a-4273-aac2-a8ee23bf6b63\") " pod="kube-system/global-pull-secret-syncer-lk7md" Jul 07 19:00:20.122261 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:20.121512 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgqkm\" (UniqueName: \"kubernetes.io/projected/1d84b355-0ac7-4d14-83b6-7c18077e9319-kube-api-access-bgqkm\") pod \"network-check-target-d8t2d\" (UID: \"1d84b355-0ac7-4d14-83b6-7c18077e9319\") " pod="openshift-network-diagnostics/network-check-target-d8t2d" Jul 07 19:00:20.122261 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:00:20.121676 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 07 19:00:20.122261 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:00:20.121698 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 07 19:00:20.122261 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:00:20.121712 2563 projected.go:196] Error preparing data for projected volume kube-api-access-bgqkm for pod openshift-network-diagnostics/network-check-target-d8t2d: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 19:00:20.122261 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:00:20.121772 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1d84b355-0ac7-4d14-83b6-7c18077e9319-kube-api-access-bgqkm podName:1d84b355-0ac7-4d14-83b6-7c18077e9319 nodeName:}" failed. No retries permitted until 2026-07-07 19:00:22.121754414 +0000 UTC m=+5.137758584 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-bgqkm" (UniqueName: "kubernetes.io/projected/1d84b355-0ac7-4d14-83b6-7c18077e9319-kube-api-access-bgqkm") pod "network-check-target-d8t2d" (UID: "1d84b355-0ac7-4d14-83b6-7c18077e9319") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 19:00:20.122261 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:00:20.122181 2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 07 19:00:20.122261 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:00:20.122235 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/59f0c8fc-bc9a-4273-aac2-a8ee23bf6b63-original-pull-secret podName:59f0c8fc-bc9a-4273-aac2-a8ee23bf6b63 nodeName:}" failed. No retries permitted until 2026-07-07 19:00:22.122220263 +0000 UTC m=+5.138224426 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/59f0c8fc-bc9a-4273-aac2-a8ee23bf6b63-original-pull-secret") pod "global-pull-secret-syncer-lk7md" (UID: "59f0c8fc-bc9a-4273-aac2-a8ee23bf6b63") : object "kube-system"/"original-pull-secret" not registered Jul 07 19:00:20.463442 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:20.463374 2563 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-06 18:55:18 +0000 UTC" deadline="2028-03-24 03:46:57.10634877 +0000 UTC" Jul 07 19:00:20.463442 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:20.463440 2563 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15008h46m36.642913109s" Jul 07 19:00:20.471250 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:20.471219 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lk7md" Jul 07 19:00:20.471410 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:00:20.471353 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lk7md" podUID="59f0c8fc-bc9a-4273-aac2-a8ee23bf6b63" Jul 07 19:00:20.471800 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:20.471778 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h8m9b" Jul 07 19:00:20.471918 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:00:20.471890 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h8m9b" podUID="396810c7-8961-4755-b0f6-02a9d7e02ae3" Jul 07 19:00:20.472171 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:20.471983 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d8t2d" Jul 07 19:00:20.472171 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:00:20.472059 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d8t2d" podUID="1d84b355-0ac7-4d14-83b6-7c18077e9319" Jul 07 19:00:22.039315 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:22.039266 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/396810c7-8961-4755-b0f6-02a9d7e02ae3-metrics-certs\") pod \"network-metrics-daemon-h8m9b\" (UID: \"396810c7-8961-4755-b0f6-02a9d7e02ae3\") " pod="openshift-multus/network-metrics-daemon-h8m9b" Jul 07 19:00:22.039772 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:00:22.039446 2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 19:00:22.039772 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:00:22.039524 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/396810c7-8961-4755-b0f6-02a9d7e02ae3-metrics-certs podName:396810c7-8961-4755-b0f6-02a9d7e02ae3 nodeName:}" failed. No retries permitted until 2026-07-07 19:00:26.039504043 +0000 UTC m=+9.055508222 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/396810c7-8961-4755-b0f6-02a9d7e02ae3-metrics-certs") pod "network-metrics-daemon-h8m9b" (UID: "396810c7-8961-4755-b0f6-02a9d7e02ae3") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 19:00:22.140337 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:22.140251 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgqkm\" (UniqueName: \"kubernetes.io/projected/1d84b355-0ac7-4d14-83b6-7c18077e9319-kube-api-access-bgqkm\") pod \"network-check-target-d8t2d\" (UID: \"1d84b355-0ac7-4d14-83b6-7c18077e9319\") " pod="openshift-network-diagnostics/network-check-target-d8t2d" Jul 07 19:00:22.140506 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:22.140341 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/59f0c8fc-bc9a-4273-aac2-a8ee23bf6b63-original-pull-secret\") pod \"global-pull-secret-syncer-lk7md\" (UID: \"59f0c8fc-bc9a-4273-aac2-a8ee23bf6b63\") " pod="kube-system/global-pull-secret-syncer-lk7md" Jul 07 19:00:22.140506 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:00:22.140486 2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 07 19:00:22.140625 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:00:22.140558 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/59f0c8fc-bc9a-4273-aac2-a8ee23bf6b63-original-pull-secret podName:59f0c8fc-bc9a-4273-aac2-a8ee23bf6b63 nodeName:}" failed. No retries permitted until 2026-07-07 19:00:26.140532557 +0000 UTC m=+9.156536720 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/59f0c8fc-bc9a-4273-aac2-a8ee23bf6b63-original-pull-secret") pod "global-pull-secret-syncer-lk7md" (UID: "59f0c8fc-bc9a-4273-aac2-a8ee23bf6b63") : object "kube-system"/"original-pull-secret" not registered Jul 07 19:00:22.141061 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:00:22.141008 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 07 19:00:22.141061 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:00:22.141034 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 07 19:00:22.141061 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:00:22.141046 2563 projected.go:196] Error preparing data for projected volume kube-api-access-bgqkm for pod openshift-network-diagnostics/network-check-target-d8t2d: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 19:00:22.141281 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:00:22.141093 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1d84b355-0ac7-4d14-83b6-7c18077e9319-kube-api-access-bgqkm podName:1d84b355-0ac7-4d14-83b6-7c18077e9319 nodeName:}" failed. No retries permitted until 2026-07-07 19:00:26.141077927 +0000 UTC m=+9.157082099 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-bgqkm" (UniqueName: "kubernetes.io/projected/1d84b355-0ac7-4d14-83b6-7c18077e9319-kube-api-access-bgqkm") pod "network-check-target-d8t2d" (UID: "1d84b355-0ac7-4d14-83b6-7c18077e9319") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 19:00:22.470269 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:22.470191 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d8t2d" Jul 07 19:00:22.470426 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:00:22.470379 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d8t2d" podUID="1d84b355-0ac7-4d14-83b6-7c18077e9319" Jul 07 19:00:22.470794 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:22.470773 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lk7md" Jul 07 19:00:22.471182 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:00:22.470918 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lk7md" podUID="59f0c8fc-bc9a-4273-aac2-a8ee23bf6b63" Jul 07 19:00:22.471182 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:22.470994 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h8m9b" Jul 07 19:00:22.471182 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:00:22.471112 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h8m9b" podUID="396810c7-8961-4755-b0f6-02a9d7e02ae3" Jul 07 19:00:24.470281 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:24.470236 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d8t2d" Jul 07 19:00:24.470808 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:24.470376 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lk7md" Jul 07 19:00:24.470808 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:00:24.470385 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d8t2d" podUID="1d84b355-0ac7-4d14-83b6-7c18077e9319" Jul 07 19:00:24.470808 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:24.470402 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h8m9b" Jul 07 19:00:24.470808 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:00:24.470510 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lk7md" podUID="59f0c8fc-bc9a-4273-aac2-a8ee23bf6b63" Jul 07 19:00:24.470808 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:00:24.470614 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h8m9b" podUID="396810c7-8961-4755-b0f6-02a9d7e02ae3" Jul 07 19:00:26.075437 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:26.075389 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/396810c7-8961-4755-b0f6-02a9d7e02ae3-metrics-certs\") pod \"network-metrics-daemon-h8m9b\" (UID: \"396810c7-8961-4755-b0f6-02a9d7e02ae3\") " pod="openshift-multus/network-metrics-daemon-h8m9b" Jul 07 19:00:26.075893 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:00:26.075539 2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 19:00:26.075893 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:00:26.075614 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/396810c7-8961-4755-b0f6-02a9d7e02ae3-metrics-certs podName:396810c7-8961-4755-b0f6-02a9d7e02ae3 nodeName:}" failed. No retries permitted until 2026-07-07 19:00:34.075593876 +0000 UTC m=+17.091598060 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/396810c7-8961-4755-b0f6-02a9d7e02ae3-metrics-certs") pod "network-metrics-daemon-h8m9b" (UID: "396810c7-8961-4755-b0f6-02a9d7e02ae3") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 19:00:26.176735 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:26.176694 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/59f0c8fc-bc9a-4273-aac2-a8ee23bf6b63-original-pull-secret\") pod \"global-pull-secret-syncer-lk7md\" (UID: \"59f0c8fc-bc9a-4273-aac2-a8ee23bf6b63\") " pod="kube-system/global-pull-secret-syncer-lk7md" Jul 07 19:00:26.176927 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:26.176767 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgqkm\" (UniqueName: \"kubernetes.io/projected/1d84b355-0ac7-4d14-83b6-7c18077e9319-kube-api-access-bgqkm\") pod \"network-check-target-d8t2d\" (UID: \"1d84b355-0ac7-4d14-83b6-7c18077e9319\") " pod="openshift-network-diagnostics/network-check-target-d8t2d" Jul 07 19:00:26.176927 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:00:26.176880 2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 07 19:00:26.176927 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:00:26.176902 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 07 19:00:26.176927 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:00:26.176922 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 07 19:00:26.177173 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:00:26.176935 2563 projected.go:196] Error preparing data for projected volume kube-api-access-bgqkm for pod openshift-network-diagnostics/network-check-target-d8t2d: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 19:00:26.177173 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:00:26.176975 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/59f0c8fc-bc9a-4273-aac2-a8ee23bf6b63-original-pull-secret podName:59f0c8fc-bc9a-4273-aac2-a8ee23bf6b63 nodeName:}" failed. No retries permitted until 2026-07-07 19:00:34.176937005 +0000 UTC m=+17.192941169 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/59f0c8fc-bc9a-4273-aac2-a8ee23bf6b63-original-pull-secret") pod "global-pull-secret-syncer-lk7md" (UID: "59f0c8fc-bc9a-4273-aac2-a8ee23bf6b63") : object "kube-system"/"original-pull-secret" not registered Jul 07 19:00:26.177173 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:00:26.176999 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1d84b355-0ac7-4d14-83b6-7c18077e9319-kube-api-access-bgqkm podName:1d84b355-0ac7-4d14-83b6-7c18077e9319 nodeName:}" failed. No retries permitted until 2026-07-07 19:00:34.176984236 +0000 UTC m=+17.192988416 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-bgqkm" (UniqueName: "kubernetes.io/projected/1d84b355-0ac7-4d14-83b6-7c18077e9319-kube-api-access-bgqkm") pod "network-check-target-d8t2d" (UID: "1d84b355-0ac7-4d14-83b6-7c18077e9319") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 19:00:26.471003 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:26.470968 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d8t2d" Jul 07 19:00:26.471177 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:26.470968 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lk7md" Jul 07 19:00:26.471177 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:26.470972 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h8m9b" Jul 07 19:00:26.471300 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:00:26.471181 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d8t2d" podUID="1d84b355-0ac7-4d14-83b6-7c18077e9319" Jul 07 19:00:26.471300 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:00:26.471253 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lk7md" podUID="59f0c8fc-bc9a-4273-aac2-a8ee23bf6b63" Jul 07 19:00:26.471400 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:00:26.471333 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h8m9b" podUID="396810c7-8961-4755-b0f6-02a9d7e02ae3" Jul 07 19:00:28.471552 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:28.471053 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d8t2d" Jul 07 19:00:28.471552 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:28.471210 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lk7md" Jul 07 19:00:28.471552 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:00:28.471297 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lk7md" podUID="59f0c8fc-bc9a-4273-aac2-a8ee23bf6b63" Jul 07 19:00:28.471552 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:00:28.471202 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d8t2d" podUID="1d84b355-0ac7-4d14-83b6-7c18077e9319" Jul 07 19:00:28.471552 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:28.471363 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h8m9b" Jul 07 19:00:28.471552 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:00:28.471432 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h8m9b" podUID="396810c7-8961-4755-b0f6-02a9d7e02ae3" Jul 07 19:00:30.471044 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:30.471000 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lk7md" Jul 07 19:00:30.471044 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:30.471043 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d8t2d" Jul 07 19:00:30.471559 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:30.471009 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h8m9b" Jul 07 19:00:30.471559 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:00:30.471126 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lk7md" podUID="59f0c8fc-bc9a-4273-aac2-a8ee23bf6b63" Jul 07 19:00:30.471559 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:00:30.471189 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h8m9b" podUID="396810c7-8961-4755-b0f6-02a9d7e02ae3" Jul 07 19:00:30.471559 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:00:30.471295 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d8t2d" podUID="1d84b355-0ac7-4d14-83b6-7c18077e9319" Jul 07 19:00:32.471085 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:32.470842 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lk7md" Jul 07 19:00:32.471563 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:32.470902 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h8m9b" Jul 07 19:00:32.471563 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:00:32.471159 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lk7md" podUID="59f0c8fc-bc9a-4273-aac2-a8ee23bf6b63" Jul 07 19:00:32.471563 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:00:32.471248 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h8m9b" podUID="396810c7-8961-4755-b0f6-02a9d7e02ae3" Jul 07 19:00:32.471563 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:32.470906 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d8t2d" Jul 07 19:00:32.471563 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:00:32.471332 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d8t2d" podUID="1d84b355-0ac7-4d14-83b6-7c18077e9319" Jul 07 19:00:34.139534 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:34.139496 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/396810c7-8961-4755-b0f6-02a9d7e02ae3-metrics-certs\") pod \"network-metrics-daemon-h8m9b\" (UID: \"396810c7-8961-4755-b0f6-02a9d7e02ae3\") " pod="openshift-multus/network-metrics-daemon-h8m9b" Jul 07 19:00:34.139897 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:00:34.139607 2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 19:00:34.139897 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:00:34.139656 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/396810c7-8961-4755-b0f6-02a9d7e02ae3-metrics-certs podName:396810c7-8961-4755-b0f6-02a9d7e02ae3 nodeName:}" failed. No retries permitted until 2026-07-07 19:00:50.139643489 +0000 UTC m=+33.155647647 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/396810c7-8961-4755-b0f6-02a9d7e02ae3-metrics-certs") pod "network-metrics-daemon-h8m9b" (UID: "396810c7-8961-4755-b0f6-02a9d7e02ae3") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 19:00:34.239923 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:34.239892 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/59f0c8fc-bc9a-4273-aac2-a8ee23bf6b63-original-pull-secret\") pod \"global-pull-secret-syncer-lk7md\" (UID: \"59f0c8fc-bc9a-4273-aac2-a8ee23bf6b63\") " pod="kube-system/global-pull-secret-syncer-lk7md" Jul 07 19:00:34.240098 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:34.239939 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgqkm\" (UniqueName: \"kubernetes.io/projected/1d84b355-0ac7-4d14-83b6-7c18077e9319-kube-api-access-bgqkm\") pod \"network-check-target-d8t2d\" (UID: \"1d84b355-0ac7-4d14-83b6-7c18077e9319\") " pod="openshift-network-diagnostics/network-check-target-d8t2d" Jul 07 19:00:34.240098 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:00:34.240050 2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 07 19:00:34.240175 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:00:34.240103 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/59f0c8fc-bc9a-4273-aac2-a8ee23bf6b63-original-pull-secret podName:59f0c8fc-bc9a-4273-aac2-a8ee23bf6b63 nodeName:}" failed. No retries permitted until 2026-07-07 19:00:50.240089189 +0000 UTC m=+33.256093348 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/59f0c8fc-bc9a-4273-aac2-a8ee23bf6b63-original-pull-secret") pod "global-pull-secret-syncer-lk7md" (UID: "59f0c8fc-bc9a-4273-aac2-a8ee23bf6b63") : object "kube-system"/"original-pull-secret" not registered Jul 07 19:00:34.240175 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:00:34.240124 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 07 19:00:34.240175 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:00:34.240139 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 07 19:00:34.240175 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:00:34.240148 2563 projected.go:196] Error preparing data for projected volume kube-api-access-bgqkm for pod openshift-network-diagnostics/network-check-target-d8t2d: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 19:00:34.240316 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:00:34.240194 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1d84b355-0ac7-4d14-83b6-7c18077e9319-kube-api-access-bgqkm podName:1d84b355-0ac7-4d14-83b6-7c18077e9319 nodeName:}" failed. No retries permitted until 2026-07-07 19:00:50.240182042 +0000 UTC m=+33.256186201 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-bgqkm" (UniqueName: "kubernetes.io/projected/1d84b355-0ac7-4d14-83b6-7c18077e9319-kube-api-access-bgqkm") pod "network-check-target-d8t2d" (UID: "1d84b355-0ac7-4d14-83b6-7c18077e9319") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 19:00:34.470561 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:34.470521 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lk7md" Jul 07 19:00:34.470749 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:34.470521 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d8t2d" Jul 07 19:00:34.470749 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:00:34.470664 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lk7md" podUID="59f0c8fc-bc9a-4273-aac2-a8ee23bf6b63" Jul 07 19:00:34.470749 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:34.470521 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h8m9b" Jul 07 19:00:34.470749 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:00:34.470737 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d8t2d" podUID="1d84b355-0ac7-4d14-83b6-7c18077e9319" Jul 07 19:00:34.470939 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:00:34.470828 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h8m9b" podUID="396810c7-8961-4755-b0f6-02a9d7e02ae3" Jul 07 19:00:36.471018 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:36.470986 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lk7md" Jul 07 19:00:36.471554 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:36.470986 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h8m9b" Jul 07 19:00:36.471554 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:00:36.471105 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lk7md" podUID="59f0c8fc-bc9a-4273-aac2-a8ee23bf6b63" Jul 07 19:00:36.471554 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:00:36.471211 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h8m9b" podUID="396810c7-8961-4755-b0f6-02a9d7e02ae3" Jul 07 19:00:36.471554 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:36.470998 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d8t2d" Jul 07 19:00:36.471554 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:00:36.471304 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d8t2d" podUID="1d84b355-0ac7-4d14-83b6-7c18077e9319" Jul 07 19:00:38.470381 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:38.470341 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d8t2d" Jul 07 19:00:38.470872 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:38.470341 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lk7md" Jul 07 19:00:38.470872 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:00:38.470464 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d8t2d" podUID="1d84b355-0ac7-4d14-83b6-7c18077e9319" Jul 07 19:00:38.470872 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:38.470341 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h8m9b" Jul 07 19:00:38.470872 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:00:38.470519 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lk7md" podUID="59f0c8fc-bc9a-4273-aac2-a8ee23bf6b63" Jul 07 19:00:38.470872 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:00:38.470624 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h8m9b" podUID="396810c7-8961-4755-b0f6-02a9d7e02ae3" Jul 07 19:00:39.575012 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:39.574946 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-bjzmq" event={"ID":"cbf5b271-d24c-47eb-9971-3bab86804424","Type":"ContainerStarted","Data":"1e4be0916cf294450dfa351b7a6013a2b45313dfdc9b30d6e180bcd73259bf01"} Jul 07 19:00:39.576983 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:39.576772 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-x7d7k" event={"ID":"a1221769-6ea8-43b7-aa11-e4b2d3699a7b","Type":"ContainerStarted","Data":"4ed17e3463ece5c5c19224c60177f95e8bbaae423bcbd5d1c4c6ab903494bda0"} Jul 07 19:00:39.579554 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:39.578945 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zzwbq_4f0c5d5c-147f-426a-965d-8886e5e79984/ovn-acl-logging/0.log" Jul 07 19:00:39.580237 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:39.579830 2563 generic.go:356] "Generic (PLEG): container finished" podID="4f0c5d5c-147f-426a-965d-8886e5e79984" containerID="fdf9bcfa9b519b00546cd2af216b8f46200511ea16ef09be05a65db88742c143" exitCode=1 Jul 07 19:00:39.580237 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:39.579927 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zzwbq" event={"ID":"4f0c5d5c-147f-426a-965d-8886e5e79984","Type":"ContainerDied","Data":"fdf9bcfa9b519b00546cd2af216b8f46200511ea16ef09be05a65db88742c143"} Jul 07 19:00:39.580237 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:39.580081 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zzwbq" event={"ID":"4f0c5d5c-147f-426a-965d-8886e5e79984","Type":"ContainerStarted","Data":"29350960ee7e896879f99c81e7fde92309a249322f1cdf0f3131947efaab0f53"} Jul 07 19:00:39.585464 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:39.585439 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-134.ec2.internal" event={"ID":"d64b863190df7f2a681aceffe233fdc3","Type":"ContainerStarted","Data":"11a828983f71d4664a8b675aeb44dc3877b2451b8aedb3c4627c5a1cab70dee7"} Jul 07 19:00:39.597132 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:39.597070 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-bjzmq" podStartSLOduration=2.823739525 podStartE2EDuration="22.597056361s" totalImagesPullingTime="19.773316836s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 19:00:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:00:39.596882922 +0000 UTC m=+22.612887104" watchObservedRunningTime="2026-07-07 19:00:39.597056361 +0000 UTC m=+22.613060541" Jul 07 19:00:39.621776 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:39.621724 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-x7d7k" podStartSLOduration=2.478385291 podStartE2EDuration="22.62170938s" totalImagesPullingTime="20.143324089s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 19:00:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:00:39.621232305 +0000 UTC m=+22.637236487" watchObservedRunningTime="2026-07-07 19:00:39.62170938 +0000 UTC m=+22.637713561" Jul 07 19:00:40.470186 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:40.470112 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lk7md" Jul 07 19:00:40.470186 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:40.470141 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h8m9b" Jul 07 19:00:40.470406 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:00:40.470234 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lk7md" podUID="59f0c8fc-bc9a-4273-aac2-a8ee23bf6b63" Jul 07 19:00:40.470406 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:40.470285 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d8t2d" Jul 07 19:00:40.470489 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:00:40.470398 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h8m9b" podUID="396810c7-8961-4755-b0f6-02a9d7e02ae3" Jul 07 19:00:40.470523 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:00:40.470499 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d8t2d" podUID="1d84b355-0ac7-4d14-83b6-7c18077e9319" Jul 07 19:00:40.588835 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:40.588800 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-dsvgs" event={"ID":"4f25fdbd-3251-4ef6-ab69-e87cc915a469","Type":"ContainerStarted","Data":"61032d876941c25e51792e59e0cbc49fd467f8afc9eae3af9832c1e317ccc9f7"} Jul 07 19:00:40.591722 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:40.591575 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zzwbq_4f0c5d5c-147f-426a-965d-8886e5e79984/ovn-acl-logging/0.log" Jul 07 19:00:40.592098 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:40.592079 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zzwbq" event={"ID":"4f0c5d5c-147f-426a-965d-8886e5e79984","Type":"ContainerStarted","Data":"93dc735f45682973ea3b8f005b84b2ab158041ebf0736c091c94b034e8786635"} Jul 07 19:00:40.592182 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:40.592165 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zzwbq" event={"ID":"4f0c5d5c-147f-426a-965d-8886e5e79984","Type":"ContainerStarted","Data":"bbb40d2fe30e62e434db17b95bc5ccc3358676696ba87a5fc27ab0f2e575f38c"} Jul 07 19:00:40.592235 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:40.592216 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zzwbq" event={"ID":"4f0c5d5c-147f-426a-965d-8886e5e79984","Type":"ContainerStarted","Data":"08cfd0afed0bf74469ceffb5afe4c4d97f0c5aef38fb1f14214021afb5562904"} Jul 07 19:00:40.592286 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:40.592252 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zzwbq" event={"ID":"4f0c5d5c-147f-426a-965d-8886e5e79984","Type":"ContainerStarted","Data":"59b489cf1a85cabb0864c0f4c694d00a6f202a0fe858bd0c4b196c8dc7d4bba6"} Jul 07 19:00:40.593553 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:40.593531 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-8ksxw" event={"ID":"63c4ef25-f98e-4224-ad6c-1eb0267d23df","Type":"ContainerStarted","Data":"9d089858942c6474ec7c2519a4c2a5a4c0b165d1f46fa0384d9160c8896e33a3"} Jul 07 19:00:40.594981 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:40.594930 2563 generic.go:356] "Generic (PLEG): container finished" podID="067754c774fbc99720cce604846fd8f3" containerID="ef24971c8018c448de02157f932cb025688aeaa1172f4c6e222e3e689b0f64ad" exitCode=0 Jul 07 19:00:40.595073 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:40.595017 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-134.ec2.internal" event={"ID":"067754c774fbc99720cce604846fd8f3","Type":"ContainerDied","Data":"ef24971c8018c448de02157f932cb025688aeaa1172f4c6e222e3e689b0f64ad"} Jul 07 19:00:40.597292 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:40.597264 2563 generic.go:356] "Generic (PLEG): container finished" podID="794c654b-f09b-4936-b01f-eccd5e7a2a95" containerID="fff54ed1e8ace12fa95580733bba427d37425ef1165a02de9709e93f57528221" exitCode=0 Jul 07 19:00:40.597385 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:40.597331 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jls8k" event={"ID":"794c654b-f09b-4936-b01f-eccd5e7a2a95","Type":"ContainerDied","Data":"fff54ed1e8ace12fa95580733bba427d37425ef1165a02de9709e93f57528221"} Jul 07 19:00:40.599448 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:40.599427 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qkmgh" event={"ID":"d6c3675f-6173-4275-8da8-3af895093d8a","Type":"ContainerStarted","Data":"ed91a3a2539e4626dff1825d7a2f6aae2542a0059c352927466ff6b44d3a4f49"} Jul 07 19:00:40.600735 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:40.600710 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-s272w" event={"ID":"a6abe188-2b1b-427a-ba0c-e93930187c7e","Type":"ContainerStarted","Data":"b3ebf553367c974b05620eec0f5dc18f7dcfaace5384ba021f67547a377880e4"} Jul 07 19:00:40.602030 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:40.602009 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-whw2v" event={"ID":"3953dadf-98ed-4933-a330-1191906b6cff","Type":"ContainerStarted","Data":"d365669425b7d6b78c632bed758368a1e2c45710a9cd841ba87ea87de179f830"} Jul 07 19:00:40.605160 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:40.605130 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-134.ec2.internal" podStartSLOduration=22.6051096 podStartE2EDuration="22.6051096s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 19:00:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:00:39.648309304 +0000 UTC m=+22.664313496" watchObservedRunningTime="2026-07-07 19:00:40.6051096 +0000 UTC m=+23.621113789" Jul 07 19:00:40.605496 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:40.605475 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-dsvgs" podStartSLOduration=12.026966731 podStartE2EDuration="23.605467627s" totalImagesPullingTime="11.578500896s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:00:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:00:40.604891191 +0000 UTC m=+23.620895374" watchObservedRunningTime="2026-07-07 19:00:40.605467627 +0000 UTC m=+23.621471808" Jul 07 19:00:40.622437 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:40.621394 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-whw2v" podStartSLOduration=3.737832313 podStartE2EDuration="23.621374999s" totalImagesPullingTime="19.883542686s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 19:00:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:00:40.62010667 +0000 UTC m=+23.636110853" watchObservedRunningTime="2026-07-07 19:00:40.621374999 +0000 UTC m=+23.637379181" Jul 07 19:00:40.639191 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:40.639122 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-8ksxw" podStartSLOduration=3.821891462 podStartE2EDuration="23.639106561s" totalImagesPullingTime="19.817215099s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 19:00:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:00:40.638463212 +0000 UTC m=+23.654467394" watchObservedRunningTime="2026-07-07 19:00:40.639106561 +0000 UTC m=+23.655110744" Jul 07 19:00:40.692237 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:40.692202 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-s272w" podStartSLOduration=3.8622847719999998 podStartE2EDuration="23.692184725s" totalImagesPullingTime="19.829899953s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 19:00:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:00:40.692109015 +0000 UTC m=+23.708113197" watchObservedRunningTime="2026-07-07 19:00:40.692184725 +0000 UTC m=+23.708188905" Jul 07 19:00:41.423795 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:41.423757 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-dsvgs" Jul 07 19:00:41.424432 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:41.424417 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-dsvgs" Jul 07 19:00:41.605580 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:41.605544 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-134.ec2.internal" event={"ID":"067754c774fbc99720cce604846fd8f3","Type":"ContainerStarted","Data":"fa0de76188125159e9f40fb5c90672f0dacad1eb6131f891bf3e99ef08d1d5b7"} Jul 07 19:00:41.606185 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:41.606158 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-dsvgs" Jul 07 19:00:41.606364 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:41.606348 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-dsvgs" Jul 07 19:00:41.641117 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:41.641073 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-134.ec2.internal" podStartSLOduration=23.641059345 podStartE2EDuration="23.641059345s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 19:00:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:00:41.625481349 +0000 UTC m=+24.641485531" watchObservedRunningTime="2026-07-07 19:00:41.641059345 +0000 UTC m=+24.657063525" Jul 07 19:00:42.470545 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:42.470509 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lk7md" Jul 07 19:00:42.470858 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:42.470605 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h8m9b" Jul 07 19:00:42.470858 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:00:42.470613 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lk7md" podUID="59f0c8fc-bc9a-4273-aac2-a8ee23bf6b63" Jul 07 19:00:42.470858 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:00:42.470697 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h8m9b" podUID="396810c7-8961-4755-b0f6-02a9d7e02ae3" Jul 07 19:00:42.470858 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:42.470730 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d8t2d" Jul 07 19:00:42.470858 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:00:42.470831 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d8t2d" podUID="1d84b355-0ac7-4d14-83b6-7c18077e9319" Jul 07 19:00:42.609784 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:42.609733 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qkmgh" event={"ID":"d6c3675f-6173-4275-8da8-3af895093d8a","Type":"ContainerStarted","Data":"a023789e323702c6c54ade286eddb65825e275ef632c0b696f46977783a70956"} Jul 07 19:00:42.614443 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:42.614402 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zzwbq_4f0c5d5c-147f-426a-965d-8886e5e79984/ovn-acl-logging/0.log" Jul 07 19:00:42.615068 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:42.614907 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zzwbq" event={"ID":"4f0c5d5c-147f-426a-965d-8886e5e79984","Type":"ContainerStarted","Data":"009a711c2c5274e7cf0f34a3d4b9fd5578a7237dbea6bb75a30579bb1cfa8d90"} Jul 07 19:00:42.616474 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:42.616392 2563 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 07 19:00:43.477851 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:43.477767 2563 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-07T19:00:42.616407605Z","UUID":"663d0b24-92e9-43d9-b910-21ae6a3ae9f6","Handler":null,"Name":"","Endpoint":""} Jul 07 19:00:43.479528 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:43.479504 2563 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 07 19:00:43.479667 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:43.479544 2563 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 07 19:00:44.470734 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:44.470492 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h8m9b" Jul 07 19:00:44.471330 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:44.470492 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lk7md" Jul 07 19:00:44.471330 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:00:44.470787 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h8m9b" podUID="396810c7-8961-4755-b0f6-02a9d7e02ae3" Jul 07 19:00:44.471330 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:44.470522 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d8t2d" Jul 07 19:00:44.471330 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:00:44.470854 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lk7md" podUID="59f0c8fc-bc9a-4273-aac2-a8ee23bf6b63" Jul 07 19:00:44.471330 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:00:44.470909 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d8t2d" podUID="1d84b355-0ac7-4d14-83b6-7c18077e9319" Jul 07 19:00:44.621441 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:44.621406 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qkmgh" event={"ID":"d6c3675f-6173-4275-8da8-3af895093d8a","Type":"ContainerStarted","Data":"c5ffe7a3980d585735650bfa3823ae8c24f57cef582e159e9aa2dfe68b5ac8c0"} Jul 07 19:00:44.624481 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:44.624462 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zzwbq_4f0c5d5c-147f-426a-965d-8886e5e79984/ovn-acl-logging/0.log" Jul 07 19:00:44.624938 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:44.624916 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zzwbq" event={"ID":"4f0c5d5c-147f-426a-965d-8886e5e79984","Type":"ContainerStarted","Data":"faf0cae748944ce05e97b41ebe35127015bf16b749fd7998f8fd9196e343151a"} Jul 07 19:00:44.625165 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:44.625145 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-zzwbq" Jul 07 19:00:44.625262 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:44.625245 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-zzwbq" Jul 07 19:00:44.625393 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:44.625375 2563 scope.go:117] "RemoveContainer" containerID="fdf9bcfa9b519b00546cd2af216b8f46200511ea16ef09be05a65db88742c143" Jul 07 19:00:44.640275 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:44.640229 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qkmgh" podStartSLOduration=3.985128096 podStartE2EDuration="27.640211606s" totalImagesPullingTime="23.65508351s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 19:00:17 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:00:44.639713246 +0000 UTC m=+27.655717428" watchObservedRunningTime="2026-07-07 19:00:44.640211606 +0000 UTC m=+27.656215788" Jul 07 19:00:44.641184 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:44.641168 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-zzwbq" Jul 07 19:00:45.631296 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:45.631265 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zzwbq_4f0c5d5c-147f-426a-965d-8886e5e79984/ovn-acl-logging/0.log" Jul 07 19:00:45.631818 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:45.631673 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zzwbq" event={"ID":"4f0c5d5c-147f-426a-965d-8886e5e79984","Type":"ContainerStarted","Data":"a4982688d58233e5eb13cdf88ac45bca8b8732a38ab27129e2ef666dbe276d81"} Jul 07 19:00:45.632159 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:45.632134 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-zzwbq" Jul 07 19:00:45.647640 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:45.647614 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-zzwbq" Jul 07 19:00:45.660518 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:45.660469 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-zzwbq" podStartSLOduration=8.171551111 podStartE2EDuration="28.66045481s" totalImagesPullingTime="20.488903699s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 19:00:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:00:45.660064754 +0000 UTC m=+28.676068937" watchObservedRunningTime="2026-07-07 19:00:45.66045481 +0000 UTC m=+28.676458991" Jul 07 19:00:46.083989 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:46.083931 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-lk7md"] Jul 07 19:00:46.084142 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:46.084082 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lk7md" Jul 07 19:00:46.084223 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:00:46.084201 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lk7md" podUID="59f0c8fc-bc9a-4273-aac2-a8ee23bf6b63" Jul 07 19:00:46.087226 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:46.087194 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-h8m9b"] Jul 07 19:00:46.088230 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:46.088208 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h8m9b" Jul 07 19:00:46.088423 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:00:46.088398 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h8m9b" podUID="396810c7-8961-4755-b0f6-02a9d7e02ae3" Jul 07 19:00:46.090555 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:46.090209 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-d8t2d"] Jul 07 19:00:46.090555 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:46.090360 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d8t2d" Jul 07 19:00:46.090555 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:00:46.090470 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d8t2d" podUID="1d84b355-0ac7-4d14-83b6-7c18077e9319" Jul 07 19:00:47.471190 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:47.471161 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lk7md" Jul 07 19:00:47.471742 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:47.471270 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h8m9b" Jul 07 19:00:47.471742 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:00:47.471273 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lk7md" podUID="59f0c8fc-bc9a-4273-aac2-a8ee23bf6b63" Jul 07 19:00:47.471742 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:00:47.471356 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h8m9b" podUID="396810c7-8961-4755-b0f6-02a9d7e02ae3" Jul 07 19:00:48.470427 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:48.470261 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d8t2d" Jul 07 19:00:48.470551 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:00:48.470506 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d8t2d" podUID="1d84b355-0ac7-4d14-83b6-7c18077e9319" Jul 07 19:00:48.640377 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:48.640341 2563 generic.go:356] "Generic (PLEG): container finished" podID="794c654b-f09b-4936-b01f-eccd5e7a2a95" containerID="f746ba344be4b1fb73d1c9e958975d0a46b9379d7867df36c1f57802154830cf" exitCode=0 Jul 07 19:00:48.640792 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:48.640400 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jls8k" event={"ID":"794c654b-f09b-4936-b01f-eccd5e7a2a95","Type":"ContainerDied","Data":"f746ba344be4b1fb73d1c9e958975d0a46b9379d7867df36c1f57802154830cf"} Jul 07 19:00:49.471095 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:49.471057 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lk7md" Jul 07 19:00:49.471305 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:49.471069 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h8m9b" Jul 07 19:00:49.471305 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:00:49.471286 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h8m9b" podUID="396810c7-8961-4755-b0f6-02a9d7e02ae3" Jul 07 19:00:49.471393 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:00:49.471162 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lk7md" podUID="59f0c8fc-bc9a-4273-aac2-a8ee23bf6b63" Jul 07 19:00:50.159148 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:50.159099 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/396810c7-8961-4755-b0f6-02a9d7e02ae3-metrics-certs\") pod \"network-metrics-daemon-h8m9b\" (UID: \"396810c7-8961-4755-b0f6-02a9d7e02ae3\") " pod="openshift-multus/network-metrics-daemon-h8m9b" Jul 07 19:00:50.159536 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:00:50.159242 2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 19:00:50.159536 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:00:50.159302 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/396810c7-8961-4755-b0f6-02a9d7e02ae3-metrics-certs podName:396810c7-8961-4755-b0f6-02a9d7e02ae3 nodeName:}" failed. No retries permitted until 2026-07-07 19:01:22.15928562 +0000 UTC m=+65.175289784 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/396810c7-8961-4755-b0f6-02a9d7e02ae3-metrics-certs") pod "network-metrics-daemon-h8m9b" (UID: "396810c7-8961-4755-b0f6-02a9d7e02ae3") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 19:00:50.259698 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:50.259660 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/59f0c8fc-bc9a-4273-aac2-a8ee23bf6b63-original-pull-secret\") pod \"global-pull-secret-syncer-lk7md\" (UID: \"59f0c8fc-bc9a-4273-aac2-a8ee23bf6b63\") " pod="kube-system/global-pull-secret-syncer-lk7md" Jul 07 19:00:50.259857 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:50.259707 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgqkm\" (UniqueName: \"kubernetes.io/projected/1d84b355-0ac7-4d14-83b6-7c18077e9319-kube-api-access-bgqkm\") pod \"network-check-target-d8t2d\" (UID: \"1d84b355-0ac7-4d14-83b6-7c18077e9319\") " pod="openshift-network-diagnostics/network-check-target-d8t2d" Jul 07 19:00:50.259857 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:00:50.259800 2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 07 19:00:50.259929 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:00:50.259861 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/59f0c8fc-bc9a-4273-aac2-a8ee23bf6b63-original-pull-secret podName:59f0c8fc-bc9a-4273-aac2-a8ee23bf6b63 nodeName:}" failed. No retries permitted until 2026-07-07 19:01:22.259844812 +0000 UTC m=+65.275848971 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/59f0c8fc-bc9a-4273-aac2-a8ee23bf6b63-original-pull-secret") pod "global-pull-secret-syncer-lk7md" (UID: "59f0c8fc-bc9a-4273-aac2-a8ee23bf6b63") : object "kube-system"/"original-pull-secret" not registered Jul 07 19:00:50.259929 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:00:50.259811 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 07 19:00:50.259929 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:00:50.259895 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 07 19:00:50.259929 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:00:50.259905 2563 projected.go:196] Error preparing data for projected volume kube-api-access-bgqkm for pod openshift-network-diagnostics/network-check-target-d8t2d: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 19:00:50.260074 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:00:50.259945 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1d84b355-0ac7-4d14-83b6-7c18077e9319-kube-api-access-bgqkm podName:1d84b355-0ac7-4d14-83b6-7c18077e9319 nodeName:}" failed. No retries permitted until 2026-07-07 19:01:22.259934549 +0000 UTC m=+65.275938707 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-bgqkm" (UniqueName: "kubernetes.io/projected/1d84b355-0ac7-4d14-83b6-7c18077e9319-kube-api-access-bgqkm") pod "network-check-target-d8t2d" (UID: "1d84b355-0ac7-4d14-83b6-7c18077e9319") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 19:00:50.470438 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:50.470354 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d8t2d" Jul 07 19:00:50.470571 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:00:50.470458 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d8t2d" podUID="1d84b355-0ac7-4d14-83b6-7c18077e9319" Jul 07 19:00:50.646818 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:50.646789 2563 generic.go:356] "Generic (PLEG): container finished" podID="794c654b-f09b-4936-b01f-eccd5e7a2a95" containerID="0b0bf2b205c49564919238a52d6bd69f1d5316c73522daf98b1a245c3c19ce8e" exitCode=0 Jul 07 19:00:50.646992 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:50.646848 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jls8k" event={"ID":"794c654b-f09b-4936-b01f-eccd5e7a2a95","Type":"ContainerDied","Data":"0b0bf2b205c49564919238a52d6bd69f1d5316c73522daf98b1a245c3c19ce8e"} Jul 07 19:00:51.473500 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:51.473473 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lk7md" Jul 07 19:00:51.473944 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:00:51.473565 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lk7md" podUID="59f0c8fc-bc9a-4273-aac2-a8ee23bf6b63" Jul 07 19:00:51.473944 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:51.473601 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h8m9b" Jul 07 19:00:51.473944 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:00:51.473663 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h8m9b" podUID="396810c7-8961-4755-b0f6-02a9d7e02ae3" Jul 07 19:00:52.470397 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:52.470355 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d8t2d" Jul 07 19:00:52.470579 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:00:52.470482 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d8t2d" podUID="1d84b355-0ac7-4d14-83b6-7c18077e9319" Jul 07 19:00:52.656699 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:52.656665 2563 generic.go:356] "Generic (PLEG): container finished" podID="794c654b-f09b-4936-b01f-eccd5e7a2a95" containerID="3f5e75f3b91c3219aa1e1ac69c8b17108ad55eb26edc026b1217c0f9338a4401" exitCode=0 Jul 07 19:00:52.657144 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:52.656717 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jls8k" event={"ID":"794c654b-f09b-4936-b01f-eccd5e7a2a95","Type":"ContainerDied","Data":"3f5e75f3b91c3219aa1e1ac69c8b17108ad55eb26edc026b1217c0f9338a4401"} Jul 07 19:00:52.815303 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:52.815276 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-134.ec2.internal" event="NodeReady" Jul 07 19:00:52.815467 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:52.815391 2563 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 07 19:00:52.855888 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:52.855862 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5d5c5c78bb-kzvl6"] Jul 07 19:00:52.858644 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:52.858629 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5d5c5c78bb-kzvl6" Jul 07 19:00:52.862806 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:52.862783 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-8tx7z\"" Jul 07 19:00:52.862995 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:52.862980 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 07 19:00:52.863087 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:52.863062 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 07 19:00:52.863290 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:52.863277 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 07 19:00:52.872264 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:52.872244 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 07 19:00:52.874500 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:52.874477 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5d5c5c78bb-kzvl6"] Jul 07 19:00:52.875209 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:52.875157 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-hq4fc"] Jul 07 19:00:52.878074 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:52.878056 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-28nms"] Jul 07 19:00:52.878216 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:52.878200 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-hq4fc" Jul 07 19:00:52.879997 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:52.879981 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-d4n46\"" Jul 07 19:00:52.880093 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:52.879979 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 07 19:00:52.880093 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:52.880043 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 07 19:00:52.880602 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:52.880585 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-28nms" Jul 07 19:00:52.882594 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:52.882574 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 07 19:00:52.883096 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:52.883078 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 07 19:00:52.883594 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:52.883575 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 07 19:00:52.883796 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:52.883782 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-hwc8x\"" Jul 07 19:00:52.889392 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:52.889369 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-hq4fc"] Jul 07 19:00:52.892403 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:52.892382 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-28nms"] Jul 07 19:00:52.981167 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:52.981110 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1e7deba9-69b3-4260-b6d0-037f274fb311-registry-certificates\") pod \"image-registry-5d5c5c78bb-kzvl6\" (UID: \"1e7deba9-69b3-4260-b6d0-037f274fb311\") " pod="openshift-image-registry/image-registry-5d5c5c78bb-kzvl6" Jul 07 19:00:52.981167 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:52.981175 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c338a925-fe6b-40a0-b669-ce49a20cc370-metrics-tls\") pod \"dns-default-hq4fc\" (UID: \"c338a925-fe6b-40a0-b669-ce49a20cc370\") " pod="openshift-dns/dns-default-hq4fc" Jul 07 19:00:52.981397 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:52.981199 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1e7deba9-69b3-4260-b6d0-037f274fb311-trusted-ca\") pod \"image-registry-5d5c5c78bb-kzvl6\" (UID: \"1e7deba9-69b3-4260-b6d0-037f274fb311\") " pod="openshift-image-registry/image-registry-5d5c5c78bb-kzvl6" Jul 07 19:00:52.981397 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:52.981238 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c338a925-fe6b-40a0-b669-ce49a20cc370-config-volume\") pod \"dns-default-hq4fc\" (UID: \"c338a925-fe6b-40a0-b669-ce49a20cc370\") " pod="openshift-dns/dns-default-hq4fc" Jul 07 19:00:52.981397 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:52.981253 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c338a925-fe6b-40a0-b669-ce49a20cc370-tmp-dir\") pod \"dns-default-hq4fc\" (UID: \"c338a925-fe6b-40a0-b669-ce49a20cc370\") " pod="openshift-dns/dns-default-hq4fc" Jul 07 19:00:52.981397 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:52.981290 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a6f2e7e7-5e73-40a1-a05a-d5a338128dd7-cert\") pod \"ingress-canary-28nms\" (UID: \"a6f2e7e7-5e73-40a1-a05a-d5a338128dd7\") " pod="openshift-ingress-canary/ingress-canary-28nms" Jul 07 19:00:52.981397 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:52.981304 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1e7deba9-69b3-4260-b6d0-037f274fb311-bound-sa-token\") pod \"image-registry-5d5c5c78bb-kzvl6\" (UID: \"1e7deba9-69b3-4260-b6d0-037f274fb311\") " pod="openshift-image-registry/image-registry-5d5c5c78bb-kzvl6" Jul 07 19:00:52.981397 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:52.981319 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lk5z4\" (UniqueName: \"kubernetes.io/projected/1e7deba9-69b3-4260-b6d0-037f274fb311-kube-api-access-lk5z4\") pod \"image-registry-5d5c5c78bb-kzvl6\" (UID: \"1e7deba9-69b3-4260-b6d0-037f274fb311\") " pod="openshift-image-registry/image-registry-5d5c5c78bb-kzvl6" Jul 07 19:00:52.981397 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:52.981343 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1e7deba9-69b3-4260-b6d0-037f274fb311-ca-trust-extracted\") pod \"image-registry-5d5c5c78bb-kzvl6\" (UID: \"1e7deba9-69b3-4260-b6d0-037f274fb311\") " pod="openshift-image-registry/image-registry-5d5c5c78bb-kzvl6" Jul 07 19:00:52.981397 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:52.981357 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1e7deba9-69b3-4260-b6d0-037f274fb311-installation-pull-secrets\") pod \"image-registry-5d5c5c78bb-kzvl6\" (UID: \"1e7deba9-69b3-4260-b6d0-037f274fb311\") " pod="openshift-image-registry/image-registry-5d5c5c78bb-kzvl6" Jul 07 19:00:52.981397 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:52.981380 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5dh8x\" (UniqueName: \"kubernetes.io/projected/a6f2e7e7-5e73-40a1-a05a-d5a338128dd7-kube-api-access-5dh8x\") pod \"ingress-canary-28nms\" (UID: \"a6f2e7e7-5e73-40a1-a05a-d5a338128dd7\") " pod="openshift-ingress-canary/ingress-canary-28nms" Jul 07 19:00:52.981397 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:52.981396 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1e7deba9-69b3-4260-b6d0-037f274fb311-image-registry-private-configuration\") pod \"image-registry-5d5c5c78bb-kzvl6\" (UID: \"1e7deba9-69b3-4260-b6d0-037f274fb311\") " pod="openshift-image-registry/image-registry-5d5c5c78bb-kzvl6" Jul 07 19:00:52.981685 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:52.981410 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1e7deba9-69b3-4260-b6d0-037f274fb311-registry-tls\") pod \"image-registry-5d5c5c78bb-kzvl6\" (UID: \"1e7deba9-69b3-4260-b6d0-037f274fb311\") " pod="openshift-image-registry/image-registry-5d5c5c78bb-kzvl6" Jul 07 19:00:52.981685 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:52.981428 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lb5c9\" (UniqueName: \"kubernetes.io/projected/c338a925-fe6b-40a0-b669-ce49a20cc370-kube-api-access-lb5c9\") pod \"dns-default-hq4fc\" (UID: \"c338a925-fe6b-40a0-b669-ce49a20cc370\") " pod="openshift-dns/dns-default-hq4fc" Jul 07 19:00:53.081192 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:53.081105 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:00:53.082021 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:53.081991 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1e7deba9-69b3-4260-b6d0-037f274fb311-installation-pull-secrets\") pod \"image-registry-5d5c5c78bb-kzvl6\" (UID: \"1e7deba9-69b3-4260-b6d0-037f274fb311\") " pod="openshift-image-registry/image-registry-5d5c5c78bb-kzvl6" Jul 07 19:00:53.082077 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:53.082047 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dh8x\" (UniqueName: \"kubernetes.io/projected/a6f2e7e7-5e73-40a1-a05a-d5a338128dd7-kube-api-access-5dh8x\") pod \"ingress-canary-28nms\" (UID: \"a6f2e7e7-5e73-40a1-a05a-d5a338128dd7\") " pod="openshift-ingress-canary/ingress-canary-28nms" Jul 07 19:00:53.082077 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:53.082065 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1e7deba9-69b3-4260-b6d0-037f274fb311-image-registry-private-configuration\") pod \"image-registry-5d5c5c78bb-kzvl6\" (UID: \"1e7deba9-69b3-4260-b6d0-037f274fb311\") " pod="openshift-image-registry/image-registry-5d5c5c78bb-kzvl6" Jul 07 19:00:53.082185 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:53.082131 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1e7deba9-69b3-4260-b6d0-037f274fb311-registry-tls\") pod \"image-registry-5d5c5c78bb-kzvl6\" (UID: \"1e7deba9-69b3-4260-b6d0-037f274fb311\") " pod="openshift-image-registry/image-registry-5d5c5c78bb-kzvl6" Jul 07 19:00:53.082185 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:53.082175 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lb5c9\" (UniqueName: \"kubernetes.io/projected/c338a925-fe6b-40a0-b669-ce49a20cc370-kube-api-access-lb5c9\") pod \"dns-default-hq4fc\" (UID: \"c338a925-fe6b-40a0-b669-ce49a20cc370\") " pod="openshift-dns/dns-default-hq4fc" Jul 07 19:00:53.082246 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:53.082195 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1e7deba9-69b3-4260-b6d0-037f274fb311-registry-certificates\") pod \"image-registry-5d5c5c78bb-kzvl6\" (UID: \"1e7deba9-69b3-4260-b6d0-037f274fb311\") " pod="openshift-image-registry/image-registry-5d5c5c78bb-kzvl6" Jul 07 19:00:53.082246 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:53.082228 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c338a925-fe6b-40a0-b669-ce49a20cc370-metrics-tls\") pod \"dns-default-hq4fc\" (UID: \"c338a925-fe6b-40a0-b669-ce49a20cc370\") " pod="openshift-dns/dns-default-hq4fc" Jul 07 19:00:53.082341 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:53.082254 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1e7deba9-69b3-4260-b6d0-037f274fb311-trusted-ca\") pod \"image-registry-5d5c5c78bb-kzvl6\" (UID: \"1e7deba9-69b3-4260-b6d0-037f274fb311\") " pod="openshift-image-registry/image-registry-5d5c5c78bb-kzvl6" Jul 07 19:00:53.082390 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:00:53.082369 2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 07 19:00:53.082438 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:00:53.082412 2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 07 19:00:53.082438 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:00:53.082431 2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5d5c5c78bb-kzvl6: secret "image-registry-tls" not found Jul 07 19:00:53.082525 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:00:53.082449 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c338a925-fe6b-40a0-b669-ce49a20cc370-metrics-tls podName:c338a925-fe6b-40a0-b669-ce49a20cc370 nodeName:}" failed. No retries permitted until 2026-07-07 19:00:53.582427915 +0000 UTC m=+36.598432093 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c338a925-fe6b-40a0-b669-ce49a20cc370-metrics-tls") pod "dns-default-hq4fc" (UID: "c338a925-fe6b-40a0-b669-ce49a20cc370") : secret "dns-default-metrics-tls" not found Jul 07 19:00:53.082525 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:00:53.082489 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1e7deba9-69b3-4260-b6d0-037f274fb311-registry-tls podName:1e7deba9-69b3-4260-b6d0-037f274fb311 nodeName:}" failed. No retries permitted until 2026-07-07 19:00:53.582473686 +0000 UTC m=+36.598477844 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1e7deba9-69b3-4260-b6d0-037f274fb311-registry-tls") pod "image-registry-5d5c5c78bb-kzvl6" (UID: "1e7deba9-69b3-4260-b6d0-037f274fb311") : secret "image-registry-tls" not found Jul 07 19:00:53.082525 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:53.082419 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c338a925-fe6b-40a0-b669-ce49a20cc370-config-volume\") pod \"dns-default-hq4fc\" (UID: \"c338a925-fe6b-40a0-b669-ce49a20cc370\") " pod="openshift-dns/dns-default-hq4fc" Jul 07 19:00:53.082750 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:53.082537 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c338a925-fe6b-40a0-b669-ce49a20cc370-tmp-dir\") pod \"dns-default-hq4fc\" (UID: \"c338a925-fe6b-40a0-b669-ce49a20cc370\") " pod="openshift-dns/dns-default-hq4fc" Jul 07 19:00:53.082800 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:53.082784 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a6f2e7e7-5e73-40a1-a05a-d5a338128dd7-cert\") pod \"ingress-canary-28nms\" (UID: \"a6f2e7e7-5e73-40a1-a05a-d5a338128dd7\") " pod="openshift-ingress-canary/ingress-canary-28nms" Jul 07 19:00:53.082843 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:53.082821 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1e7deba9-69b3-4260-b6d0-037f274fb311-bound-sa-token\") pod \"image-registry-5d5c5c78bb-kzvl6\" (UID: \"1e7deba9-69b3-4260-b6d0-037f274fb311\") " pod="openshift-image-registry/image-registry-5d5c5c78bb-kzvl6" Jul 07 19:00:53.082899 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:53.082849 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lk5z4\" (UniqueName: \"kubernetes.io/projected/1e7deba9-69b3-4260-b6d0-037f274fb311-kube-api-access-lk5z4\") pod \"image-registry-5d5c5c78bb-kzvl6\" (UID: \"1e7deba9-69b3-4260-b6d0-037f274fb311\") " pod="openshift-image-registry/image-registry-5d5c5c78bb-kzvl6" Jul 07 19:00:53.082899 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:53.082894 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1e7deba9-69b3-4260-b6d0-037f274fb311-ca-trust-extracted\") pod \"image-registry-5d5c5c78bb-kzvl6\" (UID: \"1e7deba9-69b3-4260-b6d0-037f274fb311\") " pod="openshift-image-registry/image-registry-5d5c5c78bb-kzvl6" Jul 07 19:00:53.083018 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:53.082918 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c338a925-fe6b-40a0-b669-ce49a20cc370-tmp-dir\") pod \"dns-default-hq4fc\" (UID: \"c338a925-fe6b-40a0-b669-ce49a20cc370\") " pod="openshift-dns/dns-default-hq4fc" Jul 07 19:00:53.083018 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:53.082979 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c338a925-fe6b-40a0-b669-ce49a20cc370-config-volume\") pod \"dns-default-hq4fc\" (UID: \"c338a925-fe6b-40a0-b669-ce49a20cc370\") " pod="openshift-dns/dns-default-hq4fc" Jul 07 19:00:53.083266 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:53.083242 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1e7deba9-69b3-4260-b6d0-037f274fb311-ca-trust-extracted\") pod \"image-registry-5d5c5c78bb-kzvl6\" (UID: \"1e7deba9-69b3-4260-b6d0-037f274fb311\") " pod="openshift-image-registry/image-registry-5d5c5c78bb-kzvl6" Jul 07 19:00:53.083474 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:53.083422 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1e7deba9-69b3-4260-b6d0-037f274fb311-trusted-ca\") pod \"image-registry-5d5c5c78bb-kzvl6\" (UID: \"1e7deba9-69b3-4260-b6d0-037f274fb311\") " pod="openshift-image-registry/image-registry-5d5c5c78bb-kzvl6" Jul 07 19:00:53.083474 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:53.083422 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1e7deba9-69b3-4260-b6d0-037f274fb311-registry-certificates\") pod \"image-registry-5d5c5c78bb-kzvl6\" (UID: \"1e7deba9-69b3-4260-b6d0-037f274fb311\") " pod="openshift-image-registry/image-registry-5d5c5c78bb-kzvl6" Jul 07 19:00:53.086398 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:53.086378 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a6f2e7e7-5e73-40a1-a05a-d5a338128dd7-cert\") pod \"ingress-canary-28nms\" (UID: \"a6f2e7e7-5e73-40a1-a05a-d5a338128dd7\") " pod="openshift-ingress-canary/ingress-canary-28nms" Jul 07 19:00:53.086475 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:53.086411 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1e7deba9-69b3-4260-b6d0-037f274fb311-image-registry-private-configuration\") pod \"image-registry-5d5c5c78bb-kzvl6\" (UID: \"1e7deba9-69b3-4260-b6d0-037f274fb311\") " pod="openshift-image-registry/image-registry-5d5c5c78bb-kzvl6" Jul 07 19:00:53.086475 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:53.086424 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1e7deba9-69b3-4260-b6d0-037f274fb311-installation-pull-secrets\") pod \"image-registry-5d5c5c78bb-kzvl6\" (UID: \"1e7deba9-69b3-4260-b6d0-037f274fb311\") " pod="openshift-image-registry/image-registry-5d5c5c78bb-kzvl6" Jul 07 19:00:53.093239 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:53.093212 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lb5c9\" (UniqueName: \"kubernetes.io/projected/c338a925-fe6b-40a0-b669-ce49a20cc370-kube-api-access-lb5c9\") pod \"dns-default-hq4fc\" (UID: \"c338a925-fe6b-40a0-b669-ce49a20cc370\") " pod="openshift-dns/dns-default-hq4fc" Jul 07 19:00:53.093373 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:53.093355 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dh8x\" (UniqueName: \"kubernetes.io/projected/a6f2e7e7-5e73-40a1-a05a-d5a338128dd7-kube-api-access-5dh8x\") pod \"ingress-canary-28nms\" (UID: \"a6f2e7e7-5e73-40a1-a05a-d5a338128dd7\") " pod="openshift-ingress-canary/ingress-canary-28nms" Jul 07 19:00:53.094192 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:53.094177 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lk5z4\" (UniqueName: \"kubernetes.io/projected/1e7deba9-69b3-4260-b6d0-037f274fb311-kube-api-access-lk5z4\") pod \"image-registry-5d5c5c78bb-kzvl6\" (UID: \"1e7deba9-69b3-4260-b6d0-037f274fb311\") " pod="openshift-image-registry/image-registry-5d5c5c78bb-kzvl6" Jul 07 19:00:53.094290 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:53.094271 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1e7deba9-69b3-4260-b6d0-037f274fb311-bound-sa-token\") pod \"image-registry-5d5c5c78bb-kzvl6\" (UID: \"1e7deba9-69b3-4260-b6d0-037f274fb311\") " pod="openshift-image-registry/image-registry-5d5c5c78bb-kzvl6" Jul 07 19:00:53.171439 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:53.171406 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:00:53.192576 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:53.192540 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-28nms" Jul 07 19:00:53.330327 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:53.330291 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-28nms"] Jul 07 19:00:53.334029 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:00:53.334002 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6f2e7e7_5e73_40a1_a05a_d5a338128dd7.slice/crio-3c9d67dbd41594a8fa7af0e59814ef482afea86d0bc984c6b8f348e97e8bd933 WatchSource:0}: Error finding container 3c9d67dbd41594a8fa7af0e59814ef482afea86d0bc984c6b8f348e97e8bd933: Status 404 returned error can't find the container with id 3c9d67dbd41594a8fa7af0e59814ef482afea86d0bc984c6b8f348e97e8bd933 Jul 07 19:00:53.474407 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:53.474384 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lk7md" Jul 07 19:00:53.474564 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:53.474384 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h8m9b" Jul 07 19:00:53.476631 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:53.476608 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-84rbs\"" Jul 07 19:00:53.476754 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:53.476724 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 07 19:00:53.477186 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:53.477166 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 07 19:00:53.557582 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:53.557541 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:00:53.587076 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:53.586994 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1e7deba9-69b3-4260-b6d0-037f274fb311-registry-tls\") pod \"image-registry-5d5c5c78bb-kzvl6\" (UID: \"1e7deba9-69b3-4260-b6d0-037f274fb311\") " pod="openshift-image-registry/image-registry-5d5c5c78bb-kzvl6" Jul 07 19:00:53.587076 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:53.587046 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c338a925-fe6b-40a0-b669-ce49a20cc370-metrics-tls\") pod \"dns-default-hq4fc\" (UID: \"c338a925-fe6b-40a0-b669-ce49a20cc370\") " pod="openshift-dns/dns-default-hq4fc" Jul 07 19:00:53.587305 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:00:53.587152 2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 07 19:00:53.587305 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:00:53.587160 2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 07 19:00:53.587305 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:00:53.587180 2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5d5c5c78bb-kzvl6: secret "image-registry-tls" not found Jul 07 19:00:53.587305 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:00:53.587209 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c338a925-fe6b-40a0-b669-ce49a20cc370-metrics-tls podName:c338a925-fe6b-40a0-b669-ce49a20cc370 nodeName:}" failed. No retries permitted until 2026-07-07 19:00:54.587192598 +0000 UTC m=+37.603196762 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c338a925-fe6b-40a0-b669-ce49a20cc370-metrics-tls") pod "dns-default-hq4fc" (UID: "c338a925-fe6b-40a0-b669-ce49a20cc370") : secret "dns-default-metrics-tls" not found Jul 07 19:00:53.587305 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:00:53.587223 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1e7deba9-69b3-4260-b6d0-037f274fb311-registry-tls podName:1e7deba9-69b3-4260-b6d0-037f274fb311 nodeName:}" failed. No retries permitted until 2026-07-07 19:00:54.587216733 +0000 UTC m=+37.603220892 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1e7deba9-69b3-4260-b6d0-037f274fb311-registry-tls") pod "image-registry-5d5c5c78bb-kzvl6" (UID: "1e7deba9-69b3-4260-b6d0-037f274fb311") : secret "image-registry-tls" not found Jul 07 19:00:53.644167 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:53.644140 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:00:53.659878 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:53.659848 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-28nms" event={"ID":"a6f2e7e7-5e73-40a1-a05a-d5a338128dd7","Type":"ContainerStarted","Data":"3c9d67dbd41594a8fa7af0e59814ef482afea86d0bc984c6b8f348e97e8bd933"} Jul 07 19:00:53.963113 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:53.963031 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:00:54.038474 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:54.038439 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:00:54.470498 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:54.470462 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d8t2d" Jul 07 19:00:54.473122 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:54.472893 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 07 19:00:54.473122 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:54.472982 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 07 19:00:54.473122 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:54.472994 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-6dffl\"" Jul 07 19:00:54.596887 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:54.596850 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1e7deba9-69b3-4260-b6d0-037f274fb311-registry-tls\") pod \"image-registry-5d5c5c78bb-kzvl6\" (UID: \"1e7deba9-69b3-4260-b6d0-037f274fb311\") " pod="openshift-image-registry/image-registry-5d5c5c78bb-kzvl6" Jul 07 19:00:54.597089 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:54.596906 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c338a925-fe6b-40a0-b669-ce49a20cc370-metrics-tls\") pod \"dns-default-hq4fc\" (UID: \"c338a925-fe6b-40a0-b669-ce49a20cc370\") " pod="openshift-dns/dns-default-hq4fc" Jul 07 19:00:54.597089 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:00:54.597062 2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 07 19:00:54.597089 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:00:54.597071 2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 07 19:00:54.597089 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:00:54.597091 2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5d5c5c78bb-kzvl6: secret "image-registry-tls" not found Jul 07 19:00:54.597304 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:00:54.597115 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c338a925-fe6b-40a0-b669-ce49a20cc370-metrics-tls podName:c338a925-fe6b-40a0-b669-ce49a20cc370 nodeName:}" failed. No retries permitted until 2026-07-07 19:00:56.597101893 +0000 UTC m=+39.613106051 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c338a925-fe6b-40a0-b669-ce49a20cc370-metrics-tls") pod "dns-default-hq4fc" (UID: "c338a925-fe6b-40a0-b669-ce49a20cc370") : secret "dns-default-metrics-tls" not found Jul 07 19:00:54.597304 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:00:54.597149 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1e7deba9-69b3-4260-b6d0-037f274fb311-registry-tls podName:1e7deba9-69b3-4260-b6d0-037f274fb311 nodeName:}" failed. No retries permitted until 2026-07-07 19:00:56.597133647 +0000 UTC m=+39.613137806 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1e7deba9-69b3-4260-b6d0-037f274fb311-registry-tls") pod "image-registry-5d5c5c78bb-kzvl6" (UID: "1e7deba9-69b3-4260-b6d0-037f274fb311") : secret "image-registry-tls" not found Jul 07 19:00:56.613494 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:56.613250 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1e7deba9-69b3-4260-b6d0-037f274fb311-registry-tls\") pod \"image-registry-5d5c5c78bb-kzvl6\" (UID: \"1e7deba9-69b3-4260-b6d0-037f274fb311\") " pod="openshift-image-registry/image-registry-5d5c5c78bb-kzvl6" Jul 07 19:00:56.613909 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:56.613524 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c338a925-fe6b-40a0-b669-ce49a20cc370-metrics-tls\") pod \"dns-default-hq4fc\" (UID: \"c338a925-fe6b-40a0-b669-ce49a20cc370\") " pod="openshift-dns/dns-default-hq4fc" Jul 07 19:00:56.613909 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:00:56.613399 2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 07 19:00:56.613909 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:00:56.613591 2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5d5c5c78bb-kzvl6: secret "image-registry-tls" not found Jul 07 19:00:56.613909 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:00:56.613664 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1e7deba9-69b3-4260-b6d0-037f274fb311-registry-tls podName:1e7deba9-69b3-4260-b6d0-037f274fb311 nodeName:}" failed. No retries permitted until 2026-07-07 19:01:00.613643632 +0000 UTC m=+43.629647794 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1e7deba9-69b3-4260-b6d0-037f274fb311-registry-tls") pod "image-registry-5d5c5c78bb-kzvl6" (UID: "1e7deba9-69b3-4260-b6d0-037f274fb311") : secret "image-registry-tls" not found Jul 07 19:00:56.613909 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:00:56.613703 2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 07 19:00:56.613909 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:00:56.613772 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c338a925-fe6b-40a0-b669-ce49a20cc370-metrics-tls podName:c338a925-fe6b-40a0-b669-ce49a20cc370 nodeName:}" failed. No retries permitted until 2026-07-07 19:01:00.61375455 +0000 UTC m=+43.629758722 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c338a925-fe6b-40a0-b669-ce49a20cc370-metrics-tls") pod "dns-default-hq4fc" (UID: "c338a925-fe6b-40a0-b669-ce49a20cc370") : secret "dns-default-metrics-tls" not found Jul 07 19:00:57.279861 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:57.279764 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:00:57.671113 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:57.671071 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-28nms" event={"ID":"a6f2e7e7-5e73-40a1-a05a-d5a338128dd7","Type":"ContainerStarted","Data":"764c87c023c5859b4f0f0dd2ab339f19993c466b24c747761f84fe7b3b23907e"} Jul 07 19:00:57.701069 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:57.701005 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-28nms" podStartSLOduration=2.272430531 podStartE2EDuration="5.700987923s" totalImagesPullingTime="3.428557392s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:00:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:00:57.700865088 +0000 UTC m=+40.716869281" watchObservedRunningTime="2026-07-07 19:00:57.700987923 +0000 UTC m=+40.716992108" Jul 07 19:00:57.745155 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:57.745117 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:00:57.846494 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:57.846459 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:00:57.941765 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:57.941676 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:00:59.679023 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:59.678990 2563 generic.go:356] "Generic (PLEG): container finished" podID="794c654b-f09b-4936-b01f-eccd5e7a2a95" containerID="6fa568a0ad0ae3f02c1f9472864e440ae95c09981ccc976ca52168c225848e9a" exitCode=0 Jul 07 19:00:59.679362 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:00:59.679062 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jls8k" event={"ID":"794c654b-f09b-4936-b01f-eccd5e7a2a95","Type":"ContainerDied","Data":"6fa568a0ad0ae3f02c1f9472864e440ae95c09981ccc976ca52168c225848e9a"} Jul 07 19:01:00.539353 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:00.539313 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:01:00.647418 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:00.647378 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1e7deba9-69b3-4260-b6d0-037f274fb311-registry-tls\") pod \"image-registry-5d5c5c78bb-kzvl6\" (UID: \"1e7deba9-69b3-4260-b6d0-037f274fb311\") " pod="openshift-image-registry/image-registry-5d5c5c78bb-kzvl6" Jul 07 19:01:00.647418 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:00.647426 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c338a925-fe6b-40a0-b669-ce49a20cc370-metrics-tls\") pod \"dns-default-hq4fc\" (UID: \"c338a925-fe6b-40a0-b669-ce49a20cc370\") " pod="openshift-dns/dns-default-hq4fc" Jul 07 19:01:00.647584 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:01:00.647526 2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 07 19:01:00.647584 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:01:00.647530 2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 07 19:01:00.647584 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:01:00.647550 2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5d5c5c78bb-kzvl6: secret "image-registry-tls" not found Jul 07 19:01:00.647584 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:01:00.647583 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c338a925-fe6b-40a0-b669-ce49a20cc370-metrics-tls podName:c338a925-fe6b-40a0-b669-ce49a20cc370 nodeName:}" failed. No retries permitted until 2026-07-07 19:01:08.647565836 +0000 UTC m=+51.663570012 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c338a925-fe6b-40a0-b669-ce49a20cc370-metrics-tls") pod "dns-default-hq4fc" (UID: "c338a925-fe6b-40a0-b669-ce49a20cc370") : secret "dns-default-metrics-tls" not found Jul 07 19:01:00.647779 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:01:00.647598 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1e7deba9-69b3-4260-b6d0-037f274fb311-registry-tls podName:1e7deba9-69b3-4260-b6d0-037f274fb311 nodeName:}" failed. No retries permitted until 2026-07-07 19:01:08.647591898 +0000 UTC m=+51.663596057 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1e7deba9-69b3-4260-b6d0-037f274fb311-registry-tls") pod "image-registry-5d5c5c78bb-kzvl6" (UID: "1e7deba9-69b3-4260-b6d0-037f274fb311") : secret "image-registry-tls" not found Jul 07 19:01:00.687335 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:00.687300 2563 generic.go:356] "Generic (PLEG): container finished" podID="794c654b-f09b-4936-b01f-eccd5e7a2a95" containerID="6f2c320240699f6d5907edd7116221057c3b3832d2ff3745cc80428caa333cf3" exitCode=0 Jul 07 19:01:00.687686 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:00.687369 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jls8k" event={"ID":"794c654b-f09b-4936-b01f-eccd5e7a2a95","Type":"ContainerDied","Data":"6f2c320240699f6d5907edd7116221057c3b3832d2ff3745cc80428caa333cf3"} Jul 07 19:01:01.694922 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:01.694882 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jls8k" event={"ID":"794c654b-f09b-4936-b01f-eccd5e7a2a95","Type":"ContainerStarted","Data":"4162b0580e12a6f6bbb04bcc082245ca53b8f0b76f67cd8b47920f7f143e493d"} Jul 07 19:01:01.720379 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:01.720288 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-jls8k" podStartSLOduration=8.651664714 podStartE2EDuration="44.720267062s" totalImagesPullingTime="36.068602348s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 19:00:17 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:01:01.719535012 +0000 UTC m=+44.735539194" watchObservedRunningTime="2026-07-07 19:01:01.720267062 +0000 UTC m=+44.736271243" Jul 07 19:01:01.849686 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:01.849653 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:01:01.950013 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:01.949908 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:01:02.051773 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:02.051735 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:01:06.334629 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:06.334593 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:01:06.435465 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:06.435433 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:01:08.706198 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:08.706165 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1e7deba9-69b3-4260-b6d0-037f274fb311-registry-tls\") pod \"image-registry-5d5c5c78bb-kzvl6\" (UID: \"1e7deba9-69b3-4260-b6d0-037f274fb311\") " pod="openshift-image-registry/image-registry-5d5c5c78bb-kzvl6" Jul 07 19:01:08.706198 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:08.706203 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c338a925-fe6b-40a0-b669-ce49a20cc370-metrics-tls\") pod \"dns-default-hq4fc\" (UID: \"c338a925-fe6b-40a0-b669-ce49a20cc370\") " pod="openshift-dns/dns-default-hq4fc" Jul 07 19:01:08.706619 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:01:08.706298 2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 07 19:01:08.706619 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:01:08.706303 2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 07 19:01:08.706619 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:01:08.706318 2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5d5c5c78bb-kzvl6: secret "image-registry-tls" not found Jul 07 19:01:08.706619 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:01:08.706346 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c338a925-fe6b-40a0-b669-ce49a20cc370-metrics-tls podName:c338a925-fe6b-40a0-b669-ce49a20cc370 nodeName:}" failed. No retries permitted until 2026-07-07 19:01:24.706332988 +0000 UTC m=+67.722337146 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c338a925-fe6b-40a0-b669-ce49a20cc370-metrics-tls") pod "dns-default-hq4fc" (UID: "c338a925-fe6b-40a0-b669-ce49a20cc370") : secret "dns-default-metrics-tls" not found Jul 07 19:01:08.706619 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:01:08.706360 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1e7deba9-69b3-4260-b6d0-037f274fb311-registry-tls podName:1e7deba9-69b3-4260-b6d0-037f274fb311 nodeName:}" failed. No retries permitted until 2026-07-07 19:01:24.706353173 +0000 UTC m=+67.722357332 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1e7deba9-69b3-4260-b6d0-037f274fb311-registry-tls") pod "image-registry-5d5c5c78bb-kzvl6" (UID: "1e7deba9-69b3-4260-b6d0-037f274fb311") : secret "image-registry-tls" not found Jul 07 19:01:10.073086 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:10.073056 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:01:10.151926 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:10.151894 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:01:10.959015 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:10.958977 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:01:11.065137 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:11.065098 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:01:11.827502 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:11.827470 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:01:11.967379 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:11.967339 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:01:12.747085 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:12.747050 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:01:12.847066 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:12.847034 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:01:17.647425 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:17.647400 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-zzwbq" Jul 07 19:01:17.817778 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:17.817721 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:01:17.914178 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:17.914093 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:01:21.269191 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:21.269155 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:01:21.346195 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:21.346159 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:01:22.201515 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:22.201462 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/396810c7-8961-4755-b0f6-02a9d7e02ae3-metrics-certs\") pod \"network-metrics-daemon-h8m9b\" (UID: \"396810c7-8961-4755-b0f6-02a9d7e02ae3\") " pod="openshift-multus/network-metrics-daemon-h8m9b" Jul 07 19:01:22.203837 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:22.203815 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 07 19:01:22.212273 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:01:22.212251 2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 07 19:01:22.212669 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:01:22.212634 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/396810c7-8961-4755-b0f6-02a9d7e02ae3-metrics-certs podName:396810c7-8961-4755-b0f6-02a9d7e02ae3 nodeName:}" failed. No retries permitted until 2026-07-07 19:02:26.212595486 +0000 UTC m=+129.228599659 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/396810c7-8961-4755-b0f6-02a9d7e02ae3-metrics-certs") pod "network-metrics-daemon-h8m9b" (UID: "396810c7-8961-4755-b0f6-02a9d7e02ae3") : secret "metrics-daemon-secret" not found Jul 07 19:01:22.302114 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:22.302063 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/59f0c8fc-bc9a-4273-aac2-a8ee23bf6b63-original-pull-secret\") pod \"global-pull-secret-syncer-lk7md\" (UID: \"59f0c8fc-bc9a-4273-aac2-a8ee23bf6b63\") " pod="kube-system/global-pull-secret-syncer-lk7md" Jul 07 19:01:22.302114 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:22.302128 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgqkm\" (UniqueName: \"kubernetes.io/projected/1d84b355-0ac7-4d14-83b6-7c18077e9319-kube-api-access-bgqkm\") pod \"network-check-target-d8t2d\" (UID: \"1d84b355-0ac7-4d14-83b6-7c18077e9319\") " pod="openshift-network-diagnostics/network-check-target-d8t2d" Jul 07 19:01:22.304493 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:22.304473 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 07 19:01:22.304575 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:22.304503 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 07 19:01:22.314610 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:22.314585 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 07 19:01:22.315458 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:22.315441 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/59f0c8fc-bc9a-4273-aac2-a8ee23bf6b63-original-pull-secret\") pod \"global-pull-secret-syncer-lk7md\" (UID: \"59f0c8fc-bc9a-4273-aac2-a8ee23bf6b63\") " pod="kube-system/global-pull-secret-syncer-lk7md" Jul 07 19:01:22.325028 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:22.325007 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bgqkm\" (UniqueName: \"kubernetes.io/projected/1d84b355-0ac7-4d14-83b6-7c18077e9319-kube-api-access-bgqkm\") pod \"network-check-target-d8t2d\" (UID: \"1d84b355-0ac7-4d14-83b6-7c18077e9319\") " pod="openshift-network-diagnostics/network-check-target-d8t2d" Jul 07 19:01:22.385312 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:22.385285 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-6dffl\"" Jul 07 19:01:22.393811 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:22.393765 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d8t2d" Jul 07 19:01:22.532935 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:22.532903 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-d8t2d"] Jul 07 19:01:22.536184 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:01:22.536153 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d84b355_0ac7_4d14_83b6_7c18077e9319.slice/crio-6b77933f947ad8812320993a1e5d7d9d38cd5dea2b7a53dec5fcebbf1d5280bf WatchSource:0}: Error finding container 6b77933f947ad8812320993a1e5d7d9d38cd5dea2b7a53dec5fcebbf1d5280bf: Status 404 returned error can't find the container with id 6b77933f947ad8812320993a1e5d7d9d38cd5dea2b7a53dec5fcebbf1d5280bf Jul 07 19:01:22.586282 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:22.586243 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lk7md" Jul 07 19:01:22.714797 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:22.714757 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-lk7md"] Jul 07 19:01:22.719311 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:01:22.719286 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59f0c8fc_bc9a_4273_aac2_a8ee23bf6b63.slice/crio-e66dcf9e6cbeec74e73d368192f03f4983b109b8ea64d92f03d1b65bd4883fa5 WatchSource:0}: Error finding container e66dcf9e6cbeec74e73d368192f03f4983b109b8ea64d92f03d1b65bd4883fa5: Status 404 returned error can't find the container with id e66dcf9e6cbeec74e73d368192f03f4983b109b8ea64d92f03d1b65bd4883fa5 Jul 07 19:01:22.743692 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:22.743600 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-lk7md" event={"ID":"59f0c8fc-bc9a-4273-aac2-a8ee23bf6b63","Type":"ContainerStarted","Data":"e66dcf9e6cbeec74e73d368192f03f4983b109b8ea64d92f03d1b65bd4883fa5"} Jul 07 19:01:22.744672 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:22.744651 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-d8t2d" event={"ID":"1d84b355-0ac7-4d14-83b6-7c18077e9319","Type":"ContainerStarted","Data":"6b77933f947ad8812320993a1e5d7d9d38cd5dea2b7a53dec5fcebbf1d5280bf"} Jul 07 19:01:24.719223 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:24.719187 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1e7deba9-69b3-4260-b6d0-037f274fb311-registry-tls\") pod \"image-registry-5d5c5c78bb-kzvl6\" (UID: \"1e7deba9-69b3-4260-b6d0-037f274fb311\") " pod="openshift-image-registry/image-registry-5d5c5c78bb-kzvl6" Jul 07 19:01:24.719591 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:24.719231 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c338a925-fe6b-40a0-b669-ce49a20cc370-metrics-tls\") pod \"dns-default-hq4fc\" (UID: \"c338a925-fe6b-40a0-b669-ce49a20cc370\") " pod="openshift-dns/dns-default-hq4fc" Jul 07 19:01:24.719591 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:01:24.719335 2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 07 19:01:24.719591 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:01:24.719362 2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5d5c5c78bb-kzvl6: secret "image-registry-tls" not found Jul 07 19:01:24.719591 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:01:24.719340 2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 07 19:01:24.719591 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:01:24.719414 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c338a925-fe6b-40a0-b669-ce49a20cc370-metrics-tls podName:c338a925-fe6b-40a0-b669-ce49a20cc370 nodeName:}" failed. No retries permitted until 2026-07-07 19:01:56.719400646 +0000 UTC m=+99.735404805 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c338a925-fe6b-40a0-b669-ce49a20cc370-metrics-tls") pod "dns-default-hq4fc" (UID: "c338a925-fe6b-40a0-b669-ce49a20cc370") : secret "dns-default-metrics-tls" not found Jul 07 19:01:24.719591 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:01:24.719426 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1e7deba9-69b3-4260-b6d0-037f274fb311-registry-tls podName:1e7deba9-69b3-4260-b6d0-037f274fb311 nodeName:}" failed. No retries permitted until 2026-07-07 19:01:56.719420461 +0000 UTC m=+99.735424620 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1e7deba9-69b3-4260-b6d0-037f274fb311-registry-tls") pod "image-registry-5d5c5c78bb-kzvl6" (UID: "1e7deba9-69b3-4260-b6d0-037f274fb311") : secret "image-registry-tls" not found Jul 07 19:01:25.502938 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:25.502901 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-68b757fb67-frt7c"] Jul 07 19:01:25.505068 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:25.505031 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-64786d9cbb-pnvkh"] Jul 07 19:01:25.505068 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:25.505048 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-68b757fb67-frt7c" Jul 07 19:01:25.506730 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:25.506702 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-pnvkh" Jul 07 19:01:25.507369 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:25.507349 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Jul 07 19:01:25.507483 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:25.507398 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Jul 07 19:01:25.507756 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:25.507738 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-xzdlr\"" Jul 07 19:01:25.507912 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:25.507895 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Jul 07 19:01:25.508110 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:25.508079 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Jul 07 19:01:25.509898 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:25.509880 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 07 19:01:25.510020 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:25.509926 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Jul 07 19:01:25.510573 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:25.510414 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Jul 07 19:01:25.510573 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:25.510414 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-bswzq\"" Jul 07 19:01:25.510573 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:25.510545 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 07 19:01:25.515944 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:25.515917 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Jul 07 19:01:25.516826 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:25.516809 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-68b757fb67-frt7c"] Jul 07 19:01:25.519831 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:25.519809 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-64786d9cbb-pnvkh"] Jul 07 19:01:25.526282 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:25.526254 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmthl\" (UniqueName: \"kubernetes.io/projected/ecead548-45c5-4195-95e8-a5c058d5958b-kube-api-access-wmthl\") pod \"cluster-monitoring-operator-64786d9cbb-pnvkh\" (UID: \"ecead548-45c5-4195-95e8-a5c058d5958b\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-pnvkh" Jul 07 19:01:25.526402 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:25.526299 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e25ab82f-72fb-4cde-89c4-0a0dd80f5feb-config\") pod \"console-operator-68b757fb67-frt7c\" (UID: \"e25ab82f-72fb-4cde-89c4-0a0dd80f5feb\") " pod="openshift-console-operator/console-operator-68b757fb67-frt7c" Jul 07 19:01:25.526402 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:25.526324 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e25ab82f-72fb-4cde-89c4-0a0dd80f5feb-trusted-ca\") pod \"console-operator-68b757fb67-frt7c\" (UID: \"e25ab82f-72fb-4cde-89c4-0a0dd80f5feb\") " pod="openshift-console-operator/console-operator-68b757fb67-frt7c" Jul 07 19:01:25.526514 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:25.526452 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/ecead548-45c5-4195-95e8-a5c058d5958b-telemetry-config\") pod \"cluster-monitoring-operator-64786d9cbb-pnvkh\" (UID: \"ecead548-45c5-4195-95e8-a5c058d5958b\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-pnvkh" Jul 07 19:01:25.526514 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:25.526498 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ecead548-45c5-4195-95e8-a5c058d5958b-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-pnvkh\" (UID: \"ecead548-45c5-4195-95e8-a5c058d5958b\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-pnvkh" Jul 07 19:01:25.526612 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:25.526547 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e25ab82f-72fb-4cde-89c4-0a0dd80f5feb-serving-cert\") pod \"console-operator-68b757fb67-frt7c\" (UID: \"e25ab82f-72fb-4cde-89c4-0a0dd80f5feb\") " pod="openshift-console-operator/console-operator-68b757fb67-frt7c" Jul 07 19:01:25.526612 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:25.526574 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nbmwd\" (UniqueName: \"kubernetes.io/projected/e25ab82f-72fb-4cde-89c4-0a0dd80f5feb-kube-api-access-nbmwd\") pod \"console-operator-68b757fb67-frt7c\" (UID: \"e25ab82f-72fb-4cde-89c4-0a0dd80f5feb\") " pod="openshift-console-operator/console-operator-68b757fb67-frt7c" Jul 07 19:01:25.601617 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:25.601585 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-g97sh"] Jul 07 19:01:25.603675 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:25.603652 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-g97sh" Jul 07 19:01:25.605716 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:25.605691 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Jul 07 19:01:25.605869 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:25.605749 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Jul 07 19:01:25.606146 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:25.606129 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Jul 07 19:01:25.606245 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:25.606148 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-kkqfg\"" Jul 07 19:01:25.615389 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:25.615362 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-g97sh"] Jul 07 19:01:25.627323 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:25.627284 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/ecead548-45c5-4195-95e8-a5c058d5958b-telemetry-config\") pod \"cluster-monitoring-operator-64786d9cbb-pnvkh\" (UID: \"ecead548-45c5-4195-95e8-a5c058d5958b\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-pnvkh" Jul 07 19:01:25.627479 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:25.627329 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ecead548-45c5-4195-95e8-a5c058d5958b-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-pnvkh\" (UID: \"ecead548-45c5-4195-95e8-a5c058d5958b\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-pnvkh" Jul 07 19:01:25.627479 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:25.627375 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e25ab82f-72fb-4cde-89c4-0a0dd80f5feb-serving-cert\") pod \"console-operator-68b757fb67-frt7c\" (UID: \"e25ab82f-72fb-4cde-89c4-0a0dd80f5feb\") " pod="openshift-console-operator/console-operator-68b757fb67-frt7c" Jul 07 19:01:25.627577 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:01:25.627512 2563 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 07 19:01:25.627577 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:25.627539 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbmwd\" (UniqueName: \"kubernetes.io/projected/e25ab82f-72fb-4cde-89c4-0a0dd80f5feb-kube-api-access-nbmwd\") pod \"console-operator-68b757fb67-frt7c\" (UID: \"e25ab82f-72fb-4cde-89c4-0a0dd80f5feb\") " pod="openshift-console-operator/console-operator-68b757fb67-frt7c" Jul 07 19:01:25.627683 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:25.627581 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sn2l4\" (UniqueName: \"kubernetes.io/projected/b645a2b4-4d47-4cf9-80e9-be54aa8b722f-kube-api-access-sn2l4\") pod \"cluster-samples-operator-56fcf86bf-g97sh\" (UID: \"b645a2b4-4d47-4cf9-80e9-be54aa8b722f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-g97sh" Jul 07 19:01:25.627683 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:01:25.627602 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ecead548-45c5-4195-95e8-a5c058d5958b-cluster-monitoring-operator-tls podName:ecead548-45c5-4195-95e8-a5c058d5958b nodeName:}" failed. No retries permitted until 2026-07-07 19:01:26.127582265 +0000 UTC m=+69.143586424 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/ecead548-45c5-4195-95e8-a5c058d5958b-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-pnvkh" (UID: "ecead548-45c5-4195-95e8-a5c058d5958b") : secret "cluster-monitoring-operator-tls" not found Jul 07 19:01:25.627790 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:25.627735 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmthl\" (UniqueName: \"kubernetes.io/projected/ecead548-45c5-4195-95e8-a5c058d5958b-kube-api-access-wmthl\") pod \"cluster-monitoring-operator-64786d9cbb-pnvkh\" (UID: \"ecead548-45c5-4195-95e8-a5c058d5958b\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-pnvkh" Jul 07 19:01:25.627790 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:25.627774 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b645a2b4-4d47-4cf9-80e9-be54aa8b722f-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-g97sh\" (UID: \"b645a2b4-4d47-4cf9-80e9-be54aa8b722f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-g97sh" Jul 07 19:01:25.627946 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:25.627924 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e25ab82f-72fb-4cde-89c4-0a0dd80f5feb-config\") pod \"console-operator-68b757fb67-frt7c\" (UID: \"e25ab82f-72fb-4cde-89c4-0a0dd80f5feb\") " pod="openshift-console-operator/console-operator-68b757fb67-frt7c" Jul 07 19:01:25.628022 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:25.627979 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e25ab82f-72fb-4cde-89c4-0a0dd80f5feb-trusted-ca\") pod \"console-operator-68b757fb67-frt7c\" (UID: \"e25ab82f-72fb-4cde-89c4-0a0dd80f5feb\") " pod="openshift-console-operator/console-operator-68b757fb67-frt7c" Jul 07 19:01:25.628198 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:25.628177 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/ecead548-45c5-4195-95e8-a5c058d5958b-telemetry-config\") pod \"cluster-monitoring-operator-64786d9cbb-pnvkh\" (UID: \"ecead548-45c5-4195-95e8-a5c058d5958b\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-pnvkh" Jul 07 19:01:25.628738 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:25.628714 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e25ab82f-72fb-4cde-89c4-0a0dd80f5feb-config\") pod \"console-operator-68b757fb67-frt7c\" (UID: \"e25ab82f-72fb-4cde-89c4-0a0dd80f5feb\") " pod="openshift-console-operator/console-operator-68b757fb67-frt7c" Jul 07 19:01:25.628832 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:25.628779 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e25ab82f-72fb-4cde-89c4-0a0dd80f5feb-trusted-ca\") pod \"console-operator-68b757fb67-frt7c\" (UID: \"e25ab82f-72fb-4cde-89c4-0a0dd80f5feb\") " pod="openshift-console-operator/console-operator-68b757fb67-frt7c" Jul 07 19:01:25.630140 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:25.630118 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e25ab82f-72fb-4cde-89c4-0a0dd80f5feb-serving-cert\") pod \"console-operator-68b757fb67-frt7c\" (UID: \"e25ab82f-72fb-4cde-89c4-0a0dd80f5feb\") " pod="openshift-console-operator/console-operator-68b757fb67-frt7c" Jul 07 19:01:25.643295 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:25.643263 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmthl\" (UniqueName: \"kubernetes.io/projected/ecead548-45c5-4195-95e8-a5c058d5958b-kube-api-access-wmthl\") pod \"cluster-monitoring-operator-64786d9cbb-pnvkh\" (UID: \"ecead548-45c5-4195-95e8-a5c058d5958b\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-pnvkh" Jul 07 19:01:25.649143 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:25.649103 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nbmwd\" (UniqueName: \"kubernetes.io/projected/e25ab82f-72fb-4cde-89c4-0a0dd80f5feb-kube-api-access-nbmwd\") pod \"console-operator-68b757fb67-frt7c\" (UID: \"e25ab82f-72fb-4cde-89c4-0a0dd80f5feb\") " pod="openshift-console-operator/console-operator-68b757fb67-frt7c" Jul 07 19:01:25.728463 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:25.728427 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sn2l4\" (UniqueName: \"kubernetes.io/projected/b645a2b4-4d47-4cf9-80e9-be54aa8b722f-kube-api-access-sn2l4\") pod \"cluster-samples-operator-56fcf86bf-g97sh\" (UID: \"b645a2b4-4d47-4cf9-80e9-be54aa8b722f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-g97sh" Jul 07 19:01:25.728966 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:25.728480 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b645a2b4-4d47-4cf9-80e9-be54aa8b722f-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-g97sh\" (UID: \"b645a2b4-4d47-4cf9-80e9-be54aa8b722f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-g97sh" Jul 07 19:01:25.728966 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:01:25.728619 2563 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 07 19:01:25.728966 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:01:25.728714 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b645a2b4-4d47-4cf9-80e9-be54aa8b722f-samples-operator-tls podName:b645a2b4-4d47-4cf9-80e9-be54aa8b722f nodeName:}" failed. No retries permitted until 2026-07-07 19:01:26.228694536 +0000 UTC m=+69.244698695 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/b645a2b4-4d47-4cf9-80e9-be54aa8b722f-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-g97sh" (UID: "b645a2b4-4d47-4cf9-80e9-be54aa8b722f") : secret "samples-operator-tls" not found Jul 07 19:01:25.740799 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:25.740767 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sn2l4\" (UniqueName: \"kubernetes.io/projected/b645a2b4-4d47-4cf9-80e9-be54aa8b722f-kube-api-access-sn2l4\") pod \"cluster-samples-operator-56fcf86bf-g97sh\" (UID: \"b645a2b4-4d47-4cf9-80e9-be54aa8b722f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-g97sh" Jul 07 19:01:25.817596 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:25.817559 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-68b757fb67-frt7c" Jul 07 19:01:25.834974 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:25.834918 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:01:25.966989 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:25.966940 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:01:26.132220 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:26.132128 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ecead548-45c5-4195-95e8-a5c058d5958b-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-pnvkh\" (UID: \"ecead548-45c5-4195-95e8-a5c058d5958b\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-pnvkh" Jul 07 19:01:26.132389 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:01:26.132306 2563 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 07 19:01:26.132389 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:01:26.132385 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ecead548-45c5-4195-95e8-a5c058d5958b-cluster-monitoring-operator-tls podName:ecead548-45c5-4195-95e8-a5c058d5958b nodeName:}" failed. No retries permitted until 2026-07-07 19:01:27.132365637 +0000 UTC m=+70.148369795 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/ecead548-45c5-4195-95e8-a5c058d5958b-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-pnvkh" (UID: "ecead548-45c5-4195-95e8-a5c058d5958b") : secret "cluster-monitoring-operator-tls" not found Jul 07 19:01:26.233640 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:26.233598 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b645a2b4-4d47-4cf9-80e9-be54aa8b722f-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-g97sh\" (UID: \"b645a2b4-4d47-4cf9-80e9-be54aa8b722f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-g97sh" Jul 07 19:01:26.233861 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:01:26.233768 2563 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 07 19:01:26.233861 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:01:26.233849 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b645a2b4-4d47-4cf9-80e9-be54aa8b722f-samples-operator-tls podName:b645a2b4-4d47-4cf9-80e9-be54aa8b722f nodeName:}" failed. No retries permitted until 2026-07-07 19:01:27.233831984 +0000 UTC m=+70.249836142 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/b645a2b4-4d47-4cf9-80e9-be54aa8b722f-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-g97sh" (UID: "b645a2b4-4d47-4cf9-80e9-be54aa8b722f") : secret "samples-operator-tls" not found Jul 07 19:01:26.777980 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:26.777924 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-68b757fb67-frt7c"] Jul 07 19:01:26.782026 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:01:26.781990 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode25ab82f_72fb_4cde_89c4_0a0dd80f5feb.slice/crio-61dbc7ff482ce494a59395b019e10b2e2c4a3bcc70ca23b5c29741bc0477a80f WatchSource:0}: Error finding container 61dbc7ff482ce494a59395b019e10b2e2c4a3bcc70ca23b5c29741bc0477a80f: Status 404 returned error can't find the container with id 61dbc7ff482ce494a59395b019e10b2e2c4a3bcc70ca23b5c29741bc0477a80f Jul 07 19:01:27.140635 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:27.140598 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ecead548-45c5-4195-95e8-a5c058d5958b-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-pnvkh\" (UID: \"ecead548-45c5-4195-95e8-a5c058d5958b\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-pnvkh" Jul 07 19:01:27.140820 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:01:27.140713 2563 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 07 19:01:27.140820 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:01:27.140791 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ecead548-45c5-4195-95e8-a5c058d5958b-cluster-monitoring-operator-tls podName:ecead548-45c5-4195-95e8-a5c058d5958b nodeName:}" failed. No retries permitted until 2026-07-07 19:01:29.14077287 +0000 UTC m=+72.156777031 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/ecead548-45c5-4195-95e8-a5c058d5958b-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-pnvkh" (UID: "ecead548-45c5-4195-95e8-a5c058d5958b") : secret "cluster-monitoring-operator-tls" not found Jul 07 19:01:27.241326 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:27.241291 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b645a2b4-4d47-4cf9-80e9-be54aa8b722f-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-g97sh\" (UID: \"b645a2b4-4d47-4cf9-80e9-be54aa8b722f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-g97sh" Jul 07 19:01:27.241487 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:01:27.241461 2563 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 07 19:01:27.241551 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:01:27.241541 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b645a2b4-4d47-4cf9-80e9-be54aa8b722f-samples-operator-tls podName:b645a2b4-4d47-4cf9-80e9-be54aa8b722f nodeName:}" failed. No retries permitted until 2026-07-07 19:01:29.241525885 +0000 UTC m=+72.257530043 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/b645a2b4-4d47-4cf9-80e9-be54aa8b722f-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-g97sh" (UID: "b645a2b4-4d47-4cf9-80e9-be54aa8b722f") : secret "samples-operator-tls" not found Jul 07 19:01:27.760580 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:27.760534 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-d8t2d" event={"ID":"1d84b355-0ac7-4d14-83b6-7c18077e9319","Type":"ContainerStarted","Data":"a26b5ae4432a1605a228388a6a8c122f9fb9cdd8f529344a0357979671690f77"} Jul 07 19:01:27.760775 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:27.760622 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-d8t2d" Jul 07 19:01:27.761611 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:27.761590 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-68b757fb67-frt7c" event={"ID":"e25ab82f-72fb-4cde-89c4-0a0dd80f5feb","Type":"ContainerStarted","Data":"61dbc7ff482ce494a59395b019e10b2e2c4a3bcc70ca23b5c29741bc0477a80f"} Jul 07 19:01:27.785881 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:27.785844 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:01:27.883835 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:27.883791 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:01:29.159610 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:29.159566 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ecead548-45c5-4195-95e8-a5c058d5958b-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-pnvkh\" (UID: \"ecead548-45c5-4195-95e8-a5c058d5958b\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-pnvkh" Jul 07 19:01:29.160081 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:01:29.159695 2563 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 07 19:01:29.160081 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:01:29.159783 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ecead548-45c5-4195-95e8-a5c058d5958b-cluster-monitoring-operator-tls podName:ecead548-45c5-4195-95e8-a5c058d5958b nodeName:}" failed. No retries permitted until 2026-07-07 19:01:33.15976002 +0000 UTC m=+76.175764198 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/ecead548-45c5-4195-95e8-a5c058d5958b-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-pnvkh" (UID: "ecead548-45c5-4195-95e8-a5c058d5958b") : secret "cluster-monitoring-operator-tls" not found Jul 07 19:01:29.260852 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:29.260814 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b645a2b4-4d47-4cf9-80e9-be54aa8b722f-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-g97sh\" (UID: \"b645a2b4-4d47-4cf9-80e9-be54aa8b722f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-g97sh" Jul 07 19:01:29.261053 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:01:29.261029 2563 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 07 19:01:29.261119 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:01:29.261113 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b645a2b4-4d47-4cf9-80e9-be54aa8b722f-samples-operator-tls podName:b645a2b4-4d47-4cf9-80e9-be54aa8b722f nodeName:}" failed. No retries permitted until 2026-07-07 19:01:33.261092404 +0000 UTC m=+76.277096569 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/b645a2b4-4d47-4cf9-80e9-be54aa8b722f-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-g97sh" (UID: "b645a2b4-4d47-4cf9-80e9-be54aa8b722f") : secret "samples-operator-tls" not found Jul 07 19:01:29.653151 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:29.653102 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-d8t2d" podStartSLOduration=68.509808597 podStartE2EDuration="1m12.653081811s" totalImagesPullingTime="4.143273214s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:00:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:01:27.780694748 +0000 UTC m=+70.796698929" watchObservedRunningTime="2026-07-07 19:01:29.653081811 +0000 UTC m=+72.669085993" Jul 07 19:01:29.653662 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:29.653641 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-mb8xx"] Jul 07 19:01:29.655764 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:29.655748 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-mb8xx" Jul 07 19:01:29.664940 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:29.664916 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 07 19:01:29.672614 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:29.672590 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-bj6vq\"" Jul 07 19:01:29.680475 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:29.680454 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 07 19:01:29.693333 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:29.693307 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 07 19:01:29.702920 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:29.702896 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 07 19:01:29.737435 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:29.737404 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-mb8xx"] Jul 07 19:01:29.761197 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:29.761134 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:01:29.764272 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:29.764245 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8db445ca-fea3-4e5e-b946-b4509c6427c8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mb8xx\" (UID: \"8db445ca-fea3-4e5e-b946-b4509c6427c8\") " pod="openshift-insights/insights-runtime-extractor-mb8xx" Jul 07 19:01:29.764272 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:29.764280 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/8db445ca-fea3-4e5e-b946-b4509c6427c8-crio-socket\") pod \"insights-runtime-extractor-mb8xx\" (UID: \"8db445ca-fea3-4e5e-b946-b4509c6427c8\") " pod="openshift-insights/insights-runtime-extractor-mb8xx" Jul 07 19:01:29.764454 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:29.764304 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/8db445ca-fea3-4e5e-b946-b4509c6427c8-data-volume\") pod \"insights-runtime-extractor-mb8xx\" (UID: \"8db445ca-fea3-4e5e-b946-b4509c6427c8\") " pod="openshift-insights/insights-runtime-extractor-mb8xx" Jul 07 19:01:29.764454 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:29.764364 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87lll\" (UniqueName: \"kubernetes.io/projected/8db445ca-fea3-4e5e-b946-b4509c6427c8-kube-api-access-87lll\") pod \"insights-runtime-extractor-mb8xx\" (UID: \"8db445ca-fea3-4e5e-b946-b4509c6427c8\") " pod="openshift-insights/insights-runtime-extractor-mb8xx" Jul 07 19:01:29.764454 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:29.764383 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8db445ca-fea3-4e5e-b946-b4509c6427c8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mb8xx\" (UID: \"8db445ca-fea3-4e5e-b946-b4509c6427c8\") " pod="openshift-insights/insights-runtime-extractor-mb8xx" Jul 07 19:01:29.767497 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:29.767479 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-frt7c_e25ab82f-72fb-4cde-89c4-0a0dd80f5feb/console-operator/0.log" Jul 07 19:01:29.767596 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:29.767513 2563 generic.go:356] "Generic (PLEG): container finished" podID="e25ab82f-72fb-4cde-89c4-0a0dd80f5feb" containerID="06b2e1c3f17f47989e2ad9da475ec194a299dda004f3f86b91b20fa53a56fdcc" exitCode=255 Jul 07 19:01:29.767596 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:29.767576 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-68b757fb67-frt7c" event={"ID":"e25ab82f-72fb-4cde-89c4-0a0dd80f5feb","Type":"ContainerDied","Data":"06b2e1c3f17f47989e2ad9da475ec194a299dda004f3f86b91b20fa53a56fdcc"} Jul 07 19:01:29.767783 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:29.767772 2563 scope.go:117] "RemoveContainer" containerID="06b2e1c3f17f47989e2ad9da475ec194a299dda004f3f86b91b20fa53a56fdcc" Jul 07 19:01:29.844707 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:29.844668 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:01:29.865557 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:29.865520 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8db445ca-fea3-4e5e-b946-b4509c6427c8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mb8xx\" (UID: \"8db445ca-fea3-4e5e-b946-b4509c6427c8\") " pod="openshift-insights/insights-runtime-extractor-mb8xx" Jul 07 19:01:29.865753 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:01:29.865675 2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 07 19:01:29.865753 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:29.865693 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8db445ca-fea3-4e5e-b946-b4509c6427c8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mb8xx\" (UID: \"8db445ca-fea3-4e5e-b946-b4509c6427c8\") " pod="openshift-insights/insights-runtime-extractor-mb8xx" Jul 07 19:01:29.865908 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:01:29.865764 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8db445ca-fea3-4e5e-b946-b4509c6427c8-insights-runtime-extractor-tls podName:8db445ca-fea3-4e5e-b946-b4509c6427c8 nodeName:}" failed. No retries permitted until 2026-07-07 19:01:30.365741128 +0000 UTC m=+73.381745300 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/8db445ca-fea3-4e5e-b946-b4509c6427c8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-mb8xx" (UID: "8db445ca-fea3-4e5e-b946-b4509c6427c8") : secret "insights-runtime-extractor-tls" not found Jul 07 19:01:29.865908 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:01:29.865763 2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 07 19:01:29.865908 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:29.865781 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/8db445ca-fea3-4e5e-b946-b4509c6427c8-crio-socket\") pod \"insights-runtime-extractor-mb8xx\" (UID: \"8db445ca-fea3-4e5e-b946-b4509c6427c8\") " pod="openshift-insights/insights-runtime-extractor-mb8xx" Jul 07 19:01:29.865908 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:29.865819 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/8db445ca-fea3-4e5e-b946-b4509c6427c8-data-volume\") pod \"insights-runtime-extractor-mb8xx\" (UID: \"8db445ca-fea3-4e5e-b946-b4509c6427c8\") " pod="openshift-insights/insights-runtime-extractor-mb8xx" Jul 07 19:01:29.865908 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:29.865852 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87lll\" (UniqueName: \"kubernetes.io/projected/8db445ca-fea3-4e5e-b946-b4509c6427c8-kube-api-access-87lll\") pod \"insights-runtime-extractor-mb8xx\" (UID: \"8db445ca-fea3-4e5e-b946-b4509c6427c8\") " pod="openshift-insights/insights-runtime-extractor-mb8xx" Jul 07 19:01:29.866229 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:01:29.866200 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8db445ca-fea3-4e5e-b946-b4509c6427c8-kube-rbac-proxy-cm podName:8db445ca-fea3-4e5e-b946-b4509c6427c8 nodeName:}" failed. No retries permitted until 2026-07-07 19:01:30.366179182 +0000 UTC m=+73.382183342 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/8db445ca-fea3-4e5e-b946-b4509c6427c8-kube-rbac-proxy-cm") pod "insights-runtime-extractor-mb8xx" (UID: "8db445ca-fea3-4e5e-b946-b4509c6427c8") : configmap "kube-rbac-proxy" not found Jul 07 19:01:29.866229 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:29.866215 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/8db445ca-fea3-4e5e-b946-b4509c6427c8-crio-socket\") pod \"insights-runtime-extractor-mb8xx\" (UID: \"8db445ca-fea3-4e5e-b946-b4509c6427c8\") " pod="openshift-insights/insights-runtime-extractor-mb8xx" Jul 07 19:01:29.866494 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:29.866476 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/8db445ca-fea3-4e5e-b946-b4509c6427c8-data-volume\") pod \"insights-runtime-extractor-mb8xx\" (UID: \"8db445ca-fea3-4e5e-b946-b4509c6427c8\") " pod="openshift-insights/insights-runtime-extractor-mb8xx" Jul 07 19:01:29.876464 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:29.876433 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-87lll\" (UniqueName: \"kubernetes.io/projected/8db445ca-fea3-4e5e-b946-b4509c6427c8-kube-api-access-87lll\") pod \"insights-runtime-extractor-mb8xx\" (UID: \"8db445ca-fea3-4e5e-b946-b4509c6427c8\") " pod="openshift-insights/insights-runtime-extractor-mb8xx" Jul 07 19:01:30.369688 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:30.369653 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8db445ca-fea3-4e5e-b946-b4509c6427c8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mb8xx\" (UID: \"8db445ca-fea3-4e5e-b946-b4509c6427c8\") " pod="openshift-insights/insights-runtime-extractor-mb8xx" Jul 07 19:01:30.370144 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:30.369703 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8db445ca-fea3-4e5e-b946-b4509c6427c8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mb8xx\" (UID: \"8db445ca-fea3-4e5e-b946-b4509c6427c8\") " pod="openshift-insights/insights-runtime-extractor-mb8xx" Jul 07 19:01:30.370144 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:01:30.369773 2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 07 19:01:30.370144 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:01:30.369796 2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 07 19:01:30.370144 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:01:30.369847 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8db445ca-fea3-4e5e-b946-b4509c6427c8-kube-rbac-proxy-cm podName:8db445ca-fea3-4e5e-b946-b4509c6427c8 nodeName:}" failed. No retries permitted until 2026-07-07 19:01:31.369823176 +0000 UTC m=+74.385827350 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/8db445ca-fea3-4e5e-b946-b4509c6427c8-kube-rbac-proxy-cm") pod "insights-runtime-extractor-mb8xx" (UID: "8db445ca-fea3-4e5e-b946-b4509c6427c8") : configmap "kube-rbac-proxy" not found Jul 07 19:01:30.370144 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:01:30.369878 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8db445ca-fea3-4e5e-b946-b4509c6427c8-insights-runtime-extractor-tls podName:8db445ca-fea3-4e5e-b946-b4509c6427c8 nodeName:}" failed. No retries permitted until 2026-07-07 19:01:31.369866705 +0000 UTC m=+74.385870869 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/8db445ca-fea3-4e5e-b946-b4509c6427c8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-mb8xx" (UID: "8db445ca-fea3-4e5e-b946-b4509c6427c8") : secret "insights-runtime-extractor-tls" not found Jul 07 19:01:30.772192 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:30.772110 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-frt7c_e25ab82f-72fb-4cde-89c4-0a0dd80f5feb/console-operator/1.log" Jul 07 19:01:30.772629 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:30.772605 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-frt7c_e25ab82f-72fb-4cde-89c4-0a0dd80f5feb/console-operator/0.log" Jul 07 19:01:30.772809 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:30.772646 2563 generic.go:356] "Generic (PLEG): container finished" podID="e25ab82f-72fb-4cde-89c4-0a0dd80f5feb" containerID="0b7622e70aabd855861e65bc01ce89d1d937ab28c0e7067cdf210f51b2b6be56" exitCode=255 Jul 07 19:01:30.772809 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:30.772691 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-68b757fb67-frt7c" event={"ID":"e25ab82f-72fb-4cde-89c4-0a0dd80f5feb","Type":"ContainerDied","Data":"0b7622e70aabd855861e65bc01ce89d1d937ab28c0e7067cdf210f51b2b6be56"} Jul 07 19:01:30.772809 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:30.772743 2563 scope.go:117] "RemoveContainer" containerID="06b2e1c3f17f47989e2ad9da475ec194a299dda004f3f86b91b20fa53a56fdcc" Jul 07 19:01:30.773026 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:30.772933 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:01:30.773026 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:30.773024 2563 scope.go:117] "RemoveContainer" containerID="0b7622e70aabd855861e65bc01ce89d1d937ab28c0e7067cdf210f51b2b6be56" Jul 07 19:01:30.773307 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:01:30.773230 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-68b757fb67-frt7c_openshift-console-operator(e25ab82f-72fb-4cde-89c4-0a0dd80f5feb)\"" pod="openshift-console-operator/console-operator-68b757fb67-frt7c" podUID="e25ab82f-72fb-4cde-89c4-0a0dd80f5feb" Jul 07 19:01:30.874474 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:30.874439 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:01:30.977693 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:30.977659 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:01:31.079492 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:31.079459 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:01:31.201628 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:31.201592 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:01:31.279402 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:31.279366 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:01:31.379501 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:31.379409 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8db445ca-fea3-4e5e-b946-b4509c6427c8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mb8xx\" (UID: \"8db445ca-fea3-4e5e-b946-b4509c6427c8\") " pod="openshift-insights/insights-runtime-extractor-mb8xx" Jul 07 19:01:31.379987 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:31.379533 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8db445ca-fea3-4e5e-b946-b4509c6427c8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mb8xx\" (UID: \"8db445ca-fea3-4e5e-b946-b4509c6427c8\") " pod="openshift-insights/insights-runtime-extractor-mb8xx" Jul 07 19:01:31.379987 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:01:31.379548 2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 07 19:01:31.379987 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:01:31.379625 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8db445ca-fea3-4e5e-b946-b4509c6427c8-kube-rbac-proxy-cm podName:8db445ca-fea3-4e5e-b946-b4509c6427c8 nodeName:}" failed. No retries permitted until 2026-07-07 19:01:33.379599097 +0000 UTC m=+76.395603269 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/8db445ca-fea3-4e5e-b946-b4509c6427c8-kube-rbac-proxy-cm") pod "insights-runtime-extractor-mb8xx" (UID: "8db445ca-fea3-4e5e-b946-b4509c6427c8") : configmap "kube-rbac-proxy" not found Jul 07 19:01:31.379987 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:01:31.379650 2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 07 19:01:31.379987 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:01:31.379705 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8db445ca-fea3-4e5e-b946-b4509c6427c8-insights-runtime-extractor-tls podName:8db445ca-fea3-4e5e-b946-b4509c6427c8 nodeName:}" failed. No retries permitted until 2026-07-07 19:01:33.379689031 +0000 UTC m=+76.395693190 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/8db445ca-fea3-4e5e-b946-b4509c6427c8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-mb8xx" (UID: "8db445ca-fea3-4e5e-b946-b4509c6427c8") : secret "insights-runtime-extractor-tls" not found Jul 07 19:01:31.468234 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:31.468191 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:01:31.572416 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:31.572376 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:01:31.777212 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:31.777146 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-frt7c_e25ab82f-72fb-4cde-89c4-0a0dd80f5feb/console-operator/1.log" Jul 07 19:01:31.777479 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:31.777462 2563 scope.go:117] "RemoveContainer" containerID="0b7622e70aabd855861e65bc01ce89d1d937ab28c0e7067cdf210f51b2b6be56" Jul 07 19:01:31.777628 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:01:31.777611 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-68b757fb67-frt7c_openshift-console-operator(e25ab82f-72fb-4cde-89c4-0a0dd80f5feb)\"" pod="openshift-console-operator/console-operator-68b757fb67-frt7c" podUID="e25ab82f-72fb-4cde-89c4-0a0dd80f5feb" Jul 07 19:01:31.791088 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:31.791055 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:01:31.847641 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:31.847615 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-whw2v_3953dadf-98ed-4933-a330-1191906b6cff/dns-node-resolver/0.log" Jul 07 19:01:31.891748 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:31.891719 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:01:31.992575 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:31.992535 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:01:32.652342 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:32.652318 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-s272w_a6abe188-2b1b-427a-ba0c-e93930187c7e/node-ca/0.log" Jul 07 19:01:33.196354 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:33.196318 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ecead548-45c5-4195-95e8-a5c058d5958b-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-pnvkh\" (UID: \"ecead548-45c5-4195-95e8-a5c058d5958b\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-pnvkh" Jul 07 19:01:33.196539 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:01:33.196492 2563 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 07 19:01:33.196619 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:01:33.196572 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ecead548-45c5-4195-95e8-a5c058d5958b-cluster-monitoring-operator-tls podName:ecead548-45c5-4195-95e8-a5c058d5958b nodeName:}" failed. No retries permitted until 2026-07-07 19:01:41.196542755 +0000 UTC m=+84.212546914 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/ecead548-45c5-4195-95e8-a5c058d5958b-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-pnvkh" (UID: "ecead548-45c5-4195-95e8-a5c058d5958b") : secret "cluster-monitoring-operator-tls" not found Jul 07 19:01:33.246538 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:33.246512 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-28nms_a6f2e7e7-5e73-40a1-a05a-d5a338128dd7/serve-healthcheck-canary/0.log" Jul 07 19:01:33.296887 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:33.296851 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b645a2b4-4d47-4cf9-80e9-be54aa8b722f-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-g97sh\" (UID: \"b645a2b4-4d47-4cf9-80e9-be54aa8b722f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-g97sh" Jul 07 19:01:33.297056 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:01:33.297015 2563 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 07 19:01:33.297098 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:01:33.297075 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b645a2b4-4d47-4cf9-80e9-be54aa8b722f-samples-operator-tls podName:b645a2b4-4d47-4cf9-80e9-be54aa8b722f nodeName:}" failed. No retries permitted until 2026-07-07 19:01:41.29705871 +0000 UTC m=+84.313062869 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/b645a2b4-4d47-4cf9-80e9-be54aa8b722f-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-g97sh" (UID: "b645a2b4-4d47-4cf9-80e9-be54aa8b722f") : secret "samples-operator-tls" not found Jul 07 19:01:33.398050 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:33.398009 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8db445ca-fea3-4e5e-b946-b4509c6427c8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mb8xx\" (UID: \"8db445ca-fea3-4e5e-b946-b4509c6427c8\") " pod="openshift-insights/insights-runtime-extractor-mb8xx" Jul 07 19:01:33.398227 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:33.398066 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8db445ca-fea3-4e5e-b946-b4509c6427c8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mb8xx\" (UID: \"8db445ca-fea3-4e5e-b946-b4509c6427c8\") " pod="openshift-insights/insights-runtime-extractor-mb8xx" Jul 07 19:01:33.398227 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:01:33.398136 2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 07 19:01:33.398227 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:01:33.398180 2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 07 19:01:33.398227 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:01:33.398202 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8db445ca-fea3-4e5e-b946-b4509c6427c8-kube-rbac-proxy-cm podName:8db445ca-fea3-4e5e-b946-b4509c6427c8 nodeName:}" failed. No retries permitted until 2026-07-07 19:01:37.398186424 +0000 UTC m=+80.414190586 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/8db445ca-fea3-4e5e-b946-b4509c6427c8-kube-rbac-proxy-cm") pod "insights-runtime-extractor-mb8xx" (UID: "8db445ca-fea3-4e5e-b946-b4509c6427c8") : configmap "kube-rbac-proxy" not found Jul 07 19:01:33.398227 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:01:33.398219 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8db445ca-fea3-4e5e-b946-b4509c6427c8-insights-runtime-extractor-tls podName:8db445ca-fea3-4e5e-b946-b4509c6427c8 nodeName:}" failed. No retries permitted until 2026-07-07 19:01:37.398208276 +0000 UTC m=+80.414212435 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/8db445ca-fea3-4e5e-b946-b4509c6427c8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-mb8xx" (UID: "8db445ca-fea3-4e5e-b946-b4509c6427c8") : secret "insights-runtime-extractor-tls" not found Jul 07 19:01:33.462453 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:33.462363 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:01:33.870819 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:33.870778 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:01:33.963970 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:33.963912 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:01:34.270099 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:34.270011 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:01:35.790294 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:35.790258 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-lk7md" event={"ID":"59f0c8fc-bc9a-4273-aac2-a8ee23bf6b63","Type":"ContainerStarted","Data":"dde3c33c54f6133656582152030af43bc2b653745013bd745b5cc58d734f6af3"} Jul 07 19:01:35.812066 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:35.812001 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-lk7md" podStartSLOduration=66.676194343 podStartE2EDuration="1m18.811978289s" totalImagesPullingTime="12.135783946s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 19:00:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:01:35.810682164 +0000 UTC m=+78.826686348" watchObservedRunningTime="2026-07-07 19:01:35.811978289 +0000 UTC m=+78.827982468" Jul 07 19:01:35.818402 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:35.818365 2563 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-68b757fb67-frt7c" Jul 07 19:01:35.818572 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:35.818430 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-68b757fb67-frt7c" Jul 07 19:01:35.818811 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:35.818795 2563 scope.go:117] "RemoveContainer" containerID="0b7622e70aabd855861e65bc01ce89d1d937ab28c0e7067cdf210f51b2b6be56" Jul 07 19:01:35.819009 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:01:35.818985 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-68b757fb67-frt7c_openshift-console-operator(e25ab82f-72fb-4cde-89c4-0a0dd80f5feb)\"" pod="openshift-console-operator/console-operator-68b757fb67-frt7c" podUID="e25ab82f-72fb-4cde-89c4-0a0dd80f5feb" Jul 07 19:01:37.433167 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:37.433124 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8db445ca-fea3-4e5e-b946-b4509c6427c8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mb8xx\" (UID: \"8db445ca-fea3-4e5e-b946-b4509c6427c8\") " pod="openshift-insights/insights-runtime-extractor-mb8xx" Jul 07 19:01:37.433611 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:37.433238 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8db445ca-fea3-4e5e-b946-b4509c6427c8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mb8xx\" (UID: \"8db445ca-fea3-4e5e-b946-b4509c6427c8\") " pod="openshift-insights/insights-runtime-extractor-mb8xx" Jul 07 19:01:37.433611 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:01:37.433315 2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 07 19:01:37.433611 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:01:37.433329 2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 07 19:01:37.433611 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:01:37.433365 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8db445ca-fea3-4e5e-b946-b4509c6427c8-kube-rbac-proxy-cm podName:8db445ca-fea3-4e5e-b946-b4509c6427c8 nodeName:}" failed. No retries permitted until 2026-07-07 19:01:45.433350864 +0000 UTC m=+88.449355023 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/8db445ca-fea3-4e5e-b946-b4509c6427c8-kube-rbac-proxy-cm") pod "insights-runtime-extractor-mb8xx" (UID: "8db445ca-fea3-4e5e-b946-b4509c6427c8") : configmap "kube-rbac-proxy" not found Jul 07 19:01:37.433611 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:01:37.433399 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8db445ca-fea3-4e5e-b946-b4509c6427c8-insights-runtime-extractor-tls podName:8db445ca-fea3-4e5e-b946-b4509c6427c8 nodeName:}" failed. No retries permitted until 2026-07-07 19:01:45.433379874 +0000 UTC m=+88.449384035 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/8db445ca-fea3-4e5e-b946-b4509c6427c8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-mb8xx" (UID: "8db445ca-fea3-4e5e-b946-b4509c6427c8") : secret "insights-runtime-extractor-tls" not found Jul 07 19:01:37.996122 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:37.996081 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:01:40.401997 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:40.401870 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:01:40.652788 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:40.652700 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:01:40.803080 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:40.803048 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:01:41.262624 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:41.262576 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ecead548-45c5-4195-95e8-a5c058d5958b-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-pnvkh\" (UID: \"ecead548-45c5-4195-95e8-a5c058d5958b\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-pnvkh" Jul 07 19:01:41.262815 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:01:41.262762 2563 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 07 19:01:41.262878 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:01:41.262844 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ecead548-45c5-4195-95e8-a5c058d5958b-cluster-monitoring-operator-tls podName:ecead548-45c5-4195-95e8-a5c058d5958b nodeName:}" failed. No retries permitted until 2026-07-07 19:01:57.262823078 +0000 UTC m=+100.278827253 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/ecead548-45c5-4195-95e8-a5c058d5958b-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-pnvkh" (UID: "ecead548-45c5-4195-95e8-a5c058d5958b") : secret "cluster-monitoring-operator-tls" not found Jul 07 19:01:41.363205 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:41.363165 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b645a2b4-4d47-4cf9-80e9-be54aa8b722f-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-g97sh\" (UID: \"b645a2b4-4d47-4cf9-80e9-be54aa8b722f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-g97sh" Jul 07 19:01:41.365568 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:41.365543 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b645a2b4-4d47-4cf9-80e9-be54aa8b722f-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-g97sh\" (UID: \"b645a2b4-4d47-4cf9-80e9-be54aa8b722f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-g97sh" Jul 07 19:01:41.514656 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:41.514564 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-g97sh" Jul 07 19:01:41.518270 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:41.518237 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:01:41.634710 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:41.634678 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:01:41.648369 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:41.645752 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-g97sh"] Jul 07 19:01:41.806400 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:41.806367 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-g97sh" event={"ID":"b645a2b4-4d47-4cf9-80e9-be54aa8b722f","Type":"ContainerStarted","Data":"573f4bead506b0d62360c66196639a6b9970b44e3e9d7a39b32b670adcb56cf6"} Jul 07 19:01:42.027198 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:42.027160 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:01:42.626789 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:42.626748 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:01:42.875226 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:42.875172 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:01:43.044613 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:43.044580 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:01:43.419124 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:43.419042 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:01:43.842466 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:43.842424 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:01:44.023405 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:44.023366 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:01:44.125854 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:44.125782 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:01:44.368206 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:44.368176 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:01:44.467638 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:44.467551 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:01:44.817508 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:44.817473 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-g97sh" event={"ID":"b645a2b4-4d47-4cf9-80e9-be54aa8b722f","Type":"ContainerStarted","Data":"401e75434805c04303f4957aaf504bc568d941a14d490c9bbb60375baebef568"} Jul 07 19:01:44.817673 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:44.817533 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-g97sh" event={"ID":"b645a2b4-4d47-4cf9-80e9-be54aa8b722f","Type":"ContainerStarted","Data":"289e2fd4e77836cefef3c81274e1450ec18d6e9d30cc09a89883203a2a005048"} Jul 07 19:01:44.840178 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:44.840130 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-g97sh" podStartSLOduration=17.591925976 podStartE2EDuration="19.840111739s" totalImagesPullingTime="2.248185763s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:01:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:01:44.83862364 +0000 UTC m=+87.854627820" watchObservedRunningTime="2026-07-07 19:01:44.840111739 +0000 UTC m=+87.856115919" Jul 07 19:01:45.021276 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:45.021232 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:01:45.120658 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:45.120566 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:01:45.498607 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:45.498512 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8db445ca-fea3-4e5e-b946-b4509c6427c8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mb8xx\" (UID: \"8db445ca-fea3-4e5e-b946-b4509c6427c8\") " pod="openshift-insights/insights-runtime-extractor-mb8xx" Jul 07 19:01:45.498607 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:45.498574 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8db445ca-fea3-4e5e-b946-b4509c6427c8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mb8xx\" (UID: \"8db445ca-fea3-4e5e-b946-b4509c6427c8\") " pod="openshift-insights/insights-runtime-extractor-mb8xx" Jul 07 19:01:45.498806 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:01:45.498670 2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 07 19:01:45.498806 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:01:45.498727 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8db445ca-fea3-4e5e-b946-b4509c6427c8-insights-runtime-extractor-tls podName:8db445ca-fea3-4e5e-b946-b4509c6427c8 nodeName:}" failed. No retries permitted until 2026-07-07 19:02:01.498713308 +0000 UTC m=+104.514717470 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/8db445ca-fea3-4e5e-b946-b4509c6427c8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-mb8xx" (UID: "8db445ca-fea3-4e5e-b946-b4509c6427c8") : secret "insights-runtime-extractor-tls" not found Jul 07 19:01:45.499049 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:45.499032 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8db445ca-fea3-4e5e-b946-b4509c6427c8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mb8xx\" (UID: \"8db445ca-fea3-4e5e-b946-b4509c6427c8\") " pod="openshift-insights/insights-runtime-extractor-mb8xx" Jul 07 19:01:45.611352 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:45.611316 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:01:45.813380 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:45.813349 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:01:45.930349 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:45.930311 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:01:46.223626 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:46.223509 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:01:46.430662 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:46.430624 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:01:46.647817 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:46.647779 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:01:46.841517 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:46.841476 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:01:46.973292 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:46.973189 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:01:47.073315 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:47.073274 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:01:47.620971 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:47.620919 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:01:47.819841 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:47.819809 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:01:48.233078 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:48.233048 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:01:48.419854 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:48.419825 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:01:48.471059 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:48.471024 2563 scope.go:117] "RemoveContainer" containerID="0b7622e70aabd855861e65bc01ce89d1d937ab28c0e7067cdf210f51b2b6be56" Jul 07 19:01:48.819946 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:48.819916 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:01:48.831906 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:48.831882 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-frt7c_e25ab82f-72fb-4cde-89c4-0a0dd80f5feb/console-operator/2.log" Jul 07 19:01:48.832335 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:48.832315 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-frt7c_e25ab82f-72fb-4cde-89c4-0a0dd80f5feb/console-operator/1.log" Jul 07 19:01:48.832412 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:48.832352 2563 generic.go:356] "Generic (PLEG): container finished" podID="e25ab82f-72fb-4cde-89c4-0a0dd80f5feb" containerID="72a783bb23231b4072a6d5003512e8cb9ec34dec207234061683515181666ce7" exitCode=255 Jul 07 19:01:48.832412 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:48.832389 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-68b757fb67-frt7c" event={"ID":"e25ab82f-72fb-4cde-89c4-0a0dd80f5feb","Type":"ContainerDied","Data":"72a783bb23231b4072a6d5003512e8cb9ec34dec207234061683515181666ce7"} Jul 07 19:01:48.832473 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:48.832433 2563 scope.go:117] "RemoveContainer" containerID="0b7622e70aabd855861e65bc01ce89d1d937ab28c0e7067cdf210f51b2b6be56" Jul 07 19:01:48.832747 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:48.832731 2563 scope.go:117] "RemoveContainer" containerID="72a783bb23231b4072a6d5003512e8cb9ec34dec207234061683515181666ce7" Jul 07 19:01:48.832918 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:01:48.832893 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-68b757fb67-frt7c_openshift-console-operator(e25ab82f-72fb-4cde-89c4-0a0dd80f5feb)\"" pod="openshift-console-operator/console-operator-68b757fb67-frt7c" podUID="e25ab82f-72fb-4cde-89c4-0a0dd80f5feb" Jul 07 19:01:49.423302 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:49.423247 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:01:49.824221 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:49.824187 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:01:49.836679 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:49.836648 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-frt7c_e25ab82f-72fb-4cde-89c4-0a0dd80f5feb/console-operator/2.log" Jul 07 19:01:50.022135 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:50.022100 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:01:50.420342 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:50.420300 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:01:50.821105 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:50.821078 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:01:51.023190 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:51.023152 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:01:54.478006 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:54.477974 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7f5f69ff4c-v54qk"] Jul 07 19:01:54.482055 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:54.482039 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7f5f69ff4c-v54qk" Jul 07 19:01:54.489379 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:54.489287 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 07 19:01:54.489475 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:54.489391 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 07 19:01:54.489715 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:54.489696 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 07 19:01:54.489786 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:54.489712 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\"" Jul 07 19:01:54.505123 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:54.505101 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7f5f69ff4c-v54qk"] Jul 07 19:01:54.575336 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:54.575294 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d2eadf10-3504-4db4-9e20-2afaeaa81ce5-tmp\") pod \"klusterlet-addon-workmgr-7f5f69ff4c-v54qk\" (UID: \"d2eadf10-3504-4db4-9e20-2afaeaa81ce5\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7f5f69ff4c-v54qk" Jul 07 19:01:54.575515 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:54.575348 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t568x\" (UniqueName: \"kubernetes.io/projected/d2eadf10-3504-4db4-9e20-2afaeaa81ce5-kube-api-access-t568x\") pod \"klusterlet-addon-workmgr-7f5f69ff4c-v54qk\" (UID: \"d2eadf10-3504-4db4-9e20-2afaeaa81ce5\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7f5f69ff4c-v54qk" Jul 07 19:01:54.575515 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:54.575380 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/d2eadf10-3504-4db4-9e20-2afaeaa81ce5-klusterlet-config\") pod \"klusterlet-addon-workmgr-7f5f69ff4c-v54qk\" (UID: \"d2eadf10-3504-4db4-9e20-2afaeaa81ce5\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7f5f69ff4c-v54qk" Jul 07 19:01:54.598550 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:54.598505 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:01:54.676010 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:54.675975 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d2eadf10-3504-4db4-9e20-2afaeaa81ce5-tmp\") pod \"klusterlet-addon-workmgr-7f5f69ff4c-v54qk\" (UID: \"d2eadf10-3504-4db4-9e20-2afaeaa81ce5\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7f5f69ff4c-v54qk" Jul 07 19:01:54.676187 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:54.676021 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t568x\" (UniqueName: \"kubernetes.io/projected/d2eadf10-3504-4db4-9e20-2afaeaa81ce5-kube-api-access-t568x\") pod \"klusterlet-addon-workmgr-7f5f69ff4c-v54qk\" (UID: \"d2eadf10-3504-4db4-9e20-2afaeaa81ce5\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7f5f69ff4c-v54qk" Jul 07 19:01:54.676187 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:54.676042 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/d2eadf10-3504-4db4-9e20-2afaeaa81ce5-klusterlet-config\") pod \"klusterlet-addon-workmgr-7f5f69ff4c-v54qk\" (UID: \"d2eadf10-3504-4db4-9e20-2afaeaa81ce5\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7f5f69ff4c-v54qk" Jul 07 19:01:54.676377 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:54.676359 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d2eadf10-3504-4db4-9e20-2afaeaa81ce5-tmp\") pod \"klusterlet-addon-workmgr-7f5f69ff4c-v54qk\" (UID: \"d2eadf10-3504-4db4-9e20-2afaeaa81ce5\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7f5f69ff4c-v54qk" Jul 07 19:01:54.678461 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:54.678445 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/d2eadf10-3504-4db4-9e20-2afaeaa81ce5-klusterlet-config\") pod \"klusterlet-addon-workmgr-7f5f69ff4c-v54qk\" (UID: \"d2eadf10-3504-4db4-9e20-2afaeaa81ce5\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7f5f69ff4c-v54qk" Jul 07 19:01:54.694535 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:54.694502 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:01:54.699260 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:54.699210 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t568x\" (UniqueName: \"kubernetes.io/projected/d2eadf10-3504-4db4-9e20-2afaeaa81ce5-kube-api-access-t568x\") pod \"klusterlet-addon-workmgr-7f5f69ff4c-v54qk\" (UID: \"d2eadf10-3504-4db4-9e20-2afaeaa81ce5\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7f5f69ff4c-v54qk" Jul 07 19:01:54.791031 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:54.791006 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7f5f69ff4c-v54qk" Jul 07 19:01:54.805664 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:54.805634 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:01:54.937217 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:54.937178 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7f5f69ff4c-v54qk"] Jul 07 19:01:54.941765 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:01:54.941736 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2eadf10_3504_4db4_9e20_2afaeaa81ce5.slice/crio-82111c7fd333111b8aca1a2549beacb02ff4c453da32e96baaea96d07dca3d35 WatchSource:0}: Error finding container 82111c7fd333111b8aca1a2549beacb02ff4c453da32e96baaea96d07dca3d35: Status 404 returned error can't find the container with id 82111c7fd333111b8aca1a2549beacb02ff4c453da32e96baaea96d07dca3d35 Jul 07 19:01:55.818687 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:55.818640 2563 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-68b757fb67-frt7c" Jul 07 19:01:55.819130 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:55.818749 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-68b757fb67-frt7c" Jul 07 19:01:55.819211 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:55.819188 2563 scope.go:117] "RemoveContainer" containerID="72a783bb23231b4072a6d5003512e8cb9ec34dec207234061683515181666ce7" Jul 07 19:01:55.819441 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:01:55.819414 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-68b757fb67-frt7c_openshift-console-operator(e25ab82f-72fb-4cde-89c4-0a0dd80f5feb)\"" pod="openshift-console-operator/console-operator-68b757fb67-frt7c" podUID="e25ab82f-72fb-4cde-89c4-0a0dd80f5feb" Jul 07 19:01:55.856785 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:55.856750 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7f5f69ff4c-v54qk" event={"ID":"d2eadf10-3504-4db4-9e20-2afaeaa81ce5","Type":"ContainerStarted","Data":"82111c7fd333111b8aca1a2549beacb02ff4c453da32e96baaea96d07dca3d35"} Jul 07 19:01:56.797834 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:56.797797 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1e7deba9-69b3-4260-b6d0-037f274fb311-registry-tls\") pod \"image-registry-5d5c5c78bb-kzvl6\" (UID: \"1e7deba9-69b3-4260-b6d0-037f274fb311\") " pod="openshift-image-registry/image-registry-5d5c5c78bb-kzvl6" Jul 07 19:01:56.798033 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:56.797908 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c338a925-fe6b-40a0-b669-ce49a20cc370-metrics-tls\") pod \"dns-default-hq4fc\" (UID: \"c338a925-fe6b-40a0-b669-ce49a20cc370\") " pod="openshift-dns/dns-default-hq4fc" Jul 07 19:01:56.800241 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:56.800213 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c338a925-fe6b-40a0-b669-ce49a20cc370-metrics-tls\") pod \"dns-default-hq4fc\" (UID: \"c338a925-fe6b-40a0-b669-ce49a20cc370\") " pod="openshift-dns/dns-default-hq4fc" Jul 07 19:01:56.800501 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:56.800471 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1e7deba9-69b3-4260-b6d0-037f274fb311-registry-tls\") pod \"image-registry-5d5c5c78bb-kzvl6\" (UID: \"1e7deba9-69b3-4260-b6d0-037f274fb311\") " pod="openshift-image-registry/image-registry-5d5c5c78bb-kzvl6" Jul 07 19:01:57.070427 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:57.070340 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-8tx7z\"" Jul 07 19:01:57.078859 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:57.078825 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5d5c5c78bb-kzvl6" Jul 07 19:01:57.089895 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:57.089868 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-d4n46\"" Jul 07 19:01:57.098844 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:57.098814 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-hq4fc" Jul 07 19:01:57.231931 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:57.231901 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5d5c5c78bb-kzvl6"] Jul 07 19:01:57.235045 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:01:57.235018 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e7deba9_69b3_4260_b6d0_037f274fb311.slice/crio-866cb34a169948c20ea9120b465a4de13a8fd780b28975b1980fffb90a235880 WatchSource:0}: Error finding container 866cb34a169948c20ea9120b465a4de13a8fd780b28975b1980fffb90a235880: Status 404 returned error can't find the container with id 866cb34a169948c20ea9120b465a4de13a8fd780b28975b1980fffb90a235880 Jul 07 19:01:57.250973 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:57.250937 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-hq4fc"] Jul 07 19:01:57.255015 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:01:57.254986 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc338a925_fe6b_40a0_b669_ce49a20cc370.slice/crio-ccc742ebbac41c7eea5f9590f0934e1e6d876e0ed7c5bca56090631a563f9842 WatchSource:0}: Error finding container ccc742ebbac41c7eea5f9590f0934e1e6d876e0ed7c5bca56090631a563f9842: Status 404 returned error can't find the container with id ccc742ebbac41c7eea5f9590f0934e1e6d876e0ed7c5bca56090631a563f9842 Jul 07 19:01:57.302192 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:57.302149 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ecead548-45c5-4195-95e8-a5c058d5958b-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-pnvkh\" (UID: \"ecead548-45c5-4195-95e8-a5c058d5958b\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-pnvkh" Jul 07 19:01:57.304665 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:57.304643 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ecead548-45c5-4195-95e8-a5c058d5958b-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-pnvkh\" (UID: \"ecead548-45c5-4195-95e8-a5c058d5958b\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-pnvkh" Jul 07 19:01:57.324007 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:57.323884 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-pnvkh" Jul 07 19:01:57.462006 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:57.461967 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-64786d9cbb-pnvkh"] Jul 07 19:01:57.465668 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:01:57.465638 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podecead548_45c5_4195_95e8_a5c058d5958b.slice/crio-1ca28eeea1879e17f62321d8817f3058fb2b0b7eea12c08c6fa17b019d57051e WatchSource:0}: Error finding container 1ca28eeea1879e17f62321d8817f3058fb2b0b7eea12c08c6fa17b019d57051e: Status 404 returned error can't find the container with id 1ca28eeea1879e17f62321d8817f3058fb2b0b7eea12c08c6fa17b019d57051e Jul 07 19:01:57.864401 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:57.864368 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5d5c5c78bb-kzvl6" event={"ID":"1e7deba9-69b3-4260-b6d0-037f274fb311","Type":"ContainerStarted","Data":"b3ac1a1f7e51d666caae9cbc9ad675059e92224b8c30da56be03b2ee983d4f80"} Jul 07 19:01:57.864600 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:57.864417 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5d5c5c78bb-kzvl6" event={"ID":"1e7deba9-69b3-4260-b6d0-037f274fb311","Type":"ContainerStarted","Data":"866cb34a169948c20ea9120b465a4de13a8fd780b28975b1980fffb90a235880"} Jul 07 19:01:57.864600 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:57.864450 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5d5c5c78bb-kzvl6" Jul 07 19:01:57.865512 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:57.865488 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-pnvkh" event={"ID":"ecead548-45c5-4195-95e8-a5c058d5958b","Type":"ContainerStarted","Data":"1ca28eeea1879e17f62321d8817f3058fb2b0b7eea12c08c6fa17b019d57051e"} Jul 07 19:01:57.866480 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:57.866463 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-hq4fc" event={"ID":"c338a925-fe6b-40a0-b669-ce49a20cc370","Type":"ContainerStarted","Data":"ccc742ebbac41c7eea5f9590f0934e1e6d876e0ed7c5bca56090631a563f9842"} Jul 07 19:01:57.896559 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:57.896503 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-5d5c5c78bb-kzvl6" podStartSLOduration=99.896487185 podStartE2EDuration="1m39.896487185s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:00:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:01:57.895583178 +0000 UTC m=+100.911587359" watchObservedRunningTime="2026-07-07 19:01:57.896487185 +0000 UTC m=+100.912491366" Jul 07 19:01:58.767138 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:01:58.767105 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-d8t2d" Jul 07 19:02:00.884696 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:00.884659 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-hq4fc" event={"ID":"c338a925-fe6b-40a0-b669-ce49a20cc370","Type":"ContainerStarted","Data":"46e7e5a167fd4e3aec85c4669d4763ef274b1fce689a20abedc1d0a3e7b50db0"} Jul 07 19:02:00.885192 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:00.884723 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-hq4fc" event={"ID":"c338a925-fe6b-40a0-b669-ce49a20cc370","Type":"ContainerStarted","Data":"6caea08607ce0b078f1eb7def4aec7b3fd1cb8396a9ac43563e980aa503e3e6c"} Jul 07 19:02:00.885192 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:00.884777 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-hq4fc" Jul 07 19:02:00.886140 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:00.886117 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-pnvkh" event={"ID":"ecead548-45c5-4195-95e8-a5c058d5958b","Type":"ContainerStarted","Data":"3689af8e1e69a458cc33e0c33861ec2de5d5016b03863743305ce3ae50422f69"} Jul 07 19:02:00.931715 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:00.931642 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-pnvkh" podStartSLOduration=32.844980585 podStartE2EDuration="35.931621061s" totalImagesPullingTime="3.086640476s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:01:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:02:00.931131419 +0000 UTC m=+103.947135616" watchObservedRunningTime="2026-07-07 19:02:00.931621061 +0000 UTC m=+103.947625243" Jul 07 19:02:00.932008 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:00.931978 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-hq4fc" podStartSLOduration=65.63845283 podStartE2EDuration="1m8.931972014s" totalImagesPullingTime="3.293519184s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:00:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:02:00.90617326 +0000 UTC m=+103.922177440" watchObservedRunningTime="2026-07-07 19:02:00.931972014 +0000 UTC m=+103.947976191" Jul 07 19:02:01.106748 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:01.106668 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:02:01.242068 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:01.242033 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:02:01.318098 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:01.318061 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:02:01.535193 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:01.535160 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8db445ca-fea3-4e5e-b946-b4509c6427c8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mb8xx\" (UID: \"8db445ca-fea3-4e5e-b946-b4509c6427c8\") " pod="openshift-insights/insights-runtime-extractor-mb8xx" Jul 07 19:02:01.537456 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:01.537438 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8db445ca-fea3-4e5e-b946-b4509c6427c8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mb8xx\" (UID: \"8db445ca-fea3-4e5e-b946-b4509c6427c8\") " pod="openshift-insights/insights-runtime-extractor-mb8xx" Jul 07 19:02:01.815286 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:01.815185 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-mb8xx" Jul 07 19:02:01.946579 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:01.946549 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-mb8xx"] Jul 07 19:02:01.950684 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:02:01.950663 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8db445ca_fea3_4e5e_b946_b4509c6427c8.slice/crio-a5bd6d16a436205687ec09cf3af76d506c64fc957d39795cf0410cb0cf285b32 WatchSource:0}: Error finding container a5bd6d16a436205687ec09cf3af76d506c64fc957d39795cf0410cb0cf285b32: Status 404 returned error can't find the container with id a5bd6d16a436205687ec09cf3af76d506c64fc957d39795cf0410cb0cf285b32 Jul 07 19:02:02.895546 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:02.895471 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-mb8xx" event={"ID":"8db445ca-fea3-4e5e-b946-b4509c6427c8","Type":"ContainerStarted","Data":"a5bd6d16a436205687ec09cf3af76d506c64fc957d39795cf0410cb0cf285b32"} Jul 07 19:02:04.313967 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:04.313887 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:02:04.357798 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:04.357763 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-567f7b8b7c-dj4zk"] Jul 07 19:02:04.367317 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:04.367292 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-dj4zk" Jul 07 19:02:04.371286 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:04.371122 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Jul 07 19:02:04.371286 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:04.371250 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-xwptm\"" Jul 07 19:02:04.371697 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:04.371484 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Jul 07 19:02:04.371815 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:04.371791 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 07 19:02:04.373580 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:04.373558 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-567f7b8b7c-dj4zk"] Jul 07 19:02:04.396073 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:04.396035 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:02:04.458166 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:04.458126 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6fqrq\" (UniqueName: \"kubernetes.io/projected/53739ab1-4901-467f-9965-1a7f6888b580-kube-api-access-6fqrq\") pod \"prometheus-operator-567f7b8b7c-dj4zk\" (UID: \"53739ab1-4901-467f-9965-1a7f6888b580\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-dj4zk" Jul 07 19:02:04.458352 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:04.458211 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/53739ab1-4901-467f-9965-1a7f6888b580-prometheus-operator-tls\") pod \"prometheus-operator-567f7b8b7c-dj4zk\" (UID: \"53739ab1-4901-467f-9965-1a7f6888b580\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-dj4zk" Jul 07 19:02:04.458352 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:04.458250 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/53739ab1-4901-467f-9965-1a7f6888b580-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-567f7b8b7c-dj4zk\" (UID: \"53739ab1-4901-467f-9965-1a7f6888b580\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-dj4zk" Jul 07 19:02:04.458352 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:04.458289 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/53739ab1-4901-467f-9965-1a7f6888b580-metrics-client-ca\") pod \"prometheus-operator-567f7b8b7c-dj4zk\" (UID: \"53739ab1-4901-467f-9965-1a7f6888b580\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-dj4zk" Jul 07 19:02:04.559725 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:04.559682 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/53739ab1-4901-467f-9965-1a7f6888b580-prometheus-operator-tls\") pod \"prometheus-operator-567f7b8b7c-dj4zk\" (UID: \"53739ab1-4901-467f-9965-1a7f6888b580\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-dj4zk" Jul 07 19:02:04.559911 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:04.559741 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/53739ab1-4901-467f-9965-1a7f6888b580-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-567f7b8b7c-dj4zk\" (UID: \"53739ab1-4901-467f-9965-1a7f6888b580\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-dj4zk" Jul 07 19:02:04.559911 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:04.559784 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/53739ab1-4901-467f-9965-1a7f6888b580-metrics-client-ca\") pod \"prometheus-operator-567f7b8b7c-dj4zk\" (UID: \"53739ab1-4901-467f-9965-1a7f6888b580\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-dj4zk" Jul 07 19:02:04.559911 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:04.559834 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6fqrq\" (UniqueName: \"kubernetes.io/projected/53739ab1-4901-467f-9965-1a7f6888b580-kube-api-access-6fqrq\") pod \"prometheus-operator-567f7b8b7c-dj4zk\" (UID: \"53739ab1-4901-467f-9965-1a7f6888b580\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-dj4zk" Jul 07 19:02:04.560611 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:04.560576 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/53739ab1-4901-467f-9965-1a7f6888b580-metrics-client-ca\") pod \"prometheus-operator-567f7b8b7c-dj4zk\" (UID: \"53739ab1-4901-467f-9965-1a7f6888b580\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-dj4zk" Jul 07 19:02:04.562666 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:04.562639 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/53739ab1-4901-467f-9965-1a7f6888b580-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-567f7b8b7c-dj4zk\" (UID: \"53739ab1-4901-467f-9965-1a7f6888b580\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-dj4zk" Jul 07 19:02:04.562785 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:04.562692 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/53739ab1-4901-467f-9965-1a7f6888b580-prometheus-operator-tls\") pod \"prometheus-operator-567f7b8b7c-dj4zk\" (UID: \"53739ab1-4901-467f-9965-1a7f6888b580\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-dj4zk" Jul 07 19:02:04.569618 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:04.569547 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6fqrq\" (UniqueName: \"kubernetes.io/projected/53739ab1-4901-467f-9965-1a7f6888b580-kube-api-access-6fqrq\") pod \"prometheus-operator-567f7b8b7c-dj4zk\" (UID: \"53739ab1-4901-467f-9965-1a7f6888b580\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-dj4zk" Jul 07 19:02:04.596444 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:04.596406 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:02:04.679617 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:04.679565 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-dj4zk" Jul 07 19:02:05.905328 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:05.905298 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-567f7b8b7c-dj4zk"] Jul 07 19:02:05.908786 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:02:05.908759 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53739ab1_4901_467f_9965_1a7f6888b580.slice/crio-23475cad3525f0ac47abfcad0dab19c13ca970a199dd44e2fd265a70d22070a1 WatchSource:0}: Error finding container 23475cad3525f0ac47abfcad0dab19c13ca970a199dd44e2fd265a70d22070a1: Status 404 returned error can't find the container with id 23475cad3525f0ac47abfcad0dab19c13ca970a199dd44e2fd265a70d22070a1 Jul 07 19:02:05.909553 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:05.909528 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-mb8xx" event={"ID":"8db445ca-fea3-4e5e-b946-b4509c6427c8","Type":"ContainerStarted","Data":"0e7ca5d135131c68e2211d4471359096ab8bef92de2d466d1d5ee663afbbf2c9"} Jul 07 19:02:06.914109 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:06.914075 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-dj4zk" event={"ID":"53739ab1-4901-467f-9965-1a7f6888b580","Type":"ContainerStarted","Data":"23475cad3525f0ac47abfcad0dab19c13ca970a199dd44e2fd265a70d22070a1"} Jul 07 19:02:06.916187 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:06.916148 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-mb8xx" event={"ID":"8db445ca-fea3-4e5e-b946-b4509c6427c8","Type":"ContainerStarted","Data":"f6a74fe5a716affc0745a437155bfe1e338476a09493e705acdcf63e7fe46ccd"} Jul 07 19:02:07.472223 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:07.472190 2563 scope.go:117] "RemoveContainer" containerID="72a783bb23231b4072a6d5003512e8cb9ec34dec207234061683515181666ce7" Jul 07 19:02:07.472447 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:02:07.472406 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-68b757fb67-frt7c_openshift-console-operator(e25ab82f-72fb-4cde-89c4-0a0dd80f5feb)\"" pod="openshift-console-operator/console-operator-68b757fb67-frt7c" podUID="e25ab82f-72fb-4cde-89c4-0a0dd80f5feb" Jul 07 19:02:08.033314 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:08.033276 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:02:08.105879 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:08.105841 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:02:08.237163 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:08.237070 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:02:08.926159 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:08.926075 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-dj4zk" event={"ID":"53739ab1-4901-467f-9965-1a7f6888b580","Type":"ContainerStarted","Data":"4a02d8904707e7009851ed402743accf7ccf92bfc82c1d10d37ab975d5a25c45"} Jul 07 19:02:09.931342 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:09.931311 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-dj4zk" event={"ID":"53739ab1-4901-467f-9965-1a7f6888b580","Type":"ContainerStarted","Data":"ea23ac9bd297c96fa0f1ea96e66da7d0d2807c20e6aabe52853c0d69f88bc7fb"} Jul 07 19:02:09.933286 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:09.933251 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-mb8xx" event={"ID":"8db445ca-fea3-4e5e-b946-b4509c6427c8","Type":"ContainerStarted","Data":"b645deca4593c55a9023bb5c58400a16963cf49b50ad528c918b3b92a5eef3c4"} Jul 07 19:02:09.951054 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:09.951011 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-dj4zk" podStartSLOduration=3.9462991609999998 podStartE2EDuration="5.950996745s" totalImagesPullingTime="2.004697584s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:02:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:02:09.950429113 +0000 UTC m=+112.966433295" watchObservedRunningTime="2026-07-07 19:02:09.950996745 +0000 UTC m=+112.967000980" Jul 07 19:02:09.974571 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:09.974527 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-mb8xx" podStartSLOduration=33.857335546 podStartE2EDuration="40.974515071s" totalImagesPullingTime="7.117179525s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 19:01:29 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:02:09.972729535 +0000 UTC m=+112.988733716" watchObservedRunningTime="2026-07-07 19:02:09.974515071 +0000 UTC m=+112.990519251" Jul 07 19:02:10.892245 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:10.892214 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-hq4fc" Jul 07 19:02:11.495816 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:11.495755 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:02:11.723435 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:11.723396 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:02:11.798658 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:11.798616 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-vjxrs"] Jul 07 19:02:11.802210 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:11.802184 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-vjxrs" Jul 07 19:02:11.804787 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:11.804760 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 07 19:02:11.804916 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:11.804765 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-jc4zb\"" Jul 07 19:02:11.805029 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:11.805016 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 07 19:02:11.805169 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:11.805148 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 07 19:02:11.826945 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:11.826909 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:02:11.918470 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:11.918434 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9b37d03d-9be0-4dc0-a77a-7384e5814a4f-node-exporter-tls\") pod \"node-exporter-vjxrs\" (UID: \"9b37d03d-9be0-4dc0-a77a-7384e5814a4f\") " pod="openshift-monitoring/node-exporter-vjxrs" Jul 07 19:02:11.918470 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:11.918479 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9b37d03d-9be0-4dc0-a77a-7384e5814a4f-node-exporter-textfile\") pod \"node-exporter-vjxrs\" (UID: \"9b37d03d-9be0-4dc0-a77a-7384e5814a4f\") " pod="openshift-monitoring/node-exporter-vjxrs" Jul 07 19:02:11.918722 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:11.918503 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhqz5\" (UniqueName: \"kubernetes.io/projected/9b37d03d-9be0-4dc0-a77a-7384e5814a4f-kube-api-access-zhqz5\") pod \"node-exporter-vjxrs\" (UID: \"9b37d03d-9be0-4dc0-a77a-7384e5814a4f\") " pod="openshift-monitoring/node-exporter-vjxrs" Jul 07 19:02:11.918722 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:11.918522 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9b37d03d-9be0-4dc0-a77a-7384e5814a4f-sys\") pod \"node-exporter-vjxrs\" (UID: \"9b37d03d-9be0-4dc0-a77a-7384e5814a4f\") " pod="openshift-monitoring/node-exporter-vjxrs" Jul 07 19:02:11.918722 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:11.918598 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9b37d03d-9be0-4dc0-a77a-7384e5814a4f-root\") pod \"node-exporter-vjxrs\" (UID: \"9b37d03d-9be0-4dc0-a77a-7384e5814a4f\") " pod="openshift-monitoring/node-exporter-vjxrs" Jul 07 19:02:11.918722 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:11.918628 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9b37d03d-9be0-4dc0-a77a-7384e5814a4f-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-vjxrs\" (UID: \"9b37d03d-9be0-4dc0-a77a-7384e5814a4f\") " pod="openshift-monitoring/node-exporter-vjxrs" Jul 07 19:02:11.918722 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:11.918651 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9b37d03d-9be0-4dc0-a77a-7384e5814a4f-node-exporter-wtmp\") pod \"node-exporter-vjxrs\" (UID: \"9b37d03d-9be0-4dc0-a77a-7384e5814a4f\") " pod="openshift-monitoring/node-exporter-vjxrs" Jul 07 19:02:11.918722 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:11.918686 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9b37d03d-9be0-4dc0-a77a-7384e5814a4f-metrics-client-ca\") pod \"node-exporter-vjxrs\" (UID: \"9b37d03d-9be0-4dc0-a77a-7384e5814a4f\") " pod="openshift-monitoring/node-exporter-vjxrs" Jul 07 19:02:11.918722 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:11.918706 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9b37d03d-9be0-4dc0-a77a-7384e5814a4f-node-exporter-accelerators-collector-config\") pod \"node-exporter-vjxrs\" (UID: \"9b37d03d-9be0-4dc0-a77a-7384e5814a4f\") " pod="openshift-monitoring/node-exporter-vjxrs" Jul 07 19:02:12.019974 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:12.019925 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9b37d03d-9be0-4dc0-a77a-7384e5814a4f-metrics-client-ca\") pod \"node-exporter-vjxrs\" (UID: \"9b37d03d-9be0-4dc0-a77a-7384e5814a4f\") " pod="openshift-monitoring/node-exporter-vjxrs" Jul 07 19:02:12.019974 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:12.019982 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9b37d03d-9be0-4dc0-a77a-7384e5814a4f-node-exporter-accelerators-collector-config\") pod \"node-exporter-vjxrs\" (UID: \"9b37d03d-9be0-4dc0-a77a-7384e5814a4f\") " pod="openshift-monitoring/node-exporter-vjxrs" Jul 07 19:02:12.020233 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:12.020021 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9b37d03d-9be0-4dc0-a77a-7384e5814a4f-node-exporter-tls\") pod \"node-exporter-vjxrs\" (UID: \"9b37d03d-9be0-4dc0-a77a-7384e5814a4f\") " pod="openshift-monitoring/node-exporter-vjxrs" Jul 07 19:02:12.020233 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:12.020043 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9b37d03d-9be0-4dc0-a77a-7384e5814a4f-node-exporter-textfile\") pod \"node-exporter-vjxrs\" (UID: \"9b37d03d-9be0-4dc0-a77a-7384e5814a4f\") " pod="openshift-monitoring/node-exporter-vjxrs" Jul 07 19:02:12.020233 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:02:12.020131 2563 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 07 19:02:12.020233 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:12.020163 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhqz5\" (UniqueName: \"kubernetes.io/projected/9b37d03d-9be0-4dc0-a77a-7384e5814a4f-kube-api-access-zhqz5\") pod \"node-exporter-vjxrs\" (UID: \"9b37d03d-9be0-4dc0-a77a-7384e5814a4f\") " pod="openshift-monitoring/node-exporter-vjxrs" Jul 07 19:02:12.020233 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:02:12.020199 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9b37d03d-9be0-4dc0-a77a-7384e5814a4f-node-exporter-tls podName:9b37d03d-9be0-4dc0-a77a-7384e5814a4f nodeName:}" failed. No retries permitted until 2026-07-07 19:02:12.520180163 +0000 UTC m=+115.536184325 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/9b37d03d-9be0-4dc0-a77a-7384e5814a4f-node-exporter-tls") pod "node-exporter-vjxrs" (UID: "9b37d03d-9be0-4dc0-a77a-7384e5814a4f") : secret "node-exporter-tls" not found Jul 07 19:02:12.020233 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:12.020230 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9b37d03d-9be0-4dc0-a77a-7384e5814a4f-sys\") pod \"node-exporter-vjxrs\" (UID: \"9b37d03d-9be0-4dc0-a77a-7384e5814a4f\") " pod="openshift-monitoring/node-exporter-vjxrs" Jul 07 19:02:12.020535 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:12.020276 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9b37d03d-9be0-4dc0-a77a-7384e5814a4f-root\") pod \"node-exporter-vjxrs\" (UID: \"9b37d03d-9be0-4dc0-a77a-7384e5814a4f\") " pod="openshift-monitoring/node-exporter-vjxrs" Jul 07 19:02:12.020535 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:12.020305 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9b37d03d-9be0-4dc0-a77a-7384e5814a4f-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-vjxrs\" (UID: \"9b37d03d-9be0-4dc0-a77a-7384e5814a4f\") " pod="openshift-monitoring/node-exporter-vjxrs" Jul 07 19:02:12.020535 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:12.020333 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9b37d03d-9be0-4dc0-a77a-7384e5814a4f-node-exporter-wtmp\") pod \"node-exporter-vjxrs\" (UID: \"9b37d03d-9be0-4dc0-a77a-7384e5814a4f\") " pod="openshift-monitoring/node-exporter-vjxrs" Jul 07 19:02:12.020535 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:12.020351 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9b37d03d-9be0-4dc0-a77a-7384e5814a4f-sys\") pod \"node-exporter-vjxrs\" (UID: \"9b37d03d-9be0-4dc0-a77a-7384e5814a4f\") " pod="openshift-monitoring/node-exporter-vjxrs" Jul 07 19:02:12.020535 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:12.020372 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9b37d03d-9be0-4dc0-a77a-7384e5814a4f-root\") pod \"node-exporter-vjxrs\" (UID: \"9b37d03d-9be0-4dc0-a77a-7384e5814a4f\") " pod="openshift-monitoring/node-exporter-vjxrs" Jul 07 19:02:12.020535 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:12.020470 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9b37d03d-9be0-4dc0-a77a-7384e5814a4f-node-exporter-wtmp\") pod \"node-exporter-vjxrs\" (UID: \"9b37d03d-9be0-4dc0-a77a-7384e5814a4f\") " pod="openshift-monitoring/node-exporter-vjxrs" Jul 07 19:02:12.020535 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:12.020470 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9b37d03d-9be0-4dc0-a77a-7384e5814a4f-node-exporter-textfile\") pod \"node-exporter-vjxrs\" (UID: \"9b37d03d-9be0-4dc0-a77a-7384e5814a4f\") " pod="openshift-monitoring/node-exporter-vjxrs" Jul 07 19:02:12.020851 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:12.020634 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9b37d03d-9be0-4dc0-a77a-7384e5814a4f-metrics-client-ca\") pod \"node-exporter-vjxrs\" (UID: \"9b37d03d-9be0-4dc0-a77a-7384e5814a4f\") " pod="openshift-monitoring/node-exporter-vjxrs" Jul 07 19:02:12.020851 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:12.020730 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9b37d03d-9be0-4dc0-a77a-7384e5814a4f-node-exporter-accelerators-collector-config\") pod \"node-exporter-vjxrs\" (UID: \"9b37d03d-9be0-4dc0-a77a-7384e5814a4f\") " pod="openshift-monitoring/node-exporter-vjxrs" Jul 07 19:02:12.022591 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:12.022564 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9b37d03d-9be0-4dc0-a77a-7384e5814a4f-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-vjxrs\" (UID: \"9b37d03d-9be0-4dc0-a77a-7384e5814a4f\") " pod="openshift-monitoring/node-exporter-vjxrs" Jul 07 19:02:12.039081 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:12.039050 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhqz5\" (UniqueName: \"kubernetes.io/projected/9b37d03d-9be0-4dc0-a77a-7384e5814a4f-kube-api-access-zhqz5\") pod \"node-exporter-vjxrs\" (UID: \"9b37d03d-9be0-4dc0-a77a-7384e5814a4f\") " pod="openshift-monitoring/node-exporter-vjxrs" Jul 07 19:02:12.086438 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:12.086343 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:02:12.189242 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:12.189214 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:02:12.337584 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:12.337501 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:02:12.433643 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:12.433606 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:02:12.525246 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:12.525208 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9b37d03d-9be0-4dc0-a77a-7384e5814a4f-node-exporter-tls\") pod \"node-exporter-vjxrs\" (UID: \"9b37d03d-9be0-4dc0-a77a-7384e5814a4f\") " pod="openshift-monitoring/node-exporter-vjxrs" Jul 07 19:02:12.527874 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:12.527846 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9b37d03d-9be0-4dc0-a77a-7384e5814a4f-node-exporter-tls\") pod \"node-exporter-vjxrs\" (UID: \"9b37d03d-9be0-4dc0-a77a-7384e5814a4f\") " pod="openshift-monitoring/node-exporter-vjxrs" Jul 07 19:02:12.538159 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:12.538119 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:02:12.711392 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:12.711304 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-vjxrs" Jul 07 19:02:12.762410 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:12.762373 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:02:12.845970 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:12.845919 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:02:12.863198 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:12.863168 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 07 19:02:12.867022 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:12.866992 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 07 19:02:12.870347 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:12.870182 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 07 19:02:12.870347 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:12.870201 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 07 19:02:12.870347 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:12.870259 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 07 19:02:12.870681 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:12.870658 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 07 19:02:12.870681 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:12.870665 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 07 19:02:12.872091 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:12.872071 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 07 19:02:12.872849 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:12.872799 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 07 19:02:12.873065 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:12.873049 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 07 19:02:12.873153 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:12.873085 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 07 19:02:12.873798 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:12.873718 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-nnz82\"" Jul 07 19:02:12.885701 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:12.885574 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 07 19:02:12.927925 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:12.927883 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/f8c425a2-1747-4fc6-b0d2-4a2cb46e4929-config-volume\") pod \"alertmanager-main-0\" (UID: \"f8c425a2-1747-4fc6-b0d2-4a2cb46e4929\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 19:02:12.928149 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:12.927940 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rcgzc\" (UniqueName: \"kubernetes.io/projected/f8c425a2-1747-4fc6-b0d2-4a2cb46e4929-kube-api-access-rcgzc\") pod \"alertmanager-main-0\" (UID: \"f8c425a2-1747-4fc6-b0d2-4a2cb46e4929\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 19:02:12.928149 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:12.928040 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f8c425a2-1747-4fc6-b0d2-4a2cb46e4929-tls-assets\") pod \"alertmanager-main-0\" (UID: \"f8c425a2-1747-4fc6-b0d2-4a2cb46e4929\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 19:02:12.928149 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:12.928084 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f8c425a2-1747-4fc6-b0d2-4a2cb46e4929-web-config\") pod \"alertmanager-main-0\" (UID: \"f8c425a2-1747-4fc6-b0d2-4a2cb46e4929\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 19:02:12.928149 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:12.928117 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f8c425a2-1747-4fc6-b0d2-4a2cb46e4929-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"f8c425a2-1747-4fc6-b0d2-4a2cb46e4929\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 19:02:12.928149 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:12.928144 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/f8c425a2-1747-4fc6-b0d2-4a2cb46e4929-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"f8c425a2-1747-4fc6-b0d2-4a2cb46e4929\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 19:02:12.928398 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:12.928202 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f8c425a2-1747-4fc6-b0d2-4a2cb46e4929-config-out\") pod \"alertmanager-main-0\" (UID: \"f8c425a2-1747-4fc6-b0d2-4a2cb46e4929\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 19:02:12.928398 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:12.928229 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f8c425a2-1747-4fc6-b0d2-4a2cb46e4929-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"f8c425a2-1747-4fc6-b0d2-4a2cb46e4929\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 19:02:12.928398 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:12.928276 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/f8c425a2-1747-4fc6-b0d2-4a2cb46e4929-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"f8c425a2-1747-4fc6-b0d2-4a2cb46e4929\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 19:02:12.928398 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:12.928322 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f8c425a2-1747-4fc6-b0d2-4a2cb46e4929-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"f8c425a2-1747-4fc6-b0d2-4a2cb46e4929\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 19:02:12.928398 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:12.928370 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f8c425a2-1747-4fc6-b0d2-4a2cb46e4929-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"f8c425a2-1747-4fc6-b0d2-4a2cb46e4929\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 19:02:12.928648 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:12.928404 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f8c425a2-1747-4fc6-b0d2-4a2cb46e4929-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"f8c425a2-1747-4fc6-b0d2-4a2cb46e4929\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 19:02:12.928648 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:12.928490 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f8c425a2-1747-4fc6-b0d2-4a2cb46e4929-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"f8c425a2-1747-4fc6-b0d2-4a2cb46e4929\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 19:02:13.029565 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:13.029466 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f8c425a2-1747-4fc6-b0d2-4a2cb46e4929-tls-assets\") pod \"alertmanager-main-0\" (UID: \"f8c425a2-1747-4fc6-b0d2-4a2cb46e4929\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 19:02:13.029565 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:13.029516 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f8c425a2-1747-4fc6-b0d2-4a2cb46e4929-web-config\") pod \"alertmanager-main-0\" (UID: \"f8c425a2-1747-4fc6-b0d2-4a2cb46e4929\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 19:02:13.029565 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:13.029547 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f8c425a2-1747-4fc6-b0d2-4a2cb46e4929-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"f8c425a2-1747-4fc6-b0d2-4a2cb46e4929\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 19:02:13.029853 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:02:13.029716 2563 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found Jul 07 19:02:13.029853 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:02:13.029793 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f8c425a2-1747-4fc6-b0d2-4a2cb46e4929-secret-alertmanager-main-tls podName:f8c425a2-1747-4fc6-b0d2-4a2cb46e4929 nodeName:}" failed. No retries permitted until 2026-07-07 19:02:13.529771939 +0000 UTC m=+116.545776113 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/f8c425a2-1747-4fc6-b0d2-4a2cb46e4929-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "f8c425a2-1747-4fc6-b0d2-4a2cb46e4929") : secret "alertmanager-main-tls" not found Jul 07 19:02:13.029853 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:13.029812 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/f8c425a2-1747-4fc6-b0d2-4a2cb46e4929-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"f8c425a2-1747-4fc6-b0d2-4a2cb46e4929\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 19:02:13.030037 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:13.029855 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f8c425a2-1747-4fc6-b0d2-4a2cb46e4929-config-out\") pod \"alertmanager-main-0\" (UID: \"f8c425a2-1747-4fc6-b0d2-4a2cb46e4929\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 19:02:13.030037 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:13.029881 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f8c425a2-1747-4fc6-b0d2-4a2cb46e4929-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"f8c425a2-1747-4fc6-b0d2-4a2cb46e4929\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 19:02:13.030037 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:13.029920 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/f8c425a2-1747-4fc6-b0d2-4a2cb46e4929-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"f8c425a2-1747-4fc6-b0d2-4a2cb46e4929\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 19:02:13.030037 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:13.029986 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f8c425a2-1747-4fc6-b0d2-4a2cb46e4929-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"f8c425a2-1747-4fc6-b0d2-4a2cb46e4929\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 19:02:13.030037 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:13.030010 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f8c425a2-1747-4fc6-b0d2-4a2cb46e4929-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"f8c425a2-1747-4fc6-b0d2-4a2cb46e4929\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 19:02:13.030270 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:13.030038 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f8c425a2-1747-4fc6-b0d2-4a2cb46e4929-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"f8c425a2-1747-4fc6-b0d2-4a2cb46e4929\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 19:02:13.030270 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:13.030106 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f8c425a2-1747-4fc6-b0d2-4a2cb46e4929-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"f8c425a2-1747-4fc6-b0d2-4a2cb46e4929\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 19:02:13.030270 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:13.030139 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/f8c425a2-1747-4fc6-b0d2-4a2cb46e4929-config-volume\") pod \"alertmanager-main-0\" (UID: \"f8c425a2-1747-4fc6-b0d2-4a2cb46e4929\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 19:02:13.030270 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:13.030167 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rcgzc\" (UniqueName: \"kubernetes.io/projected/f8c425a2-1747-4fc6-b0d2-4a2cb46e4929-kube-api-access-rcgzc\") pod \"alertmanager-main-0\" (UID: \"f8c425a2-1747-4fc6-b0d2-4a2cb46e4929\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 19:02:13.031448 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:13.031364 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/f8c425a2-1747-4fc6-b0d2-4a2cb46e4929-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"f8c425a2-1747-4fc6-b0d2-4a2cb46e4929\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 19:02:13.031448 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:13.031357 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f8c425a2-1747-4fc6-b0d2-4a2cb46e4929-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"f8c425a2-1747-4fc6-b0d2-4a2cb46e4929\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 19:02:13.032054 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:13.032029 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f8c425a2-1747-4fc6-b0d2-4a2cb46e4929-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"f8c425a2-1747-4fc6-b0d2-4a2cb46e4929\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 19:02:13.033118 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:13.033056 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f8c425a2-1747-4fc6-b0d2-4a2cb46e4929-web-config\") pod \"alertmanager-main-0\" (UID: \"f8c425a2-1747-4fc6-b0d2-4a2cb46e4929\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 19:02:13.033118 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:13.033081 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f8c425a2-1747-4fc6-b0d2-4a2cb46e4929-tls-assets\") pod \"alertmanager-main-0\" (UID: \"f8c425a2-1747-4fc6-b0d2-4a2cb46e4929\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 19:02:13.034667 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:13.034621 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f8c425a2-1747-4fc6-b0d2-4a2cb46e4929-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"f8c425a2-1747-4fc6-b0d2-4a2cb46e4929\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 19:02:13.035548 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:13.035216 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f8c425a2-1747-4fc6-b0d2-4a2cb46e4929-config-out\") pod \"alertmanager-main-0\" (UID: \"f8c425a2-1747-4fc6-b0d2-4a2cb46e4929\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 19:02:13.035548 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:13.035363 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f8c425a2-1747-4fc6-b0d2-4a2cb46e4929-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"f8c425a2-1747-4fc6-b0d2-4a2cb46e4929\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 19:02:13.035548 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:13.035498 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/f8c425a2-1747-4fc6-b0d2-4a2cb46e4929-config-volume\") pod \"alertmanager-main-0\" (UID: \"f8c425a2-1747-4fc6-b0d2-4a2cb46e4929\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 19:02:13.035548 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:13.035517 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f8c425a2-1747-4fc6-b0d2-4a2cb46e4929-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"f8c425a2-1747-4fc6-b0d2-4a2cb46e4929\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 19:02:13.035548 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:13.035536 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/f8c425a2-1747-4fc6-b0d2-4a2cb46e4929-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"f8c425a2-1747-4fc6-b0d2-4a2cb46e4929\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 19:02:13.040313 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:13.040292 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rcgzc\" (UniqueName: \"kubernetes.io/projected/f8c425a2-1747-4fc6-b0d2-4a2cb46e4929-kube-api-access-rcgzc\") pod \"alertmanager-main-0\" (UID: \"f8c425a2-1747-4fc6-b0d2-4a2cb46e4929\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 19:02:13.177098 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:13.177057 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:02:13.534823 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:13.534783 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f8c425a2-1747-4fc6-b0d2-4a2cb46e4929-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"f8c425a2-1747-4fc6-b0d2-4a2cb46e4929\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 19:02:13.537812 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:13.537781 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f8c425a2-1747-4fc6-b0d2-4a2cb46e4929-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"f8c425a2-1747-4fc6-b0d2-4a2cb46e4929\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 19:02:13.578110 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:02:13.578080 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b37d03d_9be0_4dc0_a77a_7384e5814a4f.slice/crio-60faa665464e1a9c3b613b154b302dc2bf8b4b6a64b3280eaf8b03c5cb917646 WatchSource:0}: Error finding container 60faa665464e1a9c3b613b154b302dc2bf8b4b6a64b3280eaf8b03c5cb917646: Status 404 returned error can't find the container with id 60faa665464e1a9c3b613b154b302dc2bf8b4b6a64b3280eaf8b03c5cb917646 Jul 07 19:02:13.779977 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:13.779927 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 07 19:02:13.930283 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:13.930239 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 07 19:02:13.934056 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:02:13.934029 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8c425a2_1747_4fc6_b0d2_4a2cb46e4929.slice/crio-850930ea7d56f90ff35cf701b269be546b063422b98993489d9b1393dfc28476 WatchSource:0}: Error finding container 850930ea7d56f90ff35cf701b269be546b063422b98993489d9b1393dfc28476: Status 404 returned error can't find the container with id 850930ea7d56f90ff35cf701b269be546b063422b98993489d9b1393dfc28476 Jul 07 19:02:13.949780 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:13.949728 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7f5f69ff4c-v54qk" event={"ID":"d2eadf10-3504-4db4-9e20-2afaeaa81ce5","Type":"ContainerStarted","Data":"940486165ca6784d83dfe1b23f6130ad6fdda9fd88a72fff0d0ed0096a1eeabe"} Jul 07 19:02:13.950086 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:13.950031 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7f5f69ff4c-v54qk" Jul 07 19:02:13.951528 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:13.951501 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f8c425a2-1747-4fc6-b0d2-4a2cb46e4929","Type":"ContainerStarted","Data":"850930ea7d56f90ff35cf701b269be546b063422b98993489d9b1393dfc28476"} Jul 07 19:02:13.951667 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:13.951631 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7f5f69ff4c-v54qk" Jul 07 19:02:13.953007 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:13.952982 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vjxrs" event={"ID":"9b37d03d-9be0-4dc0-a77a-7384e5814a4f","Type":"ContainerStarted","Data":"60faa665464e1a9c3b613b154b302dc2bf8b4b6a64b3280eaf8b03c5cb917646"} Jul 07 19:02:13.970939 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:13.970875 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7f5f69ff4c-v54qk" podStartSLOduration=1.259458165 podStartE2EDuration="19.970852451s" totalImagesPullingTime="18.711394286s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:01:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:02:13.968591885 +0000 UTC m=+116.984596068" watchObservedRunningTime="2026-07-07 19:02:13.970852451 +0000 UTC m=+116.986856632" Jul 07 19:02:14.772467 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:14.772357 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:02:14.850571 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:14.850534 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:02:14.937693 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:14.937662 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:02:14.959359 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:14.959331 2563 generic.go:356] "Generic (PLEG): container finished" podID="9b37d03d-9be0-4dc0-a77a-7384e5814a4f" containerID="4d13df2b2f6ed9ffdc603e39bd3a77052cc82045a9e4b29a7935e11798e456e1" exitCode=0 Jul 07 19:02:14.959488 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:14.959418 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vjxrs" event={"ID":"9b37d03d-9be0-4dc0-a77a-7384e5814a4f","Type":"ContainerDied","Data":"4d13df2b2f6ed9ffdc603e39bd3a77052cc82045a9e4b29a7935e11798e456e1"} Jul 07 19:02:15.142381 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:15.142340 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:02:15.963757 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:15.963720 2563 generic.go:356] "Generic (PLEG): container finished" podID="f8c425a2-1747-4fc6-b0d2-4a2cb46e4929" containerID="bb7ea7b8f5029487b127bd8eeb1aeefb0c2fbccf711e1fae132d0310e6581328" exitCode=0 Jul 07 19:02:15.964217 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:15.963799 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f8c425a2-1747-4fc6-b0d2-4a2cb46e4929","Type":"ContainerDied","Data":"bb7ea7b8f5029487b127bd8eeb1aeefb0c2fbccf711e1fae132d0310e6581328"} Jul 07 19:02:15.966000 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:15.965975 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vjxrs" event={"ID":"9b37d03d-9be0-4dc0-a77a-7384e5814a4f","Type":"ContainerStarted","Data":"f98e2eb392a313fa464e5f10641a9c11d17baf7b3c63ba0b2923945726f7b8f4"} Jul 07 19:02:15.966091 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:15.966019 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vjxrs" event={"ID":"9b37d03d-9be0-4dc0-a77a-7384e5814a4f","Type":"ContainerStarted","Data":"fcab601223b4643f27e8adcf25b7fef3877ff2840e5085f46027e734c10e738c"} Jul 07 19:02:16.045517 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:16.045464 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-vjxrs" podStartSLOduration=4.129457046 podStartE2EDuration="5.045447402s" totalImagesPullingTime="851.972296ms" totalInitContainerRuntime="64.01806ms" isStatefulPod=true podCreationTimestamp="2026-07-07 19:02:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:02:16.044574816 +0000 UTC m=+119.060578998" watchObservedRunningTime="2026-07-07 19:02:16.045447402 +0000 UTC m=+119.061451612" Jul 07 19:02:16.205598 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:16.205556 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-5cb97f8f7b-stvw8"] Jul 07 19:02:16.208772 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:16.208749 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-5cb97f8f7b-stvw8" Jul 07 19:02:16.211471 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:16.211445 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Jul 07 19:02:16.211618 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:16.211578 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Jul 07 19:02:16.211855 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:16.211826 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-f7dfn0ck416v3\"" Jul 07 19:02:16.211855 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:16.211845 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Jul 07 19:02:16.212028 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:16.211874 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 07 19:02:16.212028 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:16.211897 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-v5t86\"" Jul 07 19:02:16.220354 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:16.220331 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-5cb97f8f7b-stvw8"] Jul 07 19:02:16.259652 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:16.259615 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/6f2e13c9-5c07-429e-850a-3816f2c24a21-secret-metrics-server-tls\") pod \"metrics-server-5cb97f8f7b-stvw8\" (UID: \"6f2e13c9-5c07-429e-850a-3816f2c24a21\") " pod="openshift-monitoring/metrics-server-5cb97f8f7b-stvw8" Jul 07 19:02:16.259652 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:16.259650 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/6f2e13c9-5c07-429e-850a-3816f2c24a21-secret-metrics-server-client-certs\") pod \"metrics-server-5cb97f8f7b-stvw8\" (UID: \"6f2e13c9-5c07-429e-850a-3816f2c24a21\") " pod="openshift-monitoring/metrics-server-5cb97f8f7b-stvw8" Jul 07 19:02:16.259866 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:16.259683 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x98v9\" (UniqueName: \"kubernetes.io/projected/6f2e13c9-5c07-429e-850a-3816f2c24a21-kube-api-access-x98v9\") pod \"metrics-server-5cb97f8f7b-stvw8\" (UID: \"6f2e13c9-5c07-429e-850a-3816f2c24a21\") " pod="openshift-monitoring/metrics-server-5cb97f8f7b-stvw8" Jul 07 19:02:16.259866 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:16.259744 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/6f2e13c9-5c07-429e-850a-3816f2c24a21-audit-log\") pod \"metrics-server-5cb97f8f7b-stvw8\" (UID: \"6f2e13c9-5c07-429e-850a-3816f2c24a21\") " pod="openshift-monitoring/metrics-server-5cb97f8f7b-stvw8" Jul 07 19:02:16.259866 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:16.259786 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/6f2e13c9-5c07-429e-850a-3816f2c24a21-metrics-server-audit-profiles\") pod \"metrics-server-5cb97f8f7b-stvw8\" (UID: \"6f2e13c9-5c07-429e-850a-3816f2c24a21\") " pod="openshift-monitoring/metrics-server-5cb97f8f7b-stvw8" Jul 07 19:02:16.259866 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:16.259846 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/6f2e13c9-5c07-429e-850a-3816f2c24a21-client-ca-bundle\") pod \"metrics-server-5cb97f8f7b-stvw8\" (UID: \"6f2e13c9-5c07-429e-850a-3816f2c24a21\") " pod="openshift-monitoring/metrics-server-5cb97f8f7b-stvw8" Jul 07 19:02:16.260033 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:16.259884 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6f2e13c9-5c07-429e-850a-3816f2c24a21-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5cb97f8f7b-stvw8\" (UID: \"6f2e13c9-5c07-429e-850a-3816f2c24a21\") " pod="openshift-monitoring/metrics-server-5cb97f8f7b-stvw8" Jul 07 19:02:16.361158 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:16.361123 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x98v9\" (UniqueName: \"kubernetes.io/projected/6f2e13c9-5c07-429e-850a-3816f2c24a21-kube-api-access-x98v9\") pod \"metrics-server-5cb97f8f7b-stvw8\" (UID: \"6f2e13c9-5c07-429e-850a-3816f2c24a21\") " pod="openshift-monitoring/metrics-server-5cb97f8f7b-stvw8" Jul 07 19:02:16.361158 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:16.361159 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/6f2e13c9-5c07-429e-850a-3816f2c24a21-audit-log\") pod \"metrics-server-5cb97f8f7b-stvw8\" (UID: \"6f2e13c9-5c07-429e-850a-3816f2c24a21\") " pod="openshift-monitoring/metrics-server-5cb97f8f7b-stvw8" Jul 07 19:02:16.361344 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:16.361178 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/6f2e13c9-5c07-429e-850a-3816f2c24a21-metrics-server-audit-profiles\") pod \"metrics-server-5cb97f8f7b-stvw8\" (UID: \"6f2e13c9-5c07-429e-850a-3816f2c24a21\") " pod="openshift-monitoring/metrics-server-5cb97f8f7b-stvw8" Jul 07 19:02:16.361344 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:16.361307 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/6f2e13c9-5c07-429e-850a-3816f2c24a21-client-ca-bundle\") pod \"metrics-server-5cb97f8f7b-stvw8\" (UID: \"6f2e13c9-5c07-429e-850a-3816f2c24a21\") " pod="openshift-monitoring/metrics-server-5cb97f8f7b-stvw8" Jul 07 19:02:16.361426 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:16.361389 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6f2e13c9-5c07-429e-850a-3816f2c24a21-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5cb97f8f7b-stvw8\" (UID: \"6f2e13c9-5c07-429e-850a-3816f2c24a21\") " pod="openshift-monitoring/metrics-server-5cb97f8f7b-stvw8" Jul 07 19:02:16.361471 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:16.361439 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/6f2e13c9-5c07-429e-850a-3816f2c24a21-secret-metrics-server-tls\") pod \"metrics-server-5cb97f8f7b-stvw8\" (UID: \"6f2e13c9-5c07-429e-850a-3816f2c24a21\") " pod="openshift-monitoring/metrics-server-5cb97f8f7b-stvw8" Jul 07 19:02:16.361607 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:16.361588 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/6f2e13c9-5c07-429e-850a-3816f2c24a21-secret-metrics-server-client-certs\") pod \"metrics-server-5cb97f8f7b-stvw8\" (UID: \"6f2e13c9-5c07-429e-850a-3816f2c24a21\") " pod="openshift-monitoring/metrics-server-5cb97f8f7b-stvw8" Jul 07 19:02:16.361607 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:16.361601 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/6f2e13c9-5c07-429e-850a-3816f2c24a21-audit-log\") pod \"metrics-server-5cb97f8f7b-stvw8\" (UID: \"6f2e13c9-5c07-429e-850a-3816f2c24a21\") " pod="openshift-monitoring/metrics-server-5cb97f8f7b-stvw8" Jul 07 19:02:16.362069 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:16.362037 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6f2e13c9-5c07-429e-850a-3816f2c24a21-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5cb97f8f7b-stvw8\" (UID: \"6f2e13c9-5c07-429e-850a-3816f2c24a21\") " pod="openshift-monitoring/metrics-server-5cb97f8f7b-stvw8" Jul 07 19:02:16.362198 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:16.362155 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/6f2e13c9-5c07-429e-850a-3816f2c24a21-metrics-server-audit-profiles\") pod \"metrics-server-5cb97f8f7b-stvw8\" (UID: \"6f2e13c9-5c07-429e-850a-3816f2c24a21\") " pod="openshift-monitoring/metrics-server-5cb97f8f7b-stvw8" Jul 07 19:02:16.363818 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:16.363795 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/6f2e13c9-5c07-429e-850a-3816f2c24a21-secret-metrics-server-tls\") pod \"metrics-server-5cb97f8f7b-stvw8\" (UID: \"6f2e13c9-5c07-429e-850a-3816f2c24a21\") " pod="openshift-monitoring/metrics-server-5cb97f8f7b-stvw8" Jul 07 19:02:16.363911 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:16.363854 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/6f2e13c9-5c07-429e-850a-3816f2c24a21-secret-metrics-server-client-certs\") pod \"metrics-server-5cb97f8f7b-stvw8\" (UID: \"6f2e13c9-5c07-429e-850a-3816f2c24a21\") " pod="openshift-monitoring/metrics-server-5cb97f8f7b-stvw8" Jul 07 19:02:16.364049 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:16.364029 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/6f2e13c9-5c07-429e-850a-3816f2c24a21-client-ca-bundle\") pod \"metrics-server-5cb97f8f7b-stvw8\" (UID: \"6f2e13c9-5c07-429e-850a-3816f2c24a21\") " pod="openshift-monitoring/metrics-server-5cb97f8f7b-stvw8" Jul 07 19:02:16.370262 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:16.370240 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x98v9\" (UniqueName: \"kubernetes.io/projected/6f2e13c9-5c07-429e-850a-3816f2c24a21-kube-api-access-x98v9\") pod \"metrics-server-5cb97f8f7b-stvw8\" (UID: \"6f2e13c9-5c07-429e-850a-3816f2c24a21\") " pod="openshift-monitoring/metrics-server-5cb97f8f7b-stvw8" Jul 07 19:02:16.499489 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:16.499397 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:02:16.519175 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:16.519134 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-5cb97f8f7b-stvw8" Jul 07 19:02:16.666847 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:16.666811 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-5cb97f8f7b-stvw8"] Jul 07 19:02:16.670449 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:02:16.670420 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f2e13c9_5c07_429e_850a_3816f2c24a21.slice/crio-40e79bf43d74fc120e13137c22d67864a962f00ae5cfc6fb424b736be633931d WatchSource:0}: Error finding container 40e79bf43d74fc120e13137c22d67864a962f00ae5cfc6fb424b736be633931d: Status 404 returned error can't find the container with id 40e79bf43d74fc120e13137c22d67864a962f00ae5cfc6fb424b736be633931d Jul 07 19:02:16.789538 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:16.789500 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:02:16.970588 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:16.970558 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-5cb97f8f7b-stvw8" event={"ID":"6f2e13c9-5c07-429e-850a-3816f2c24a21","Type":"ContainerStarted","Data":"40e79bf43d74fc120e13137c22d67864a962f00ae5cfc6fb424b736be633931d"} Jul 07 19:02:17.083843 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:17.083756 2563 patch_prober.go:28] interesting pod/image-registry-5d5c5c78bb-kzvl6 container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jul 07 19:02:17.083843 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:17.083815 2563 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-5d5c5c78bb-kzvl6" podUID="1e7deba9-69b3-4260-b6d0-037f274fb311" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 07 19:02:17.857072 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:17.857016 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5d5c5c78bb-kzvl6"] Jul 07 19:02:17.863816 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:17.863722 2563 patch_prober.go:28] interesting pod/image-registry-5d5c5c78bb-kzvl6 container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jul 07 19:02:17.863816 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:17.863782 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-5d5c5c78bb-kzvl6" podUID="1e7deba9-69b3-4260-b6d0-037f274fb311" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 07 19:02:17.978023 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:17.977812 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f8c425a2-1747-4fc6-b0d2-4a2cb46e4929","Type":"ContainerStarted","Data":"c12957fe91d96187144b551bc675d7a01f04657b8fa6e572f08437fd88c37abe"} Jul 07 19:02:17.978023 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:17.977900 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f8c425a2-1747-4fc6-b0d2-4a2cb46e4929","Type":"ContainerStarted","Data":"2cd3e07c61d50b29520bef559caefb7996da77fe3596311da99a69b23498ade3"} Jul 07 19:02:17.998382 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:17.998329 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:02:18.079305 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:18.079159 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:02:18.184189 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:18.184151 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:02:18.197013 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:18.196247 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 07 19:02:18.200876 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:18.200847 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:02:18.203937 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:18.203905 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 07 19:02:18.204121 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:18.204076 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 07 19:02:18.204195 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:18.203904 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-d9kr8\"" Jul 07 19:02:18.204884 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:18.204860 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 07 19:02:18.206560 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:18.206533 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 07 19:02:18.206560 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:18.206544 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 07 19:02:18.206717 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:18.206564 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 07 19:02:18.206853 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:18.206827 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 07 19:02:18.206990 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:18.206896 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 07 19:02:18.207108 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:18.207088 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-d3n2opql6gns4\"" Jul 07 19:02:18.207489 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:18.207347 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 07 19:02:18.208537 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:18.208519 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 07 19:02:18.215257 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:18.214802 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 07 19:02:18.215257 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:18.215093 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 07 19:02:18.215257 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:18.215133 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 07 19:02:18.216051 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:18.216032 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 07 19:02:18.222966 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:18.222853 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 07 19:02:18.278217 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:18.278094 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:02:18.278578 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:18.278559 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/d27c9f8c-0884-467d-ae62-9193d6950191-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"d27c9f8c-0884-467d-ae62-9193d6950191\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:02:18.278659 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:18.278604 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/d27c9f8c-0884-467d-ae62-9193d6950191-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"d27c9f8c-0884-467d-ae62-9193d6950191\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:02:18.278659 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:18.278629 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d27c9f8c-0884-467d-ae62-9193d6950191-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"d27c9f8c-0884-467d-ae62-9193d6950191\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:02:18.278768 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:18.278743 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/d27c9f8c-0884-467d-ae62-9193d6950191-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"d27c9f8c-0884-467d-ae62-9193d6950191\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:02:18.278822 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:18.278783 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d27c9f8c-0884-467d-ae62-9193d6950191-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"d27c9f8c-0884-467d-ae62-9193d6950191\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:02:18.278871 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:18.278835 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/d27c9f8c-0884-467d-ae62-9193d6950191-config\") pod \"prometheus-k8s-0\" (UID: \"d27c9f8c-0884-467d-ae62-9193d6950191\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:02:18.278871 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:18.278860 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rgtvz\" (UniqueName: \"kubernetes.io/projected/d27c9f8c-0884-467d-ae62-9193d6950191-kube-api-access-rgtvz\") pod \"prometheus-k8s-0\" (UID: \"d27c9f8c-0884-467d-ae62-9193d6950191\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:02:18.278996 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:18.278885 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/d27c9f8c-0884-467d-ae62-9193d6950191-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"d27c9f8c-0884-467d-ae62-9193d6950191\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:02:18.278996 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:18.278910 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/d27c9f8c-0884-467d-ae62-9193d6950191-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"d27c9f8c-0884-467d-ae62-9193d6950191\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:02:18.279646 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:18.279576 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d27c9f8c-0884-467d-ae62-9193d6950191-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"d27c9f8c-0884-467d-ae62-9193d6950191\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:02:18.279749 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:18.279680 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d27c9f8c-0884-467d-ae62-9193d6950191-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d27c9f8c-0884-467d-ae62-9193d6950191\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:02:18.279749 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:18.279728 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d27c9f8c-0884-467d-ae62-9193d6950191-config-out\") pod \"prometheus-k8s-0\" (UID: \"d27c9f8c-0884-467d-ae62-9193d6950191\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:02:18.279853 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:18.279752 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/d27c9f8c-0884-467d-ae62-9193d6950191-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"d27c9f8c-0884-467d-ae62-9193d6950191\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:02:18.279853 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:18.279820 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d27c9f8c-0884-467d-ae62-9193d6950191-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"d27c9f8c-0884-467d-ae62-9193d6950191\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:02:18.279944 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:18.279869 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/d27c9f8c-0884-467d-ae62-9193d6950191-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"d27c9f8c-0884-467d-ae62-9193d6950191\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:02:18.279944 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:18.279898 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d27c9f8c-0884-467d-ae62-9193d6950191-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"d27c9f8c-0884-467d-ae62-9193d6950191\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:02:18.279944 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:18.279919 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d27c9f8c-0884-467d-ae62-9193d6950191-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d27c9f8c-0884-467d-ae62-9193d6950191\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:02:18.280116 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:18.279974 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/d27c9f8c-0884-467d-ae62-9193d6950191-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"d27c9f8c-0884-467d-ae62-9193d6950191\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:02:18.280116 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:18.280056 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d27c9f8c-0884-467d-ae62-9193d6950191-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d27c9f8c-0884-467d-ae62-9193d6950191\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:02:18.280205 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:18.280184 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d27c9f8c-0884-467d-ae62-9193d6950191-web-config\") pod \"prometheus-k8s-0\" (UID: \"d27c9f8c-0884-467d-ae62-9193d6950191\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:02:18.381142 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:18.381102 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d27c9f8c-0884-467d-ae62-9193d6950191-web-config\") pod \"prometheus-k8s-0\" (UID: \"d27c9f8c-0884-467d-ae62-9193d6950191\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:02:18.381318 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:18.381160 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/d27c9f8c-0884-467d-ae62-9193d6950191-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"d27c9f8c-0884-467d-ae62-9193d6950191\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:02:18.381318 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:18.381220 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/d27c9f8c-0884-467d-ae62-9193d6950191-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"d27c9f8c-0884-467d-ae62-9193d6950191\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:02:18.381318 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:18.381246 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d27c9f8c-0884-467d-ae62-9193d6950191-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"d27c9f8c-0884-467d-ae62-9193d6950191\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:02:18.381318 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:18.381273 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/d27c9f8c-0884-467d-ae62-9193d6950191-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"d27c9f8c-0884-467d-ae62-9193d6950191\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:02:18.381318 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:18.381292 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d27c9f8c-0884-467d-ae62-9193d6950191-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"d27c9f8c-0884-467d-ae62-9193d6950191\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:02:18.381318 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:18.381317 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/d27c9f8c-0884-467d-ae62-9193d6950191-config\") pod \"prometheus-k8s-0\" (UID: \"d27c9f8c-0884-467d-ae62-9193d6950191\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:02:18.381631 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:18.381340 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rgtvz\" (UniqueName: \"kubernetes.io/projected/d27c9f8c-0884-467d-ae62-9193d6950191-kube-api-access-rgtvz\") pod \"prometheus-k8s-0\" (UID: \"d27c9f8c-0884-467d-ae62-9193d6950191\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:02:18.381631 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:18.381367 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/d27c9f8c-0884-467d-ae62-9193d6950191-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"d27c9f8c-0884-467d-ae62-9193d6950191\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:02:18.381631 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:18.381390 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/d27c9f8c-0884-467d-ae62-9193d6950191-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"d27c9f8c-0884-467d-ae62-9193d6950191\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:02:18.381631 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:18.381412 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d27c9f8c-0884-467d-ae62-9193d6950191-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"d27c9f8c-0884-467d-ae62-9193d6950191\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:02:18.381631 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:18.381433 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d27c9f8c-0884-467d-ae62-9193d6950191-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d27c9f8c-0884-467d-ae62-9193d6950191\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:02:18.381631 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:18.381472 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d27c9f8c-0884-467d-ae62-9193d6950191-config-out\") pod \"prometheus-k8s-0\" (UID: \"d27c9f8c-0884-467d-ae62-9193d6950191\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:02:18.381631 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:18.381496 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/d27c9f8c-0884-467d-ae62-9193d6950191-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"d27c9f8c-0884-467d-ae62-9193d6950191\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:02:18.381631 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:18.381517 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d27c9f8c-0884-467d-ae62-9193d6950191-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"d27c9f8c-0884-467d-ae62-9193d6950191\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:02:18.381631 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:18.381550 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/d27c9f8c-0884-467d-ae62-9193d6950191-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"d27c9f8c-0884-467d-ae62-9193d6950191\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:02:18.381631 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:18.381584 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d27c9f8c-0884-467d-ae62-9193d6950191-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"d27c9f8c-0884-467d-ae62-9193d6950191\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:02:18.381631 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:18.381609 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d27c9f8c-0884-467d-ae62-9193d6950191-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d27c9f8c-0884-467d-ae62-9193d6950191\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:02:18.381631 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:18.381634 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/d27c9f8c-0884-467d-ae62-9193d6950191-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"d27c9f8c-0884-467d-ae62-9193d6950191\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:02:18.382241 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:18.381664 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d27c9f8c-0884-467d-ae62-9193d6950191-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d27c9f8c-0884-467d-ae62-9193d6950191\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:02:18.384172 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:18.384137 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d27c9f8c-0884-467d-ae62-9193d6950191-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d27c9f8c-0884-467d-ae62-9193d6950191\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:02:18.387198 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:18.386322 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/d27c9f8c-0884-467d-ae62-9193d6950191-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"d27c9f8c-0884-467d-ae62-9193d6950191\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:02:18.387198 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:18.387152 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/d27c9f8c-0884-467d-ae62-9193d6950191-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"d27c9f8c-0884-467d-ae62-9193d6950191\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:02:18.387873 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:18.387840 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d27c9f8c-0884-467d-ae62-9193d6950191-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"d27c9f8c-0884-467d-ae62-9193d6950191\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:02:18.388916 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:18.388893 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d27c9f8c-0884-467d-ae62-9193d6950191-config-out\") pod \"prometheus-k8s-0\" (UID: \"d27c9f8c-0884-467d-ae62-9193d6950191\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:02:18.389647 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:18.389625 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/d27c9f8c-0884-467d-ae62-9193d6950191-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"d27c9f8c-0884-467d-ae62-9193d6950191\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:02:18.389934 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:18.389908 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/d27c9f8c-0884-467d-ae62-9193d6950191-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"d27c9f8c-0884-467d-ae62-9193d6950191\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:02:18.390115 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:18.390099 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d27c9f8c-0884-467d-ae62-9193d6950191-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d27c9f8c-0884-467d-ae62-9193d6950191\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:02:18.390469 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:18.390445 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/d27c9f8c-0884-467d-ae62-9193d6950191-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"d27c9f8c-0884-467d-ae62-9193d6950191\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:02:18.390667 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:18.390652 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d27c9f8c-0884-467d-ae62-9193d6950191-web-config\") pod \"prometheus-k8s-0\" (UID: \"d27c9f8c-0884-467d-ae62-9193d6950191\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:02:18.390924 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:18.390822 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/d27c9f8c-0884-467d-ae62-9193d6950191-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"d27c9f8c-0884-467d-ae62-9193d6950191\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:02:18.391998 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:18.391912 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d27c9f8c-0884-467d-ae62-9193d6950191-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d27c9f8c-0884-467d-ae62-9193d6950191\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:02:18.392287 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:18.392200 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/d27c9f8c-0884-467d-ae62-9193d6950191-config\") pod \"prometheus-k8s-0\" (UID: \"d27c9f8c-0884-467d-ae62-9193d6950191\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:02:18.392599 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:18.392575 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d27c9f8c-0884-467d-ae62-9193d6950191-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"d27c9f8c-0884-467d-ae62-9193d6950191\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:02:18.396877 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:18.396850 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/d27c9f8c-0884-467d-ae62-9193d6950191-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"d27c9f8c-0884-467d-ae62-9193d6950191\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:02:18.397063 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:18.396859 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/d27c9f8c-0884-467d-ae62-9193d6950191-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"d27c9f8c-0884-467d-ae62-9193d6950191\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:02:18.399038 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:18.398995 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d27c9f8c-0884-467d-ae62-9193d6950191-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"d27c9f8c-0884-467d-ae62-9193d6950191\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:02:18.399363 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:18.399340 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d27c9f8c-0884-467d-ae62-9193d6950191-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"d27c9f8c-0884-467d-ae62-9193d6950191\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:02:18.399450 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:18.399342 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d27c9f8c-0884-467d-ae62-9193d6950191-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"d27c9f8c-0884-467d-ae62-9193d6950191\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:02:18.401450 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:18.401410 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rgtvz\" (UniqueName: \"kubernetes.io/projected/d27c9f8c-0884-467d-ae62-9193d6950191-kube-api-access-rgtvz\") pod \"prometheus-k8s-0\" (UID: \"d27c9f8c-0884-467d-ae62-9193d6950191\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:02:18.471651 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:18.471615 2563 scope.go:117] "RemoveContainer" containerID="72a783bb23231b4072a6d5003512e8cb9ec34dec207234061683515181666ce7" Jul 07 19:02:18.516184 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:18.516145 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:02:18.723593 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:18.723542 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 07 19:02:18.727875 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:02:18.727815 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd27c9f8c_0884_467d_ae62_9193d6950191.slice/crio-ae73344b97776bcaca79451e6faf743e20c98450bff248bd1ce883df01dd9526 WatchSource:0}: Error finding container ae73344b97776bcaca79451e6faf743e20c98450bff248bd1ce883df01dd9526: Status 404 returned error can't find the container with id ae73344b97776bcaca79451e6faf743e20c98450bff248bd1ce883df01dd9526 Jul 07 19:02:18.987339 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:18.987314 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-frt7c_e25ab82f-72fb-4cde-89c4-0a0dd80f5feb/console-operator/2.log" Jul 07 19:02:18.987746 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:18.987417 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-68b757fb67-frt7c" event={"ID":"e25ab82f-72fb-4cde-89c4-0a0dd80f5feb","Type":"ContainerStarted","Data":"bbd1df8e21b6ffe1f108c49b61618e624f70c6c32d98497d3717d0b4ed3f60cc"} Jul 07 19:02:18.988034 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:18.987910 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-68b757fb67-frt7c" Jul 07 19:02:18.993971 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:18.993918 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f8c425a2-1747-4fc6-b0d2-4a2cb46e4929","Type":"ContainerStarted","Data":"1a71b5f2b48d7c2e2686fd9b852a6bbe82975eefc3a3545b7e9fbff5b1ffb13d"} Jul 07 19:02:18.994113 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:18.994016 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f8c425a2-1747-4fc6-b0d2-4a2cb46e4929","Type":"ContainerStarted","Data":"59dd09e938e74c2c34a783745790e1f83739d669d7f596c5e4907d0167f36f68"} Jul 07 19:02:18.994113 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:18.994050 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f8c425a2-1747-4fc6-b0d2-4a2cb46e4929","Type":"ContainerStarted","Data":"d5641498974d7e0650186c13b061224b4f64d4ad8f25190d923db39c589b93cb"} Jul 07 19:02:18.995731 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:18.995699 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-5cb97f8f7b-stvw8" event={"ID":"6f2e13c9-5c07-429e-850a-3816f2c24a21","Type":"ContainerStarted","Data":"2ca82b4e72930f78bab9f7d6d045115333e99c160f2e949da804bb338e0eb75c"} Jul 07 19:02:18.997434 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:18.997359 2563 generic.go:356] "Generic (PLEG): container finished" podID="d27c9f8c-0884-467d-ae62-9193d6950191" containerID="f9e867a77df9e6aee0670b91ed40795b79a63a7a63fcf93801429e87dc94924c" exitCode=0 Jul 07 19:02:18.997434 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:18.997407 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d27c9f8c-0884-467d-ae62-9193d6950191","Type":"ContainerDied","Data":"f9e867a77df9e6aee0670b91ed40795b79a63a7a63fcf93801429e87dc94924c"} Jul 07 19:02:18.997590 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:18.997467 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d27c9f8c-0884-467d-ae62-9193d6950191","Type":"ContainerStarted","Data":"ae73344b97776bcaca79451e6faf743e20c98450bff248bd1ce883df01dd9526"} Jul 07 19:02:19.011820 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:19.011761 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-68b757fb67-frt7c" podStartSLOduration=51.259510461 podStartE2EDuration="54.011742717s" totalImagesPullingTime="2.752232256s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:01:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:02:19.010414501 +0000 UTC m=+122.026418684" watchObservedRunningTime="2026-07-07 19:02:19.011742717 +0000 UTC m=+122.027746899" Jul 07 19:02:19.031619 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:19.031562 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-5cb97f8f7b-stvw8" podStartSLOduration=1.097059151 podStartE2EDuration="3.031542003s" totalImagesPullingTime="1.934482852s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:02:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:02:19.029098684 +0000 UTC m=+122.045102876" watchObservedRunningTime="2026-07-07 19:02:19.031542003 +0000 UTC m=+122.047546185" Jul 07 19:02:19.866588 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:19.866554 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-68b757fb67-frt7c" Jul 07 19:02:20.015831 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:20.015753 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f8c425a2-1747-4fc6-b0d2-4a2cb46e4929","Type":"ContainerStarted","Data":"b6560799533213a93b1e36e5b7eca6971f3c4d26e2fa29293801b752e334d35f"} Jul 07 19:02:20.118589 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:20.118486 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.776391471 podStartE2EDuration="8.118469091s" totalImagesPullingTime="4.324472676s" totalInitContainerRuntime="17.604944ms" isStatefulPod=false podCreationTimestamp="2026-07-07 19:02:12 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:02:20.107143032 +0000 UTC m=+123.123147216" watchObservedRunningTime="2026-07-07 19:02:20.118469091 +0000 UTC m=+123.134473272" Jul 07 19:02:20.154930 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:20.154839 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:02:20.253663 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:20.253623 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:02:20.385252 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:20.385154 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:02:20.542278 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:20.542245 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:02:20.942429 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:20.942397 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:02:21.016935 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:21.016880 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:02:21.568296 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:21.568260 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:02:21.777619 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:21.777576 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:02:21.897937 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:21.897840 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:02:22.589497 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:22.589414 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:02:22.690192 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:22.690153 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:02:24.034443 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:24.034403 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d27c9f8c-0884-467d-ae62-9193d6950191","Type":"ContainerStarted","Data":"e6fe0065e5a888e34565995c53a516d658ec05c20c9c006207a83fa67796b50e"} Jul 07 19:02:24.034882 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:24.034479 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d27c9f8c-0884-467d-ae62-9193d6950191","Type":"ContainerStarted","Data":"a46ae0c3beca66e600238a2bd178354d061b72d7fb5267ecdfb2d685cea43af0"} Jul 07 19:02:26.045352 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:26.045321 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d27c9f8c-0884-467d-ae62-9193d6950191","Type":"ContainerStarted","Data":"642e2e3f887dc2826bc7aa92e29de4499b4e1e873cea8fbec1f1e28f2f0e2091"} Jul 07 19:02:26.045857 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:26.045397 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d27c9f8c-0884-467d-ae62-9193d6950191","Type":"ContainerStarted","Data":"4ca654707d5d4fcc607ae59f52a684e43360f3946743a9e402e1261500880ac7"} Jul 07 19:02:26.045857 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:26.045442 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d27c9f8c-0884-467d-ae62-9193d6950191","Type":"ContainerStarted","Data":"1472dd326ad564274c5e93960ed0abb12f890ff81d7d8a15c278498000684c10"} Jul 07 19:02:26.045857 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:26.045482 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d27c9f8c-0884-467d-ae62-9193d6950191","Type":"ContainerStarted","Data":"146645ac04a6265cbafa9a51132ccfed6688ca9d523c3adf9a759a145167a3b7"} Jul 07 19:02:26.079064 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:26.078995 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=1.955421989 podStartE2EDuration="8.078976138s" totalImagesPullingTime="6.097832034s" totalInitContainerRuntime="25.722115ms" isStatefulPod=false podCreationTimestamp="2026-07-07 19:02:18 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:02:26.077608813 +0000 UTC m=+129.093613018" watchObservedRunningTime="2026-07-07 19:02:26.078976138 +0000 UTC m=+129.094980319" Jul 07 19:02:26.257845 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:26.257803 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/396810c7-8961-4755-b0f6-02a9d7e02ae3-metrics-certs\") pod \"network-metrics-daemon-h8m9b\" (UID: \"396810c7-8961-4755-b0f6-02a9d7e02ae3\") " pod="openshift-multus/network-metrics-daemon-h8m9b" Jul 07 19:02:26.260152 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:26.260130 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/396810c7-8961-4755-b0f6-02a9d7e02ae3-metrics-certs\") pod \"network-metrics-daemon-h8m9b\" (UID: \"396810c7-8961-4755-b0f6-02a9d7e02ae3\") " pod="openshift-multus/network-metrics-daemon-h8m9b" Jul 07 19:02:26.495034 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:26.494919 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-84rbs\"" Jul 07 19:02:26.502874 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:26.502848 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h8m9b" Jul 07 19:02:26.640438 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:26.640395 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-h8m9b"] Jul 07 19:02:26.643904 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:02:26.643875 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod396810c7_8961_4755_b0f6_02a9d7e02ae3.slice/crio-8d1986e22a67f96ac23ee928ae4d530dc919a70abf88f82cc3a814327f5ca06b WatchSource:0}: Error finding container 8d1986e22a67f96ac23ee928ae4d530dc919a70abf88f82cc3a814327f5ca06b: Status 404 returned error can't find the container with id 8d1986e22a67f96ac23ee928ae4d530dc919a70abf88f82cc3a814327f5ca06b Jul 07 19:02:27.051463 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:27.051424 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-h8m9b" event={"ID":"396810c7-8961-4755-b0f6-02a9d7e02ae3","Type":"ContainerStarted","Data":"8d1986e22a67f96ac23ee928ae4d530dc919a70abf88f82cc3a814327f5ca06b"} Jul 07 19:02:27.484983 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:27.484870 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:02:27.561833 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:27.561801 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:02:27.862019 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:27.861990 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-5d5c5c78bb-kzvl6" Jul 07 19:02:28.056758 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:28.056722 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-h8m9b" event={"ID":"396810c7-8961-4755-b0f6-02a9d7e02ae3","Type":"ContainerStarted","Data":"e0f26ba2ce3b0768898e2849d66c2980a34848c111954e60bb3b5dcbf3e1e791"} Jul 07 19:02:28.057209 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:28.056774 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-h8m9b" event={"ID":"396810c7-8961-4755-b0f6-02a9d7e02ae3","Type":"ContainerStarted","Data":"b6e1f5fa3fe065eb4fb8ed5a05b995c3b13c1f8495fb71ca68cef8fa09dc84da"} Jul 07 19:02:28.077900 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:28.077843 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-h8m9b" podStartSLOduration=130.087544319 podStartE2EDuration="2m11.077824768s" totalImagesPullingTime="990.280449ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:00:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:02:28.07673713 +0000 UTC m=+131.092741313" watchObservedRunningTime="2026-07-07 19:02:28.077824768 +0000 UTC m=+131.093828948" Jul 07 19:02:28.210279 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:28.210247 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:02:28.337607 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:28.337514 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:02:28.517300 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:28.517267 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:02:29.228808 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:29.228776 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:02:29.307705 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:29.307674 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:02:31.449512 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:31.449468 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:02:31.553594 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:31.553556 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:02:32.016132 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:32.016094 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:02:32.112115 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:32.112072 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:02:35.396522 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:35.396482 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:02:35.494507 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:35.494470 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:02:36.519445 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:36.519413 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-5cb97f8f7b-stvw8" Jul 07 19:02:36.519813 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:36.519469 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-5cb97f8f7b-stvw8" Jul 07 19:02:36.606924 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:36.606890 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:02:36.706936 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:36.706893 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:02:39.593969 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:39.593916 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:02:39.667404 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:39.667370 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:02:41.023154 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:41.023117 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:02:41.122502 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:41.122463 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:02:41.224153 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:41.224120 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:02:42.885875 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:42.885833 2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-image-registry/image-registry-5d5c5c78bb-kzvl6" podUID="1e7deba9-69b3-4260-b6d0-037f274fb311" containerName="registry" containerID="cri-o://b3ac1a1f7e51d666caae9cbc9ad675059e92224b8c30da56be03b2ee983d4f80" gracePeriod=30 Jul 07 19:02:43.122947 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:43.122912 2563 generic.go:356] "Generic (PLEG): container finished" podID="1e7deba9-69b3-4260-b6d0-037f274fb311" containerID="b3ac1a1f7e51d666caae9cbc9ad675059e92224b8c30da56be03b2ee983d4f80" exitCode=0 Jul 07 19:02:43.123097 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:43.122988 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5d5c5c78bb-kzvl6" event={"ID":"1e7deba9-69b3-4260-b6d0-037f274fb311","Type":"ContainerDied","Data":"b3ac1a1f7e51d666caae9cbc9ad675059e92224b8c30da56be03b2ee983d4f80"} Jul 07 19:02:43.123097 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:43.123063 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5d5c5c78bb-kzvl6" event={"ID":"1e7deba9-69b3-4260-b6d0-037f274fb311","Type":"ContainerDied","Data":"866cb34a169948c20ea9120b465a4de13a8fd780b28975b1980fffb90a235880"} Jul 07 19:02:43.123097 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:43.123079 2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="866cb34a169948c20ea9120b465a4de13a8fd780b28975b1980fffb90a235880" Jul 07 19:02:43.128498 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:43.128477 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5d5c5c78bb-kzvl6" Jul 07 19:02:43.197014 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:43.196925 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:02:43.277119 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:43.277079 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:02:43.308494 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:43.308456 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1e7deba9-69b3-4260-b6d0-037f274fb311-bound-sa-token\") pod \"1e7deba9-69b3-4260-b6d0-037f274fb311\" (UID: \"1e7deba9-69b3-4260-b6d0-037f274fb311\") " Jul 07 19:02:43.308674 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:43.308526 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1e7deba9-69b3-4260-b6d0-037f274fb311-registry-tls\") pod \"1e7deba9-69b3-4260-b6d0-037f274fb311\" (UID: \"1e7deba9-69b3-4260-b6d0-037f274fb311\") " Jul 07 19:02:43.308743 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:43.308670 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1e7deba9-69b3-4260-b6d0-037f274fb311-installation-pull-secrets\") pod \"1e7deba9-69b3-4260-b6d0-037f274fb311\" (UID: \"1e7deba9-69b3-4260-b6d0-037f274fb311\") " Jul 07 19:02:43.308800 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:43.308746 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1e7deba9-69b3-4260-b6d0-037f274fb311-ca-trust-extracted\") pod \"1e7deba9-69b3-4260-b6d0-037f274fb311\" (UID: \"1e7deba9-69b3-4260-b6d0-037f274fb311\") " Jul 07 19:02:43.308852 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:43.308797 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1e7deba9-69b3-4260-b6d0-037f274fb311-image-registry-private-configuration\") pod \"1e7deba9-69b3-4260-b6d0-037f274fb311\" (UID: \"1e7deba9-69b3-4260-b6d0-037f274fb311\") " Jul 07 19:02:43.308852 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:43.308829 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1e7deba9-69b3-4260-b6d0-037f274fb311-registry-certificates\") pod \"1e7deba9-69b3-4260-b6d0-037f274fb311\" (UID: \"1e7deba9-69b3-4260-b6d0-037f274fb311\") " Jul 07 19:02:43.308941 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:43.308858 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1e7deba9-69b3-4260-b6d0-037f274fb311-trusted-ca\") pod \"1e7deba9-69b3-4260-b6d0-037f274fb311\" (UID: \"1e7deba9-69b3-4260-b6d0-037f274fb311\") " Jul 07 19:02:43.308941 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:43.308887 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lk5z4\" (UniqueName: \"kubernetes.io/projected/1e7deba9-69b3-4260-b6d0-037f274fb311-kube-api-access-lk5z4\") pod \"1e7deba9-69b3-4260-b6d0-037f274fb311\" (UID: \"1e7deba9-69b3-4260-b6d0-037f274fb311\") " Jul 07 19:02:43.309370 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:43.309279 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e7deba9-69b3-4260-b6d0-037f274fb311-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "1e7deba9-69b3-4260-b6d0-037f274fb311" (UID: "1e7deba9-69b3-4260-b6d0-037f274fb311"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 19:02:43.309507 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:43.309360 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e7deba9-69b3-4260-b6d0-037f274fb311-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "1e7deba9-69b3-4260-b6d0-037f274fb311" (UID: "1e7deba9-69b3-4260-b6d0-037f274fb311"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 19:02:43.311208 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:43.311152 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e7deba9-69b3-4260-b6d0-037f274fb311-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "1e7deba9-69b3-4260-b6d0-037f274fb311" (UID: "1e7deba9-69b3-4260-b6d0-037f274fb311"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 19:02:43.311322 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:43.311286 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e7deba9-69b3-4260-b6d0-037f274fb311-kube-api-access-lk5z4" (OuterVolumeSpecName: "kube-api-access-lk5z4") pod "1e7deba9-69b3-4260-b6d0-037f274fb311" (UID: "1e7deba9-69b3-4260-b6d0-037f274fb311"). InnerVolumeSpecName "kube-api-access-lk5z4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 19:02:43.311456 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:43.311417 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e7deba9-69b3-4260-b6d0-037f274fb311-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "1e7deba9-69b3-4260-b6d0-037f274fb311" (UID: "1e7deba9-69b3-4260-b6d0-037f274fb311"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 19:02:43.311526 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:43.311437 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e7deba9-69b3-4260-b6d0-037f274fb311-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "1e7deba9-69b3-4260-b6d0-037f274fb311" (UID: "1e7deba9-69b3-4260-b6d0-037f274fb311"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 19:02:43.311526 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:43.311505 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e7deba9-69b3-4260-b6d0-037f274fb311-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "1e7deba9-69b3-4260-b6d0-037f274fb311" (UID: "1e7deba9-69b3-4260-b6d0-037f274fb311"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 19:02:43.317760 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:43.317727 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e7deba9-69b3-4260-b6d0-037f274fb311-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "1e7deba9-69b3-4260-b6d0-037f274fb311" (UID: "1e7deba9-69b3-4260-b6d0-037f274fb311"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:02:43.410386 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:43.410333 2563 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1e7deba9-69b3-4260-b6d0-037f274fb311-bound-sa-token\") on node \"ip-10-0-142-134.ec2.internal\" DevicePath \"\"" Jul 07 19:02:43.410386 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:43.410381 2563 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1e7deba9-69b3-4260-b6d0-037f274fb311-registry-tls\") on node \"ip-10-0-142-134.ec2.internal\" DevicePath \"\"" Jul 07 19:02:43.410386 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:43.410394 2563 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1e7deba9-69b3-4260-b6d0-037f274fb311-installation-pull-secrets\") on node \"ip-10-0-142-134.ec2.internal\" DevicePath \"\"" Jul 07 19:02:43.410386 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:43.410404 2563 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1e7deba9-69b3-4260-b6d0-037f274fb311-ca-trust-extracted\") on node \"ip-10-0-142-134.ec2.internal\" DevicePath \"\"" Jul 07 19:02:43.410677 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:43.410416 2563 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1e7deba9-69b3-4260-b6d0-037f274fb311-image-registry-private-configuration\") on node \"ip-10-0-142-134.ec2.internal\" DevicePath \"\"" Jul 07 19:02:43.410677 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:43.410426 2563 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1e7deba9-69b3-4260-b6d0-037f274fb311-registry-certificates\") on node \"ip-10-0-142-134.ec2.internal\" DevicePath \"\"" Jul 07 19:02:43.410677 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:43.410434 2563 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1e7deba9-69b3-4260-b6d0-037f274fb311-trusted-ca\") on node \"ip-10-0-142-134.ec2.internal\" DevicePath \"\"" Jul 07 19:02:43.410677 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:43.410443 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lk5z4\" (UniqueName: \"kubernetes.io/projected/1e7deba9-69b3-4260-b6d0-037f274fb311-kube-api-access-lk5z4\") on node \"ip-10-0-142-134.ec2.internal\" DevicePath \"\"" Jul 07 19:02:44.126883 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:44.126852 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5d5c5c78bb-kzvl6" Jul 07 19:02:44.144852 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:44.144823 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5d5c5c78bb-kzvl6"] Jul 07 19:02:44.155506 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:44.155478 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-5d5c5c78bb-kzvl6"] Jul 07 19:02:45.475669 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:45.475634 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1e7deba9-69b3-4260-b6d0-037f274fb311" path="/var/lib/kubelet/pods/1e7deba9-69b3-4260-b6d0-037f274fb311/volumes" Jul 07 19:02:46.066229 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:46.066189 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:02:46.163343 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:46.163309 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:02:47.299263 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:47.299218 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:02:47.384511 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:47.384473 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:02:48.627778 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:48.627742 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:02:48.734628 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:48.734597 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:02:52.210664 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:52.210622 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:02:52.307742 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:52.307704 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:02:56.525040 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:56.525013 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-5cb97f8f7b-stvw8" Jul 07 19:02:56.528920 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:02:56.528896 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-5cb97f8f7b-stvw8" Jul 07 19:03:01.986516 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:01.986480 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:03:02.065255 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:02.065217 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:03:18.517046 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:18.516933 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:03:18.535889 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:18.535864 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:03:19.278372 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:19.278345 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:03:23.092848 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:23.092809 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:03:23.159260 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:23.159230 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:03:31.444907 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:31.444862 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:03:31.713844 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:31.713750 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:03:32.439302 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:32.439263 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 07 19:03:32.439576 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:32.439553 2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="f8c425a2-1747-4fc6-b0d2-4a2cb46e4929" containerName="alertmanager" containerID="cri-o://2cd3e07c61d50b29520bef559caefb7996da77fe3596311da99a69b23498ade3" gracePeriod=120 Jul 07 19:03:32.439649 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:32.439619 2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="f8c425a2-1747-4fc6-b0d2-4a2cb46e4929" containerName="kube-rbac-proxy-web" containerID="cri-o://d5641498974d7e0650186c13b061224b4f64d4ad8f25190d923db39c589b93cb" gracePeriod=120 Jul 07 19:03:32.439714 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:32.439658 2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="f8c425a2-1747-4fc6-b0d2-4a2cb46e4929" containerName="kube-rbac-proxy" containerID="cri-o://59dd09e938e74c2c34a783745790e1f83739d669d7f596c5e4907d0167f36f68" gracePeriod=120 Jul 07 19:03:32.439714 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:32.439679 2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="f8c425a2-1747-4fc6-b0d2-4a2cb46e4929" containerName="prom-label-proxy" containerID="cri-o://b6560799533213a93b1e36e5b7eca6971f3c4d26e2fa29293801b752e334d35f" gracePeriod=120 Jul 07 19:03:32.439827 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:32.439619 2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="f8c425a2-1747-4fc6-b0d2-4a2cb46e4929" containerName="kube-rbac-proxy-metric" containerID="cri-o://1a71b5f2b48d7c2e2686fd9b852a6bbe82975eefc3a3545b7e9fbff5b1ffb13d" gracePeriod=120 Jul 07 19:03:32.439827 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:32.439715 2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="f8c425a2-1747-4fc6-b0d2-4a2cb46e4929" containerName="config-reloader" containerID="cri-o://c12957fe91d96187144b551bc675d7a01f04657b8fa6e572f08437fd88c37abe" gracePeriod=120 Jul 07 19:03:33.323104 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:33.323070 2563 generic.go:356] "Generic (PLEG): container finished" podID="f8c425a2-1747-4fc6-b0d2-4a2cb46e4929" containerID="b6560799533213a93b1e36e5b7eca6971f3c4d26e2fa29293801b752e334d35f" exitCode=0 Jul 07 19:03:33.323104 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:33.323096 2563 generic.go:356] "Generic (PLEG): container finished" podID="f8c425a2-1747-4fc6-b0d2-4a2cb46e4929" containerID="59dd09e938e74c2c34a783745790e1f83739d669d7f596c5e4907d0167f36f68" exitCode=0 Jul 07 19:03:33.323104 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:33.323104 2563 generic.go:356] "Generic (PLEG): container finished" podID="f8c425a2-1747-4fc6-b0d2-4a2cb46e4929" containerID="c12957fe91d96187144b551bc675d7a01f04657b8fa6e572f08437fd88c37abe" exitCode=0 Jul 07 19:03:33.323104 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:33.323109 2563 generic.go:356] "Generic (PLEG): container finished" podID="f8c425a2-1747-4fc6-b0d2-4a2cb46e4929" containerID="2cd3e07c61d50b29520bef559caefb7996da77fe3596311da99a69b23498ade3" exitCode=0 Jul 07 19:03:33.323566 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:33.323144 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f8c425a2-1747-4fc6-b0d2-4a2cb46e4929","Type":"ContainerDied","Data":"b6560799533213a93b1e36e5b7eca6971f3c4d26e2fa29293801b752e334d35f"} Jul 07 19:03:33.323566 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:33.323232 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f8c425a2-1747-4fc6-b0d2-4a2cb46e4929","Type":"ContainerDied","Data":"59dd09e938e74c2c34a783745790e1f83739d669d7f596c5e4907d0167f36f68"} Jul 07 19:03:33.323566 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:33.323258 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f8c425a2-1747-4fc6-b0d2-4a2cb46e4929","Type":"ContainerDied","Data":"c12957fe91d96187144b551bc675d7a01f04657b8fa6e572f08437fd88c37abe"} Jul 07 19:03:33.323566 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:33.323282 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f8c425a2-1747-4fc6-b0d2-4a2cb46e4929","Type":"ContainerDied","Data":"2cd3e07c61d50b29520bef559caefb7996da77fe3596311da99a69b23498ade3"} Jul 07 19:03:33.691081 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:33.691056 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 07 19:03:33.846766 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:33.846726 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f8c425a2-1747-4fc6-b0d2-4a2cb46e4929-cluster-tls-config\") pod \"f8c425a2-1747-4fc6-b0d2-4a2cb46e4929\" (UID: \"f8c425a2-1747-4fc6-b0d2-4a2cb46e4929\") " Jul 07 19:03:33.846766 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:33.846773 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f8c425a2-1747-4fc6-b0d2-4a2cb46e4929-config-out\") pod \"f8c425a2-1747-4fc6-b0d2-4a2cb46e4929\" (UID: \"f8c425a2-1747-4fc6-b0d2-4a2cb46e4929\") " Jul 07 19:03:33.847062 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:33.846798 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f8c425a2-1747-4fc6-b0d2-4a2cb46e4929-secret-alertmanager-kube-rbac-proxy\") pod \"f8c425a2-1747-4fc6-b0d2-4a2cb46e4929\" (UID: \"f8c425a2-1747-4fc6-b0d2-4a2cb46e4929\") " Jul 07 19:03:33.847062 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:33.846827 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/f8c425a2-1747-4fc6-b0d2-4a2cb46e4929-alertmanager-main-db\") pod \"f8c425a2-1747-4fc6-b0d2-4a2cb46e4929\" (UID: \"f8c425a2-1747-4fc6-b0d2-4a2cb46e4929\") " Jul 07 19:03:33.847062 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:33.846858 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/f8c425a2-1747-4fc6-b0d2-4a2cb46e4929-config-volume\") pod \"f8c425a2-1747-4fc6-b0d2-4a2cb46e4929\" (UID: \"f8c425a2-1747-4fc6-b0d2-4a2cb46e4929\") " Jul 07 19:03:33.847062 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:33.846885 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f8c425a2-1747-4fc6-b0d2-4a2cb46e4929-tls-assets\") pod \"f8c425a2-1747-4fc6-b0d2-4a2cb46e4929\" (UID: \"f8c425a2-1747-4fc6-b0d2-4a2cb46e4929\") " Jul 07 19:03:33.847062 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:33.846932 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rcgzc\" (UniqueName: \"kubernetes.io/projected/f8c425a2-1747-4fc6-b0d2-4a2cb46e4929-kube-api-access-rcgzc\") pod \"f8c425a2-1747-4fc6-b0d2-4a2cb46e4929\" (UID: \"f8c425a2-1747-4fc6-b0d2-4a2cb46e4929\") " Jul 07 19:03:33.847319 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:33.847129 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/f8c425a2-1747-4fc6-b0d2-4a2cb46e4929-secret-alertmanager-kube-rbac-proxy-metric\") pod \"f8c425a2-1747-4fc6-b0d2-4a2cb46e4929\" (UID: \"f8c425a2-1747-4fc6-b0d2-4a2cb46e4929\") " Jul 07 19:03:33.847319 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:33.847174 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f8c425a2-1747-4fc6-b0d2-4a2cb46e4929-web-config\") pod \"f8c425a2-1747-4fc6-b0d2-4a2cb46e4929\" (UID: \"f8c425a2-1747-4fc6-b0d2-4a2cb46e4929\") " Jul 07 19:03:33.847319 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:33.847212 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f8c425a2-1747-4fc6-b0d2-4a2cb46e4929-metrics-client-ca\") pod \"f8c425a2-1747-4fc6-b0d2-4a2cb46e4929\" (UID: \"f8c425a2-1747-4fc6-b0d2-4a2cb46e4929\") " Jul 07 19:03:33.847319 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:33.847229 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8c425a2-1747-4fc6-b0d2-4a2cb46e4929-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "f8c425a2-1747-4fc6-b0d2-4a2cb46e4929" (UID: "f8c425a2-1747-4fc6-b0d2-4a2cb46e4929"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:03:33.847319 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:33.847247 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f8c425a2-1747-4fc6-b0d2-4a2cb46e4929-secret-alertmanager-kube-rbac-proxy-web\") pod \"f8c425a2-1747-4fc6-b0d2-4a2cb46e4929\" (UID: \"f8c425a2-1747-4fc6-b0d2-4a2cb46e4929\") " Jul 07 19:03:33.847319 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:33.847307 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f8c425a2-1747-4fc6-b0d2-4a2cb46e4929-alertmanager-trusted-ca-bundle\") pod \"f8c425a2-1747-4fc6-b0d2-4a2cb46e4929\" (UID: \"f8c425a2-1747-4fc6-b0d2-4a2cb46e4929\") " Jul 07 19:03:33.847611 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:33.847354 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f8c425a2-1747-4fc6-b0d2-4a2cb46e4929-secret-alertmanager-main-tls\") pod \"f8c425a2-1747-4fc6-b0d2-4a2cb46e4929\" (UID: \"f8c425a2-1747-4fc6-b0d2-4a2cb46e4929\") " Jul 07 19:03:33.847667 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:33.847635 2563 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/f8c425a2-1747-4fc6-b0d2-4a2cb46e4929-alertmanager-main-db\") on node \"ip-10-0-142-134.ec2.internal\" DevicePath \"\"" Jul 07 19:03:33.849915 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:33.849879 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8c425a2-1747-4fc6-b0d2-4a2cb46e4929-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "f8c425a2-1747-4fc6-b0d2-4a2cb46e4929" (UID: "f8c425a2-1747-4fc6-b0d2-4a2cb46e4929"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 19:03:33.850065 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:33.849886 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8c425a2-1747-4fc6-b0d2-4a2cb46e4929-kube-api-access-rcgzc" (OuterVolumeSpecName: "kube-api-access-rcgzc") pod "f8c425a2-1747-4fc6-b0d2-4a2cb46e4929" (UID: "f8c425a2-1747-4fc6-b0d2-4a2cb46e4929"). InnerVolumeSpecName "kube-api-access-rcgzc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 19:03:33.850065 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:33.849896 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8c425a2-1747-4fc6-b0d2-4a2cb46e4929-config-out" (OuterVolumeSpecName: "config-out") pod "f8c425a2-1747-4fc6-b0d2-4a2cb46e4929" (UID: "f8c425a2-1747-4fc6-b0d2-4a2cb46e4929"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:03:33.850065 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:33.849968 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8c425a2-1747-4fc6-b0d2-4a2cb46e4929-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "f8c425a2-1747-4fc6-b0d2-4a2cb46e4929" (UID: "f8c425a2-1747-4fc6-b0d2-4a2cb46e4929"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 19:03:33.850325 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:33.850296 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8c425a2-1747-4fc6-b0d2-4a2cb46e4929-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "f8c425a2-1747-4fc6-b0d2-4a2cb46e4929" (UID: "f8c425a2-1747-4fc6-b0d2-4a2cb46e4929"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 19:03:33.850448 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:33.850321 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8c425a2-1747-4fc6-b0d2-4a2cb46e4929-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "f8c425a2-1747-4fc6-b0d2-4a2cb46e4929" (UID: "f8c425a2-1747-4fc6-b0d2-4a2cb46e4929"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 19:03:33.850880 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:33.850855 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8c425a2-1747-4fc6-b0d2-4a2cb46e4929-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "f8c425a2-1747-4fc6-b0d2-4a2cb46e4929" (UID: "f8c425a2-1747-4fc6-b0d2-4a2cb46e4929"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 19:03:33.850999 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:33.850894 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8c425a2-1747-4fc6-b0d2-4a2cb46e4929-config-volume" (OuterVolumeSpecName: "config-volume") pod "f8c425a2-1747-4fc6-b0d2-4a2cb46e4929" (UID: "f8c425a2-1747-4fc6-b0d2-4a2cb46e4929"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 19:03:33.850999 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:33.850901 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8c425a2-1747-4fc6-b0d2-4a2cb46e4929-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "f8c425a2-1747-4fc6-b0d2-4a2cb46e4929" (UID: "f8c425a2-1747-4fc6-b0d2-4a2cb46e4929"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 19:03:33.852218 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:33.852191 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8c425a2-1747-4fc6-b0d2-4a2cb46e4929-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "f8c425a2-1747-4fc6-b0d2-4a2cb46e4929" (UID: "f8c425a2-1747-4fc6-b0d2-4a2cb46e4929"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 19:03:33.853474 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:33.853455 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8c425a2-1747-4fc6-b0d2-4a2cb46e4929-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "f8c425a2-1747-4fc6-b0d2-4a2cb46e4929" (UID: "f8c425a2-1747-4fc6-b0d2-4a2cb46e4929"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 19:03:33.862342 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:33.862277 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8c425a2-1747-4fc6-b0d2-4a2cb46e4929-web-config" (OuterVolumeSpecName: "web-config") pod "f8c425a2-1747-4fc6-b0d2-4a2cb46e4929" (UID: "f8c425a2-1747-4fc6-b0d2-4a2cb46e4929"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 19:03:33.948934 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:33.948894 2563 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/f8c425a2-1747-4fc6-b0d2-4a2cb46e4929-config-volume\") on node \"ip-10-0-142-134.ec2.internal\" DevicePath \"\"" Jul 07 19:03:33.948934 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:33.948926 2563 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f8c425a2-1747-4fc6-b0d2-4a2cb46e4929-tls-assets\") on node \"ip-10-0-142-134.ec2.internal\" DevicePath \"\"" Jul 07 19:03:33.948934 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:33.948935 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rcgzc\" (UniqueName: \"kubernetes.io/projected/f8c425a2-1747-4fc6-b0d2-4a2cb46e4929-kube-api-access-rcgzc\") on node \"ip-10-0-142-134.ec2.internal\" DevicePath \"\"" Jul 07 19:03:33.949170 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:33.948947 2563 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/f8c425a2-1747-4fc6-b0d2-4a2cb46e4929-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-142-134.ec2.internal\" DevicePath \"\"" Jul 07 19:03:33.949170 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:33.948983 2563 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f8c425a2-1747-4fc6-b0d2-4a2cb46e4929-web-config\") on node \"ip-10-0-142-134.ec2.internal\" DevicePath \"\"" Jul 07 19:03:33.949170 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:33.948992 2563 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f8c425a2-1747-4fc6-b0d2-4a2cb46e4929-metrics-client-ca\") on node \"ip-10-0-142-134.ec2.internal\" DevicePath \"\"" Jul 07 19:03:33.949170 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:33.949001 2563 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f8c425a2-1747-4fc6-b0d2-4a2cb46e4929-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-142-134.ec2.internal\" DevicePath \"\"" Jul 07 19:03:33.949170 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:33.949010 2563 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f8c425a2-1747-4fc6-b0d2-4a2cb46e4929-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-142-134.ec2.internal\" DevicePath \"\"" Jul 07 19:03:33.949170 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:33.949020 2563 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f8c425a2-1747-4fc6-b0d2-4a2cb46e4929-secret-alertmanager-main-tls\") on node \"ip-10-0-142-134.ec2.internal\" DevicePath \"\"" Jul 07 19:03:33.949170 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:33.949028 2563 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f8c425a2-1747-4fc6-b0d2-4a2cb46e4929-cluster-tls-config\") on node \"ip-10-0-142-134.ec2.internal\" DevicePath \"\"" Jul 07 19:03:33.949170 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:33.949037 2563 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f8c425a2-1747-4fc6-b0d2-4a2cb46e4929-config-out\") on node \"ip-10-0-142-134.ec2.internal\" DevicePath \"\"" Jul 07 19:03:33.949170 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:33.949046 2563 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f8c425a2-1747-4fc6-b0d2-4a2cb46e4929-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-142-134.ec2.internal\" DevicePath \"\"" Jul 07 19:03:34.329898 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:34.329862 2563 generic.go:356] "Generic (PLEG): container finished" podID="f8c425a2-1747-4fc6-b0d2-4a2cb46e4929" containerID="1a71b5f2b48d7c2e2686fd9b852a6bbe82975eefc3a3545b7e9fbff5b1ffb13d" exitCode=0 Jul 07 19:03:34.329898 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:34.329887 2563 generic.go:356] "Generic (PLEG): container finished" podID="f8c425a2-1747-4fc6-b0d2-4a2cb46e4929" containerID="d5641498974d7e0650186c13b061224b4f64d4ad8f25190d923db39c589b93cb" exitCode=0 Jul 07 19:03:34.330379 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:34.329942 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 07 19:03:34.330379 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:34.329944 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f8c425a2-1747-4fc6-b0d2-4a2cb46e4929","Type":"ContainerDied","Data":"1a71b5f2b48d7c2e2686fd9b852a6bbe82975eefc3a3545b7e9fbff5b1ffb13d"} Jul 07 19:03:34.330379 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:34.330035 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f8c425a2-1747-4fc6-b0d2-4a2cb46e4929","Type":"ContainerDied","Data":"d5641498974d7e0650186c13b061224b4f64d4ad8f25190d923db39c589b93cb"} Jul 07 19:03:34.330379 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:34.330064 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f8c425a2-1747-4fc6-b0d2-4a2cb46e4929","Type":"ContainerDied","Data":"850930ea7d56f90ff35cf701b269be546b063422b98993489d9b1393dfc28476"} Jul 07 19:03:34.330379 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:34.330098 2563 scope.go:117] "RemoveContainer" containerID="b6560799533213a93b1e36e5b7eca6971f3c4d26e2fa29293801b752e334d35f" Jul 07 19:03:34.338205 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:34.338184 2563 scope.go:117] "RemoveContainer" containerID="1a71b5f2b48d7c2e2686fd9b852a6bbe82975eefc3a3545b7e9fbff5b1ffb13d" Jul 07 19:03:34.345765 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:34.345744 2563 scope.go:117] "RemoveContainer" containerID="59dd09e938e74c2c34a783745790e1f83739d669d7f596c5e4907d0167f36f68" Jul 07 19:03:34.352596 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:34.352573 2563 scope.go:117] "RemoveContainer" containerID="d5641498974d7e0650186c13b061224b4f64d4ad8f25190d923db39c589b93cb" Jul 07 19:03:34.359484 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:34.359455 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 07 19:03:34.360315 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:34.360294 2563 scope.go:117] "RemoveContainer" containerID="c12957fe91d96187144b551bc675d7a01f04657b8fa6e572f08437fd88c37abe" Jul 07 19:03:34.368043 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:34.368023 2563 scope.go:117] "RemoveContainer" containerID="2cd3e07c61d50b29520bef559caefb7996da77fe3596311da99a69b23498ade3" Jul 07 19:03:34.370174 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:34.370151 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 07 19:03:34.375765 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:34.375748 2563 scope.go:117] "RemoveContainer" containerID="bb7ea7b8f5029487b127bd8eeb1aeefb0c2fbccf711e1fae132d0310e6581328" Jul 07 19:03:34.382976 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:34.382941 2563 scope.go:117] "RemoveContainer" containerID="b6560799533213a93b1e36e5b7eca6971f3c4d26e2fa29293801b752e334d35f" Jul 07 19:03:34.383265 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:03:34.383244 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6560799533213a93b1e36e5b7eca6971f3c4d26e2fa29293801b752e334d35f\": container with ID starting with b6560799533213a93b1e36e5b7eca6971f3c4d26e2fa29293801b752e334d35f not found: ID does not exist" containerID="b6560799533213a93b1e36e5b7eca6971f3c4d26e2fa29293801b752e334d35f" Jul 07 19:03:34.383320 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:34.383274 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6560799533213a93b1e36e5b7eca6971f3c4d26e2fa29293801b752e334d35f"} err="failed to get container status \"b6560799533213a93b1e36e5b7eca6971f3c4d26e2fa29293801b752e334d35f\": rpc error: code = NotFound desc = could not find container \"b6560799533213a93b1e36e5b7eca6971f3c4d26e2fa29293801b752e334d35f\": container with ID starting with b6560799533213a93b1e36e5b7eca6971f3c4d26e2fa29293801b752e334d35f not found: ID does not exist" Jul 07 19:03:34.383320 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:34.383309 2563 scope.go:117] "RemoveContainer" containerID="1a71b5f2b48d7c2e2686fd9b852a6bbe82975eefc3a3545b7e9fbff5b1ffb13d" Jul 07 19:03:34.383579 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:03:34.383562 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a71b5f2b48d7c2e2686fd9b852a6bbe82975eefc3a3545b7e9fbff5b1ffb13d\": container with ID starting with 1a71b5f2b48d7c2e2686fd9b852a6bbe82975eefc3a3545b7e9fbff5b1ffb13d not found: ID does not exist" containerID="1a71b5f2b48d7c2e2686fd9b852a6bbe82975eefc3a3545b7e9fbff5b1ffb13d" Jul 07 19:03:34.383640 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:34.383590 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a71b5f2b48d7c2e2686fd9b852a6bbe82975eefc3a3545b7e9fbff5b1ffb13d"} err="failed to get container status \"1a71b5f2b48d7c2e2686fd9b852a6bbe82975eefc3a3545b7e9fbff5b1ffb13d\": rpc error: code = NotFound desc = could not find container \"1a71b5f2b48d7c2e2686fd9b852a6bbe82975eefc3a3545b7e9fbff5b1ffb13d\": container with ID starting with 1a71b5f2b48d7c2e2686fd9b852a6bbe82975eefc3a3545b7e9fbff5b1ffb13d not found: ID does not exist" Jul 07 19:03:34.383640 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:34.383615 2563 scope.go:117] "RemoveContainer" containerID="59dd09e938e74c2c34a783745790e1f83739d669d7f596c5e4907d0167f36f68" Jul 07 19:03:34.383850 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:03:34.383834 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"59dd09e938e74c2c34a783745790e1f83739d669d7f596c5e4907d0167f36f68\": container with ID starting with 59dd09e938e74c2c34a783745790e1f83739d669d7f596c5e4907d0167f36f68 not found: ID does not exist" containerID="59dd09e938e74c2c34a783745790e1f83739d669d7f596c5e4907d0167f36f68" Jul 07 19:03:34.383887 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:34.383856 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59dd09e938e74c2c34a783745790e1f83739d669d7f596c5e4907d0167f36f68"} err="failed to get container status \"59dd09e938e74c2c34a783745790e1f83739d669d7f596c5e4907d0167f36f68\": rpc error: code = NotFound desc = could not find container \"59dd09e938e74c2c34a783745790e1f83739d669d7f596c5e4907d0167f36f68\": container with ID starting with 59dd09e938e74c2c34a783745790e1f83739d669d7f596c5e4907d0167f36f68 not found: ID does not exist" Jul 07 19:03:34.383887 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:34.383870 2563 scope.go:117] "RemoveContainer" containerID="d5641498974d7e0650186c13b061224b4f64d4ad8f25190d923db39c589b93cb" Jul 07 19:03:34.384104 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:03:34.384091 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5641498974d7e0650186c13b061224b4f64d4ad8f25190d923db39c589b93cb\": container with ID starting with d5641498974d7e0650186c13b061224b4f64d4ad8f25190d923db39c589b93cb not found: ID does not exist" containerID="d5641498974d7e0650186c13b061224b4f64d4ad8f25190d923db39c589b93cb" Jul 07 19:03:34.384142 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:34.384108 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5641498974d7e0650186c13b061224b4f64d4ad8f25190d923db39c589b93cb"} err="failed to get container status \"d5641498974d7e0650186c13b061224b4f64d4ad8f25190d923db39c589b93cb\": rpc error: code = NotFound desc = could not find container \"d5641498974d7e0650186c13b061224b4f64d4ad8f25190d923db39c589b93cb\": container with ID starting with d5641498974d7e0650186c13b061224b4f64d4ad8f25190d923db39c589b93cb not found: ID does not exist" Jul 07 19:03:34.384142 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:34.384128 2563 scope.go:117] "RemoveContainer" containerID="c12957fe91d96187144b551bc675d7a01f04657b8fa6e572f08437fd88c37abe" Jul 07 19:03:34.384348 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:03:34.384335 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c12957fe91d96187144b551bc675d7a01f04657b8fa6e572f08437fd88c37abe\": container with ID starting with c12957fe91d96187144b551bc675d7a01f04657b8fa6e572f08437fd88c37abe not found: ID does not exist" containerID="c12957fe91d96187144b551bc675d7a01f04657b8fa6e572f08437fd88c37abe" Jul 07 19:03:34.384388 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:34.384350 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c12957fe91d96187144b551bc675d7a01f04657b8fa6e572f08437fd88c37abe"} err="failed to get container status \"c12957fe91d96187144b551bc675d7a01f04657b8fa6e572f08437fd88c37abe\": rpc error: code = NotFound desc = could not find container \"c12957fe91d96187144b551bc675d7a01f04657b8fa6e572f08437fd88c37abe\": container with ID starting with c12957fe91d96187144b551bc675d7a01f04657b8fa6e572f08437fd88c37abe not found: ID does not exist" Jul 07 19:03:34.384388 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:34.384361 2563 scope.go:117] "RemoveContainer" containerID="2cd3e07c61d50b29520bef559caefb7996da77fe3596311da99a69b23498ade3" Jul 07 19:03:34.384573 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:03:34.384558 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2cd3e07c61d50b29520bef559caefb7996da77fe3596311da99a69b23498ade3\": container with ID starting with 2cd3e07c61d50b29520bef559caefb7996da77fe3596311da99a69b23498ade3 not found: ID does not exist" containerID="2cd3e07c61d50b29520bef559caefb7996da77fe3596311da99a69b23498ade3" Jul 07 19:03:34.384623 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:34.384574 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2cd3e07c61d50b29520bef559caefb7996da77fe3596311da99a69b23498ade3"} err="failed to get container status \"2cd3e07c61d50b29520bef559caefb7996da77fe3596311da99a69b23498ade3\": rpc error: code = NotFound desc = could not find container \"2cd3e07c61d50b29520bef559caefb7996da77fe3596311da99a69b23498ade3\": container with ID starting with 2cd3e07c61d50b29520bef559caefb7996da77fe3596311da99a69b23498ade3 not found: ID does not exist" Jul 07 19:03:34.384623 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:34.384585 2563 scope.go:117] "RemoveContainer" containerID="bb7ea7b8f5029487b127bd8eeb1aeefb0c2fbccf711e1fae132d0310e6581328" Jul 07 19:03:34.384754 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:03:34.384742 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb7ea7b8f5029487b127bd8eeb1aeefb0c2fbccf711e1fae132d0310e6581328\": container with ID starting with bb7ea7b8f5029487b127bd8eeb1aeefb0c2fbccf711e1fae132d0310e6581328 not found: ID does not exist" containerID="bb7ea7b8f5029487b127bd8eeb1aeefb0c2fbccf711e1fae132d0310e6581328" Jul 07 19:03:34.384797 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:34.384755 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb7ea7b8f5029487b127bd8eeb1aeefb0c2fbccf711e1fae132d0310e6581328"} err="failed to get container status \"bb7ea7b8f5029487b127bd8eeb1aeefb0c2fbccf711e1fae132d0310e6581328\": rpc error: code = NotFound desc = could not find container \"bb7ea7b8f5029487b127bd8eeb1aeefb0c2fbccf711e1fae132d0310e6581328\": container with ID starting with bb7ea7b8f5029487b127bd8eeb1aeefb0c2fbccf711e1fae132d0310e6581328 not found: ID does not exist" Jul 07 19:03:34.384797 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:34.384765 2563 scope.go:117] "RemoveContainer" containerID="b6560799533213a93b1e36e5b7eca6971f3c4d26e2fa29293801b752e334d35f" Jul 07 19:03:34.385002 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:34.384974 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6560799533213a93b1e36e5b7eca6971f3c4d26e2fa29293801b752e334d35f"} err="failed to get container status \"b6560799533213a93b1e36e5b7eca6971f3c4d26e2fa29293801b752e334d35f\": rpc error: code = NotFound desc = could not find container \"b6560799533213a93b1e36e5b7eca6971f3c4d26e2fa29293801b752e334d35f\": container with ID starting with b6560799533213a93b1e36e5b7eca6971f3c4d26e2fa29293801b752e334d35f not found: ID does not exist" Jul 07 19:03:34.385002 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:34.385001 2563 scope.go:117] "RemoveContainer" containerID="1a71b5f2b48d7c2e2686fd9b852a6bbe82975eefc3a3545b7e9fbff5b1ffb13d" Jul 07 19:03:34.385221 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:34.385202 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a71b5f2b48d7c2e2686fd9b852a6bbe82975eefc3a3545b7e9fbff5b1ffb13d"} err="failed to get container status \"1a71b5f2b48d7c2e2686fd9b852a6bbe82975eefc3a3545b7e9fbff5b1ffb13d\": rpc error: code = NotFound desc = could not find container \"1a71b5f2b48d7c2e2686fd9b852a6bbe82975eefc3a3545b7e9fbff5b1ffb13d\": container with ID starting with 1a71b5f2b48d7c2e2686fd9b852a6bbe82975eefc3a3545b7e9fbff5b1ffb13d not found: ID does not exist" Jul 07 19:03:34.385274 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:34.385222 2563 scope.go:117] "RemoveContainer" containerID="59dd09e938e74c2c34a783745790e1f83739d669d7f596c5e4907d0167f36f68" Jul 07 19:03:34.385433 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:34.385416 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59dd09e938e74c2c34a783745790e1f83739d669d7f596c5e4907d0167f36f68"} err="failed to get container status \"59dd09e938e74c2c34a783745790e1f83739d669d7f596c5e4907d0167f36f68\": rpc error: code = NotFound desc = could not find container \"59dd09e938e74c2c34a783745790e1f83739d669d7f596c5e4907d0167f36f68\": container with ID starting with 59dd09e938e74c2c34a783745790e1f83739d669d7f596c5e4907d0167f36f68 not found: ID does not exist" Jul 07 19:03:34.385477 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:34.385434 2563 scope.go:117] "RemoveContainer" containerID="d5641498974d7e0650186c13b061224b4f64d4ad8f25190d923db39c589b93cb" Jul 07 19:03:34.385623 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:34.385604 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5641498974d7e0650186c13b061224b4f64d4ad8f25190d923db39c589b93cb"} err="failed to get container status \"d5641498974d7e0650186c13b061224b4f64d4ad8f25190d923db39c589b93cb\": rpc error: code = NotFound desc = could not find container \"d5641498974d7e0650186c13b061224b4f64d4ad8f25190d923db39c589b93cb\": container with ID starting with d5641498974d7e0650186c13b061224b4f64d4ad8f25190d923db39c589b93cb not found: ID does not exist" Jul 07 19:03:34.385661 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:34.385627 2563 scope.go:117] "RemoveContainer" containerID="c12957fe91d96187144b551bc675d7a01f04657b8fa6e572f08437fd88c37abe" Jul 07 19:03:34.385803 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:34.385788 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c12957fe91d96187144b551bc675d7a01f04657b8fa6e572f08437fd88c37abe"} err="failed to get container status \"c12957fe91d96187144b551bc675d7a01f04657b8fa6e572f08437fd88c37abe\": rpc error: code = NotFound desc = could not find container \"c12957fe91d96187144b551bc675d7a01f04657b8fa6e572f08437fd88c37abe\": container with ID starting with c12957fe91d96187144b551bc675d7a01f04657b8fa6e572f08437fd88c37abe not found: ID does not exist" Jul 07 19:03:34.385842 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:34.385803 2563 scope.go:117] "RemoveContainer" containerID="2cd3e07c61d50b29520bef559caefb7996da77fe3596311da99a69b23498ade3" Jul 07 19:03:34.385989 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:34.385971 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2cd3e07c61d50b29520bef559caefb7996da77fe3596311da99a69b23498ade3"} err="failed to get container status \"2cd3e07c61d50b29520bef559caefb7996da77fe3596311da99a69b23498ade3\": rpc error: code = NotFound desc = could not find container \"2cd3e07c61d50b29520bef559caefb7996da77fe3596311da99a69b23498ade3\": container with ID starting with 2cd3e07c61d50b29520bef559caefb7996da77fe3596311da99a69b23498ade3 not found: ID does not exist" Jul 07 19:03:34.386047 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:34.385989 2563 scope.go:117] "RemoveContainer" containerID="bb7ea7b8f5029487b127bd8eeb1aeefb0c2fbccf711e1fae132d0310e6581328" Jul 07 19:03:34.386214 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:34.386197 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb7ea7b8f5029487b127bd8eeb1aeefb0c2fbccf711e1fae132d0310e6581328"} err="failed to get container status \"bb7ea7b8f5029487b127bd8eeb1aeefb0c2fbccf711e1fae132d0310e6581328\": rpc error: code = NotFound desc = could not find container \"bb7ea7b8f5029487b127bd8eeb1aeefb0c2fbccf711e1fae132d0310e6581328\": container with ID starting with bb7ea7b8f5029487b127bd8eeb1aeefb0c2fbccf711e1fae132d0310e6581328 not found: ID does not exist" Jul 07 19:03:34.418340 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:34.418306 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 07 19:03:34.418587 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:34.418575 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1e7deba9-69b3-4260-b6d0-037f274fb311" containerName="registry" Jul 07 19:03:34.418628 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:34.418589 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="1e7deba9-69b3-4260-b6d0-037f274fb311" containerName="registry" Jul 07 19:03:34.418628 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:34.418601 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8c425a2-1747-4fc6-b0d2-4a2cb46e4929" containerName="kube-rbac-proxy-metric" Jul 07 19:03:34.418628 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:34.418607 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8c425a2-1747-4fc6-b0d2-4a2cb46e4929" containerName="kube-rbac-proxy-metric" Jul 07 19:03:34.418628 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:34.418613 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8c425a2-1747-4fc6-b0d2-4a2cb46e4929" containerName="prom-label-proxy" Jul 07 19:03:34.418628 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:34.418618 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8c425a2-1747-4fc6-b0d2-4a2cb46e4929" containerName="prom-label-proxy" Jul 07 19:03:34.418628 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:34.418626 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8c425a2-1747-4fc6-b0d2-4a2cb46e4929" containerName="kube-rbac-proxy-web" Jul 07 19:03:34.418796 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:34.418632 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8c425a2-1747-4fc6-b0d2-4a2cb46e4929" containerName="kube-rbac-proxy-web" Jul 07 19:03:34.418796 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:34.418641 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8c425a2-1747-4fc6-b0d2-4a2cb46e4929" containerName="init-config-reloader" Jul 07 19:03:34.418796 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:34.418648 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8c425a2-1747-4fc6-b0d2-4a2cb46e4929" containerName="init-config-reloader" Jul 07 19:03:34.418796 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:34.418656 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8c425a2-1747-4fc6-b0d2-4a2cb46e4929" containerName="alertmanager" Jul 07 19:03:34.418796 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:34.418661 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8c425a2-1747-4fc6-b0d2-4a2cb46e4929" containerName="alertmanager" Jul 07 19:03:34.418796 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:34.418666 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8c425a2-1747-4fc6-b0d2-4a2cb46e4929" containerName="config-reloader" Jul 07 19:03:34.418796 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:34.418671 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8c425a2-1747-4fc6-b0d2-4a2cb46e4929" containerName="config-reloader" Jul 07 19:03:34.418796 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:34.418679 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8c425a2-1747-4fc6-b0d2-4a2cb46e4929" containerName="kube-rbac-proxy" Jul 07 19:03:34.418796 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:34.418684 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8c425a2-1747-4fc6-b0d2-4a2cb46e4929" containerName="kube-rbac-proxy" Jul 07 19:03:34.418796 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:34.418728 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="f8c425a2-1747-4fc6-b0d2-4a2cb46e4929" containerName="prom-label-proxy" Jul 07 19:03:34.418796 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:34.418736 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="1e7deba9-69b3-4260-b6d0-037f274fb311" containerName="registry" Jul 07 19:03:34.418796 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:34.418741 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="f8c425a2-1747-4fc6-b0d2-4a2cb46e4929" containerName="kube-rbac-proxy" Jul 07 19:03:34.418796 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:34.418747 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="f8c425a2-1747-4fc6-b0d2-4a2cb46e4929" containerName="alertmanager" Jul 07 19:03:34.418796 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:34.418752 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="f8c425a2-1747-4fc6-b0d2-4a2cb46e4929" containerName="config-reloader" Jul 07 19:03:34.418796 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:34.418757 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="f8c425a2-1747-4fc6-b0d2-4a2cb46e4929" containerName="kube-rbac-proxy-web" Jul 07 19:03:34.418796 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:34.418763 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="f8c425a2-1747-4fc6-b0d2-4a2cb46e4929" containerName="kube-rbac-proxy-metric" Jul 07 19:03:34.423692 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:34.423669 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 07 19:03:34.425982 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:34.425943 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 07 19:03:34.425982 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:34.425978 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 07 19:03:34.426190 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:34.425943 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 07 19:03:34.440886 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:34.440854 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 07 19:03:34.440886 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:34.440855 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 07 19:03:34.441150 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:34.440902 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 07 19:03:34.441150 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:34.440913 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 07 19:03:34.442841 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:34.442810 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 07 19:03:34.448128 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:34.448109 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-nnz82\"" Jul 07 19:03:34.456224 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:34.456195 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 07 19:03:34.525532 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:34.525494 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 07 19:03:34.553992 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:34.553931 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b41727ec-29e6-46d3-b63a-29141930d29b-config-out\") pod \"alertmanager-main-0\" (UID: \"b41727ec-29e6-46d3-b63a-29141930d29b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 19:03:34.553992 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:34.553987 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/b41727ec-29e6-46d3-b63a-29141930d29b-config-volume\") pod \"alertmanager-main-0\" (UID: \"b41727ec-29e6-46d3-b63a-29141930d29b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 19:03:34.553992 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:34.554005 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b41727ec-29e6-46d3-b63a-29141930d29b-web-config\") pod \"alertmanager-main-0\" (UID: \"b41727ec-29e6-46d3-b63a-29141930d29b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 19:03:34.554262 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:34.554056 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b41727ec-29e6-46d3-b63a-29141930d29b-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"b41727ec-29e6-46d3-b63a-29141930d29b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 19:03:34.554262 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:34.554187 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b41727ec-29e6-46d3-b63a-29141930d29b-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"b41727ec-29e6-46d3-b63a-29141930d29b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 19:03:34.554262 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:34.554222 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b41727ec-29e6-46d3-b63a-29141930d29b-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"b41727ec-29e6-46d3-b63a-29141930d29b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 19:03:34.554376 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:34.554284 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b41727ec-29e6-46d3-b63a-29141930d29b-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"b41727ec-29e6-46d3-b63a-29141930d29b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 19:03:34.554376 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:34.554309 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b41727ec-29e6-46d3-b63a-29141930d29b-tls-assets\") pod \"alertmanager-main-0\" (UID: \"b41727ec-29e6-46d3-b63a-29141930d29b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 19:03:34.554376 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:34.554327 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b41727ec-29e6-46d3-b63a-29141930d29b-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"b41727ec-29e6-46d3-b63a-29141930d29b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 19:03:34.554376 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:34.554342 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b41727ec-29e6-46d3-b63a-29141930d29b-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"b41727ec-29e6-46d3-b63a-29141930d29b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 19:03:34.554376 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:34.554356 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7sl8\" (UniqueName: \"kubernetes.io/projected/b41727ec-29e6-46d3-b63a-29141930d29b-kube-api-access-n7sl8\") pod \"alertmanager-main-0\" (UID: \"b41727ec-29e6-46d3-b63a-29141930d29b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 19:03:34.554534 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:34.554382 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/b41727ec-29e6-46d3-b63a-29141930d29b-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"b41727ec-29e6-46d3-b63a-29141930d29b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 19:03:34.554534 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:34.554442 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/b41727ec-29e6-46d3-b63a-29141930d29b-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"b41727ec-29e6-46d3-b63a-29141930d29b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 19:03:34.655371 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:34.655269 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b41727ec-29e6-46d3-b63a-29141930d29b-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"b41727ec-29e6-46d3-b63a-29141930d29b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 19:03:34.655371 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:34.655312 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b41727ec-29e6-46d3-b63a-29141930d29b-tls-assets\") pod \"alertmanager-main-0\" (UID: \"b41727ec-29e6-46d3-b63a-29141930d29b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 19:03:34.655371 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:34.655333 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b41727ec-29e6-46d3-b63a-29141930d29b-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"b41727ec-29e6-46d3-b63a-29141930d29b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 19:03:34.655371 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:34.655357 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b41727ec-29e6-46d3-b63a-29141930d29b-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"b41727ec-29e6-46d3-b63a-29141930d29b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 19:03:34.655708 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:34.655380 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7sl8\" (UniqueName: \"kubernetes.io/projected/b41727ec-29e6-46d3-b63a-29141930d29b-kube-api-access-n7sl8\") pod \"alertmanager-main-0\" (UID: \"b41727ec-29e6-46d3-b63a-29141930d29b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 19:03:34.655708 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:34.655409 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/b41727ec-29e6-46d3-b63a-29141930d29b-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"b41727ec-29e6-46d3-b63a-29141930d29b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 19:03:34.655708 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:34.655477 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/b41727ec-29e6-46d3-b63a-29141930d29b-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"b41727ec-29e6-46d3-b63a-29141930d29b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 19:03:34.655869 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:34.655740 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b41727ec-29e6-46d3-b63a-29141930d29b-config-out\") pod \"alertmanager-main-0\" (UID: \"b41727ec-29e6-46d3-b63a-29141930d29b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 19:03:34.656015 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:34.655995 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/b41727ec-29e6-46d3-b63a-29141930d29b-config-volume\") pod \"alertmanager-main-0\" (UID: \"b41727ec-29e6-46d3-b63a-29141930d29b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 19:03:34.656098 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:34.656030 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b41727ec-29e6-46d3-b63a-29141930d29b-web-config\") pod \"alertmanager-main-0\" (UID: \"b41727ec-29e6-46d3-b63a-29141930d29b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 19:03:34.656098 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:34.656058 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b41727ec-29e6-46d3-b63a-29141930d29b-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"b41727ec-29e6-46d3-b63a-29141930d29b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 19:03:34.656315 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:34.656105 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b41727ec-29e6-46d3-b63a-29141930d29b-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"b41727ec-29e6-46d3-b63a-29141930d29b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 19:03:34.656315 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:34.656139 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b41727ec-29e6-46d3-b63a-29141930d29b-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"b41727ec-29e6-46d3-b63a-29141930d29b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 19:03:34.656315 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:34.656148 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/b41727ec-29e6-46d3-b63a-29141930d29b-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"b41727ec-29e6-46d3-b63a-29141930d29b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 19:03:34.656500 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:34.656476 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b41727ec-29e6-46d3-b63a-29141930d29b-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"b41727ec-29e6-46d3-b63a-29141930d29b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 19:03:34.657079 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:34.657053 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b41727ec-29e6-46d3-b63a-29141930d29b-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"b41727ec-29e6-46d3-b63a-29141930d29b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 19:03:34.658628 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:34.658298 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b41727ec-29e6-46d3-b63a-29141930d29b-config-out\") pod \"alertmanager-main-0\" (UID: \"b41727ec-29e6-46d3-b63a-29141930d29b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 19:03:34.658628 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:34.658399 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b41727ec-29e6-46d3-b63a-29141930d29b-tls-assets\") pod \"alertmanager-main-0\" (UID: \"b41727ec-29e6-46d3-b63a-29141930d29b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 19:03:34.658628 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:34.658610 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/b41727ec-29e6-46d3-b63a-29141930d29b-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"b41727ec-29e6-46d3-b63a-29141930d29b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 19:03:34.658842 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:34.658784 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b41727ec-29e6-46d3-b63a-29141930d29b-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"b41727ec-29e6-46d3-b63a-29141930d29b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 19:03:34.659412 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:34.659393 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b41727ec-29e6-46d3-b63a-29141930d29b-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"b41727ec-29e6-46d3-b63a-29141930d29b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 19:03:34.659543 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:34.659519 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/b41727ec-29e6-46d3-b63a-29141930d29b-config-volume\") pod \"alertmanager-main-0\" (UID: \"b41727ec-29e6-46d3-b63a-29141930d29b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 19:03:34.659725 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:34.659706 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b41727ec-29e6-46d3-b63a-29141930d29b-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"b41727ec-29e6-46d3-b63a-29141930d29b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 19:03:34.659892 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:34.659876 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b41727ec-29e6-46d3-b63a-29141930d29b-web-config\") pod \"alertmanager-main-0\" (UID: \"b41727ec-29e6-46d3-b63a-29141930d29b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 19:03:34.660673 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:34.660645 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b41727ec-29e6-46d3-b63a-29141930d29b-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"b41727ec-29e6-46d3-b63a-29141930d29b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 19:03:34.666771 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:34.666740 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7sl8\" (UniqueName: \"kubernetes.io/projected/b41727ec-29e6-46d3-b63a-29141930d29b-kube-api-access-n7sl8\") pod \"alertmanager-main-0\" (UID: \"b41727ec-29e6-46d3-b63a-29141930d29b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 19:03:34.732933 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:34.732894 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 07 19:03:34.893415 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:34.893381 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 07 19:03:34.896582 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:03:34.896555 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb41727ec_29e6_46d3_b63a_29141930d29b.slice/crio-2683ab4864395da2652c9af7210aa8e4af5e5793705e2d2298dd93d23361747e WatchSource:0}: Error finding container 2683ab4864395da2652c9af7210aa8e4af5e5793705e2d2298dd93d23361747e: Status 404 returned error can't find the container with id 2683ab4864395da2652c9af7210aa8e4af5e5793705e2d2298dd93d23361747e Jul 07 19:03:35.335765 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:35.335730 2563 generic.go:356] "Generic (PLEG): container finished" podID="b41727ec-29e6-46d3-b63a-29141930d29b" containerID="e57b24fe8b14ce22c07fef3ada4204bb82d6f48a794e2d776ccb9cda3fa8def6" exitCode=0 Jul 07 19:03:35.336217 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:35.335807 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b41727ec-29e6-46d3-b63a-29141930d29b","Type":"ContainerDied","Data":"e57b24fe8b14ce22c07fef3ada4204bb82d6f48a794e2d776ccb9cda3fa8def6"} Jul 07 19:03:35.336217 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:35.335915 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b41727ec-29e6-46d3-b63a-29141930d29b","Type":"ContainerStarted","Data":"2683ab4864395da2652c9af7210aa8e4af5e5793705e2d2298dd93d23361747e"} Jul 07 19:03:35.476525 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:35.476496 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f8c425a2-1747-4fc6-b0d2-4a2cb46e4929" path="/var/lib/kubelet/pods/f8c425a2-1747-4fc6-b0d2-4a2cb46e4929/volumes" Jul 07 19:03:36.343683 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:36.343641 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b41727ec-29e6-46d3-b63a-29141930d29b","Type":"ContainerStarted","Data":"1ac8a3e9f95d633745ea9b2b19fec7b440bc80cb70f2b73c5c99a45c10609e24"} Jul 07 19:03:36.344117 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:36.343730 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b41727ec-29e6-46d3-b63a-29141930d29b","Type":"ContainerStarted","Data":"962e317b7974c938167955283377c5240e358ae3326978cab62e564983ed726a"} Jul 07 19:03:36.344117 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:36.343776 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b41727ec-29e6-46d3-b63a-29141930d29b","Type":"ContainerStarted","Data":"87f706461bb3339193ed76673045d45a50647ea5230d360cab00d75eaf76a84a"} Jul 07 19:03:36.344117 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:36.343813 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b41727ec-29e6-46d3-b63a-29141930d29b","Type":"ContainerStarted","Data":"2a9edc72af220b425b53c79e2f91b4223e1dc4f66327eb4111139a8ba486136b"} Jul 07 19:03:36.344117 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:36.343847 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b41727ec-29e6-46d3-b63a-29141930d29b","Type":"ContainerStarted","Data":"085db74e9684897d35b47329cb5dfc27e070b9085f3e797c7fc4633abfff8666"} Jul 07 19:03:36.344117 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:36.343894 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b41727ec-29e6-46d3-b63a-29141930d29b","Type":"ContainerStarted","Data":"10838ba29677a85346e83849cc8441ff5aa883af8bba84c7e9327a94f0772692"} Jul 07 19:03:36.391105 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:36.391053 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.371142392 podStartE2EDuration="2.391035069s" totalImagesPullingTime="0s" totalInitContainerRuntime="19.892677ms" isStatefulPod=false podCreationTimestamp="2026-07-07 19:03:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:03:36.390463243 +0000 UTC m=+199.406467427" watchObservedRunningTime="2026-07-07 19:03:36.391035069 +0000 UTC m=+199.407039271" Jul 07 19:03:36.658043 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:36.657961 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-5d4848bbfb-w9dk4"] Jul 07 19:03:36.661402 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:36.661382 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-5d4848bbfb-w9dk4" Jul 07 19:03:36.663898 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:36.663872 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-sgwlf\"" Jul 07 19:03:36.664027 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:36.663899 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Jul 07 19:03:36.664633 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:36.664615 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Jul 07 19:03:36.664883 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:36.664866 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Jul 07 19:03:36.665284 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:36.665271 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Jul 07 19:03:36.667190 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:36.667173 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Jul 07 19:03:36.680289 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:36.680256 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Jul 07 19:03:36.692003 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:36.691895 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-5d4848bbfb-w9dk4"] Jul 07 19:03:36.776577 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:36.776542 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/b43c0ae2-9e13-4beb-a30b-e2147f3d4e41-telemeter-client-tls\") pod \"telemeter-client-5d4848bbfb-w9dk4\" (UID: \"b43c0ae2-9e13-4beb-a30b-e2147f3d4e41\") " pod="openshift-monitoring/telemeter-client-5d4848bbfb-w9dk4" Jul 07 19:03:36.776749 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:36.776591 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/b43c0ae2-9e13-4beb-a30b-e2147f3d4e41-secret-telemeter-client\") pod \"telemeter-client-5d4848bbfb-w9dk4\" (UID: \"b43c0ae2-9e13-4beb-a30b-e2147f3d4e41\") " pod="openshift-monitoring/telemeter-client-5d4848bbfb-w9dk4" Jul 07 19:03:36.776749 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:36.776620 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b43c0ae2-9e13-4beb-a30b-e2147f3d4e41-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5d4848bbfb-w9dk4\" (UID: \"b43c0ae2-9e13-4beb-a30b-e2147f3d4e41\") " pod="openshift-monitoring/telemeter-client-5d4848bbfb-w9dk4" Jul 07 19:03:36.776749 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:36.776640 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2s9d\" (UniqueName: \"kubernetes.io/projected/b43c0ae2-9e13-4beb-a30b-e2147f3d4e41-kube-api-access-h2s9d\") pod \"telemeter-client-5d4848bbfb-w9dk4\" (UID: \"b43c0ae2-9e13-4beb-a30b-e2147f3d4e41\") " pod="openshift-monitoring/telemeter-client-5d4848bbfb-w9dk4" Jul 07 19:03:36.776749 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:36.776658 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/b43c0ae2-9e13-4beb-a30b-e2147f3d4e41-federate-client-tls\") pod \"telemeter-client-5d4848bbfb-w9dk4\" (UID: \"b43c0ae2-9e13-4beb-a30b-e2147f3d4e41\") " pod="openshift-monitoring/telemeter-client-5d4848bbfb-w9dk4" Jul 07 19:03:36.776876 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:36.776777 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b43c0ae2-9e13-4beb-a30b-e2147f3d4e41-serving-certs-ca-bundle\") pod \"telemeter-client-5d4848bbfb-w9dk4\" (UID: \"b43c0ae2-9e13-4beb-a30b-e2147f3d4e41\") " pod="openshift-monitoring/telemeter-client-5d4848bbfb-w9dk4" Jul 07 19:03:36.776876 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:36.776831 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b43c0ae2-9e13-4beb-a30b-e2147f3d4e41-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5d4848bbfb-w9dk4\" (UID: \"b43c0ae2-9e13-4beb-a30b-e2147f3d4e41\") " pod="openshift-monitoring/telemeter-client-5d4848bbfb-w9dk4" Jul 07 19:03:36.776876 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:36.776862 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b43c0ae2-9e13-4beb-a30b-e2147f3d4e41-metrics-client-ca\") pod \"telemeter-client-5d4848bbfb-w9dk4\" (UID: \"b43c0ae2-9e13-4beb-a30b-e2147f3d4e41\") " pod="openshift-monitoring/telemeter-client-5d4848bbfb-w9dk4" Jul 07 19:03:36.877840 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:36.877802 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/b43c0ae2-9e13-4beb-a30b-e2147f3d4e41-telemeter-client-tls\") pod \"telemeter-client-5d4848bbfb-w9dk4\" (UID: \"b43c0ae2-9e13-4beb-a30b-e2147f3d4e41\") " pod="openshift-monitoring/telemeter-client-5d4848bbfb-w9dk4" Jul 07 19:03:36.878074 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:36.877856 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/b43c0ae2-9e13-4beb-a30b-e2147f3d4e41-secret-telemeter-client\") pod \"telemeter-client-5d4848bbfb-w9dk4\" (UID: \"b43c0ae2-9e13-4beb-a30b-e2147f3d4e41\") " pod="openshift-monitoring/telemeter-client-5d4848bbfb-w9dk4" Jul 07 19:03:36.878074 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:36.877883 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b43c0ae2-9e13-4beb-a30b-e2147f3d4e41-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5d4848bbfb-w9dk4\" (UID: \"b43c0ae2-9e13-4beb-a30b-e2147f3d4e41\") " pod="openshift-monitoring/telemeter-client-5d4848bbfb-w9dk4" Jul 07 19:03:36.878074 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:36.877916 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2s9d\" (UniqueName: \"kubernetes.io/projected/b43c0ae2-9e13-4beb-a30b-e2147f3d4e41-kube-api-access-h2s9d\") pod \"telemeter-client-5d4848bbfb-w9dk4\" (UID: \"b43c0ae2-9e13-4beb-a30b-e2147f3d4e41\") " pod="openshift-monitoring/telemeter-client-5d4848bbfb-w9dk4" Jul 07 19:03:36.878074 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:36.877943 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/b43c0ae2-9e13-4beb-a30b-e2147f3d4e41-federate-client-tls\") pod \"telemeter-client-5d4848bbfb-w9dk4\" (UID: \"b43c0ae2-9e13-4beb-a30b-e2147f3d4e41\") " pod="openshift-monitoring/telemeter-client-5d4848bbfb-w9dk4" Jul 07 19:03:36.878074 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:36.878032 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b43c0ae2-9e13-4beb-a30b-e2147f3d4e41-serving-certs-ca-bundle\") pod \"telemeter-client-5d4848bbfb-w9dk4\" (UID: \"b43c0ae2-9e13-4beb-a30b-e2147f3d4e41\") " pod="openshift-monitoring/telemeter-client-5d4848bbfb-w9dk4" Jul 07 19:03:36.878074 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:36.878068 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b43c0ae2-9e13-4beb-a30b-e2147f3d4e41-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5d4848bbfb-w9dk4\" (UID: \"b43c0ae2-9e13-4beb-a30b-e2147f3d4e41\") " pod="openshift-monitoring/telemeter-client-5d4848bbfb-w9dk4" Jul 07 19:03:36.878376 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:36.878115 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b43c0ae2-9e13-4beb-a30b-e2147f3d4e41-metrics-client-ca\") pod \"telemeter-client-5d4848bbfb-w9dk4\" (UID: \"b43c0ae2-9e13-4beb-a30b-e2147f3d4e41\") " pod="openshift-monitoring/telemeter-client-5d4848bbfb-w9dk4" Jul 07 19:03:36.878806 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:36.878782 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b43c0ae2-9e13-4beb-a30b-e2147f3d4e41-metrics-client-ca\") pod \"telemeter-client-5d4848bbfb-w9dk4\" (UID: \"b43c0ae2-9e13-4beb-a30b-e2147f3d4e41\") " pod="openshift-monitoring/telemeter-client-5d4848bbfb-w9dk4" Jul 07 19:03:36.878922 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:36.878898 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b43c0ae2-9e13-4beb-a30b-e2147f3d4e41-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5d4848bbfb-w9dk4\" (UID: \"b43c0ae2-9e13-4beb-a30b-e2147f3d4e41\") " pod="openshift-monitoring/telemeter-client-5d4848bbfb-w9dk4" Jul 07 19:03:36.879005 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:36.878982 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b43c0ae2-9e13-4beb-a30b-e2147f3d4e41-serving-certs-ca-bundle\") pod \"telemeter-client-5d4848bbfb-w9dk4\" (UID: \"b43c0ae2-9e13-4beb-a30b-e2147f3d4e41\") " pod="openshift-monitoring/telemeter-client-5d4848bbfb-w9dk4" Jul 07 19:03:36.880499 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:36.880468 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b43c0ae2-9e13-4beb-a30b-e2147f3d4e41-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5d4848bbfb-w9dk4\" (UID: \"b43c0ae2-9e13-4beb-a30b-e2147f3d4e41\") " pod="openshift-monitoring/telemeter-client-5d4848bbfb-w9dk4" Jul 07 19:03:36.880653 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:36.880630 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/b43c0ae2-9e13-4beb-a30b-e2147f3d4e41-secret-telemeter-client\") pod \"telemeter-client-5d4848bbfb-w9dk4\" (UID: \"b43c0ae2-9e13-4beb-a30b-e2147f3d4e41\") " pod="openshift-monitoring/telemeter-client-5d4848bbfb-w9dk4" Jul 07 19:03:36.880772 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:36.880758 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/b43c0ae2-9e13-4beb-a30b-e2147f3d4e41-federate-client-tls\") pod \"telemeter-client-5d4848bbfb-w9dk4\" (UID: \"b43c0ae2-9e13-4beb-a30b-e2147f3d4e41\") " pod="openshift-monitoring/telemeter-client-5d4848bbfb-w9dk4" Jul 07 19:03:36.880851 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:36.880836 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/b43c0ae2-9e13-4beb-a30b-e2147f3d4e41-telemeter-client-tls\") pod \"telemeter-client-5d4848bbfb-w9dk4\" (UID: \"b43c0ae2-9e13-4beb-a30b-e2147f3d4e41\") " pod="openshift-monitoring/telemeter-client-5d4848bbfb-w9dk4" Jul 07 19:03:36.894716 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:36.894690 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2s9d\" (UniqueName: \"kubernetes.io/projected/b43c0ae2-9e13-4beb-a30b-e2147f3d4e41-kube-api-access-h2s9d\") pod \"telemeter-client-5d4848bbfb-w9dk4\" (UID: \"b43c0ae2-9e13-4beb-a30b-e2147f3d4e41\") " pod="openshift-monitoring/telemeter-client-5d4848bbfb-w9dk4" Jul 07 19:03:36.971323 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:36.971217 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-5d4848bbfb-w9dk4" Jul 07 19:03:36.979533 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:36.977500 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 07 19:03:36.979533 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:36.978058 2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="d27c9f8c-0884-467d-ae62-9193d6950191" containerName="prometheus" containerID="cri-o://a46ae0c3beca66e600238a2bd178354d061b72d7fb5267ecdfb2d685cea43af0" gracePeriod=600 Jul 07 19:03:36.979533 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:36.979026 2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="d27c9f8c-0884-467d-ae62-9193d6950191" containerName="kube-rbac-proxy-thanos" containerID="cri-o://642e2e3f887dc2826bc7aa92e29de4499b4e1e873cea8fbec1f1e28f2f0e2091" gracePeriod=600 Jul 07 19:03:36.979533 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:36.979174 2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="d27c9f8c-0884-467d-ae62-9193d6950191" containerName="kube-rbac-proxy" containerID="cri-o://4ca654707d5d4fcc607ae59f52a684e43360f3946743a9e402e1261500880ac7" gracePeriod=600 Jul 07 19:03:36.979533 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:36.979267 2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="d27c9f8c-0884-467d-ae62-9193d6950191" containerName="kube-rbac-proxy-web" containerID="cri-o://1472dd326ad564274c5e93960ed0abb12f890ff81d7d8a15c278498000684c10" gracePeriod=600 Jul 07 19:03:36.979533 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:36.979360 2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="d27c9f8c-0884-467d-ae62-9193d6950191" containerName="thanos-sidecar" containerID="cri-o://146645ac04a6265cbafa9a51132ccfed6688ca9d523c3adf9a759a145167a3b7" gracePeriod=600 Jul 07 19:03:36.979533 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:36.979449 2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="d27c9f8c-0884-467d-ae62-9193d6950191" containerName="config-reloader" containerID="cri-o://e6fe0065e5a888e34565995c53a516d658ec05c20c9c006207a83fa67796b50e" gracePeriod=600 Jul 07 19:03:37.162480 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.162444 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-5d4848bbfb-w9dk4"] Jul 07 19:03:37.165337 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:03:37.165299 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb43c0ae2_9e13_4beb_a30b_e2147f3d4e41.slice/crio-26d2b0d0b01e156e70e063635fbe390384220cf2f656185ea9f294c24bebb34c WatchSource:0}: Error finding container 26d2b0d0b01e156e70e063635fbe390384220cf2f656185ea9f294c24bebb34c: Status 404 returned error can't find the container with id 26d2b0d0b01e156e70e063635fbe390384220cf2f656185ea9f294c24bebb34c Jul 07 19:03:37.246189 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.246158 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:03:37.350000 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.349945 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5d4848bbfb-w9dk4" event={"ID":"b43c0ae2-9e13-4beb-a30b-e2147f3d4e41","Type":"ContainerStarted","Data":"26d2b0d0b01e156e70e063635fbe390384220cf2f656185ea9f294c24bebb34c"} Jul 07 19:03:37.353253 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.353225 2563 generic.go:356] "Generic (PLEG): container finished" podID="d27c9f8c-0884-467d-ae62-9193d6950191" containerID="642e2e3f887dc2826bc7aa92e29de4499b4e1e873cea8fbec1f1e28f2f0e2091" exitCode=0 Jul 07 19:03:37.353253 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.353251 2563 generic.go:356] "Generic (PLEG): container finished" podID="d27c9f8c-0884-467d-ae62-9193d6950191" containerID="4ca654707d5d4fcc607ae59f52a684e43360f3946743a9e402e1261500880ac7" exitCode=0 Jul 07 19:03:37.353403 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.353261 2563 generic.go:356] "Generic (PLEG): container finished" podID="d27c9f8c-0884-467d-ae62-9193d6950191" containerID="1472dd326ad564274c5e93960ed0abb12f890ff81d7d8a15c278498000684c10" exitCode=0 Jul 07 19:03:37.353403 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.353273 2563 generic.go:356] "Generic (PLEG): container finished" podID="d27c9f8c-0884-467d-ae62-9193d6950191" containerID="146645ac04a6265cbafa9a51132ccfed6688ca9d523c3adf9a759a145167a3b7" exitCode=0 Jul 07 19:03:37.353403 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.353281 2563 generic.go:356] "Generic (PLEG): container finished" podID="d27c9f8c-0884-467d-ae62-9193d6950191" containerID="e6fe0065e5a888e34565995c53a516d658ec05c20c9c006207a83fa67796b50e" exitCode=0 Jul 07 19:03:37.353403 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.353289 2563 generic.go:356] "Generic (PLEG): container finished" podID="d27c9f8c-0884-467d-ae62-9193d6950191" containerID="a46ae0c3beca66e600238a2bd178354d061b72d7fb5267ecdfb2d685cea43af0" exitCode=0 Jul 07 19:03:37.353403 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.353303 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d27c9f8c-0884-467d-ae62-9193d6950191","Type":"ContainerDied","Data":"642e2e3f887dc2826bc7aa92e29de4499b4e1e873cea8fbec1f1e28f2f0e2091"} Jul 07 19:03:37.353403 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.353315 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:03:37.353403 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.353364 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d27c9f8c-0884-467d-ae62-9193d6950191","Type":"ContainerDied","Data":"4ca654707d5d4fcc607ae59f52a684e43360f3946743a9e402e1261500880ac7"} Jul 07 19:03:37.353403 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.353381 2563 scope.go:117] "RemoveContainer" containerID="642e2e3f887dc2826bc7aa92e29de4499b4e1e873cea8fbec1f1e28f2f0e2091" Jul 07 19:03:37.353403 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.353397 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d27c9f8c-0884-467d-ae62-9193d6950191","Type":"ContainerDied","Data":"1472dd326ad564274c5e93960ed0abb12f890ff81d7d8a15c278498000684c10"} Jul 07 19:03:37.353732 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.353422 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d27c9f8c-0884-467d-ae62-9193d6950191","Type":"ContainerDied","Data":"146645ac04a6265cbafa9a51132ccfed6688ca9d523c3adf9a759a145167a3b7"} Jul 07 19:03:37.353732 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.353456 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d27c9f8c-0884-467d-ae62-9193d6950191","Type":"ContainerDied","Data":"e6fe0065e5a888e34565995c53a516d658ec05c20c9c006207a83fa67796b50e"} Jul 07 19:03:37.353732 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.353480 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d27c9f8c-0884-467d-ae62-9193d6950191","Type":"ContainerDied","Data":"a46ae0c3beca66e600238a2bd178354d061b72d7fb5267ecdfb2d685cea43af0"} Jul 07 19:03:37.353732 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.353504 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d27c9f8c-0884-467d-ae62-9193d6950191","Type":"ContainerDied","Data":"ae73344b97776bcaca79451e6faf743e20c98450bff248bd1ce883df01dd9526"} Jul 07 19:03:37.361925 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.361836 2563 scope.go:117] "RemoveContainer" containerID="4ca654707d5d4fcc607ae59f52a684e43360f3946743a9e402e1261500880ac7" Jul 07 19:03:37.369325 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.369307 2563 scope.go:117] "RemoveContainer" containerID="1472dd326ad564274c5e93960ed0abb12f890ff81d7d8a15c278498000684c10" Jul 07 19:03:37.378320 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.377984 2563 scope.go:117] "RemoveContainer" containerID="146645ac04a6265cbafa9a51132ccfed6688ca9d523c3adf9a759a145167a3b7" Jul 07 19:03:37.383748 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.383720 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/d27c9f8c-0884-467d-ae62-9193d6950191-secret-prometheus-k8s-tls\") pod \"d27c9f8c-0884-467d-ae62-9193d6950191\" (UID: \"d27c9f8c-0884-467d-ae62-9193d6950191\") " Jul 07 19:03:37.383847 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.383756 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/d27c9f8c-0884-467d-ae62-9193d6950191-thanos-prometheus-http-client-file\") pod \"d27c9f8c-0884-467d-ae62-9193d6950191\" (UID: \"d27c9f8c-0884-467d-ae62-9193d6950191\") " Jul 07 19:03:37.383847 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.383789 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d27c9f8c-0884-467d-ae62-9193d6950191-secret-grpc-tls\") pod \"d27c9f8c-0884-467d-ae62-9193d6950191\" (UID: \"d27c9f8c-0884-467d-ae62-9193d6950191\") " Jul 07 19:03:37.383847 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.383817 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d27c9f8c-0884-467d-ae62-9193d6950191-configmap-kubelet-serving-ca-bundle\") pod \"d27c9f8c-0884-467d-ae62-9193d6950191\" (UID: \"d27c9f8c-0884-467d-ae62-9193d6950191\") " Jul 07 19:03:37.383847 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.383847 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/d27c9f8c-0884-467d-ae62-9193d6950191-secret-metrics-client-certs\") pod \"d27c9f8c-0884-467d-ae62-9193d6950191\" (UID: \"d27c9f8c-0884-467d-ae62-9193d6950191\") " Jul 07 19:03:37.384082 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.383871 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/d27c9f8c-0884-467d-ae62-9193d6950191-prometheus-k8s-db\") pod \"d27c9f8c-0884-467d-ae62-9193d6950191\" (UID: \"d27c9f8c-0884-467d-ae62-9193d6950191\") " Jul 07 19:03:37.384082 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.383896 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d27c9f8c-0884-467d-ae62-9193d6950191-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"d27c9f8c-0884-467d-ae62-9193d6950191\" (UID: \"d27c9f8c-0884-467d-ae62-9193d6950191\") " Jul 07 19:03:37.384082 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.383937 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/d27c9f8c-0884-467d-ae62-9193d6950191-prometheus-k8s-rulefiles-0\") pod \"d27c9f8c-0884-467d-ae62-9193d6950191\" (UID: \"d27c9f8c-0884-467d-ae62-9193d6950191\") " Jul 07 19:03:37.384082 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.383984 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/d27c9f8c-0884-467d-ae62-9193d6950191-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"d27c9f8c-0884-467d-ae62-9193d6950191\" (UID: \"d27c9f8c-0884-467d-ae62-9193d6950191\") " Jul 07 19:03:37.384082 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.384017 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/d27c9f8c-0884-467d-ae62-9193d6950191-config\") pod \"d27c9f8c-0884-467d-ae62-9193d6950191\" (UID: \"d27c9f8c-0884-467d-ae62-9193d6950191\") " Jul 07 19:03:37.384082 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.384048 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d27c9f8c-0884-467d-ae62-9193d6950191-prometheus-trusted-ca-bundle\") pod \"d27c9f8c-0884-467d-ae62-9193d6950191\" (UID: \"d27c9f8c-0884-467d-ae62-9193d6950191\") " Jul 07 19:03:37.384082 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.384081 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d27c9f8c-0884-467d-ae62-9193d6950191-tls-assets\") pod \"d27c9f8c-0884-467d-ae62-9193d6950191\" (UID: \"d27c9f8c-0884-467d-ae62-9193d6950191\") " Jul 07 19:03:37.384447 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.384104 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d27c9f8c-0884-467d-ae62-9193d6950191-configmap-serving-certs-ca-bundle\") pod \"d27c9f8c-0884-467d-ae62-9193d6950191\" (UID: \"d27c9f8c-0884-467d-ae62-9193d6950191\") " Jul 07 19:03:37.384447 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.384130 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/d27c9f8c-0884-467d-ae62-9193d6950191-prometheus-k8s-rulefiles-1\") pod \"d27c9f8c-0884-467d-ae62-9193d6950191\" (UID: \"d27c9f8c-0884-467d-ae62-9193d6950191\") " Jul 07 19:03:37.384447 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.384196 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d27c9f8c-0884-467d-ae62-9193d6950191-secret-kube-rbac-proxy\") pod \"d27c9f8c-0884-467d-ae62-9193d6950191\" (UID: \"d27c9f8c-0884-467d-ae62-9193d6950191\") " Jul 07 19:03:37.384447 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.384221 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/d27c9f8c-0884-467d-ae62-9193d6950191-prometheus-k8s-rulefiles-2\") pod \"d27c9f8c-0884-467d-ae62-9193d6950191\" (UID: \"d27c9f8c-0884-467d-ae62-9193d6950191\") " Jul 07 19:03:37.384447 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.384260 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d27c9f8c-0884-467d-ae62-9193d6950191-configmap-metrics-client-ca\") pod \"d27c9f8c-0884-467d-ae62-9193d6950191\" (UID: \"d27c9f8c-0884-467d-ae62-9193d6950191\") " Jul 07 19:03:37.384447 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.384286 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d27c9f8c-0884-467d-ae62-9193d6950191-web-config\") pod \"d27c9f8c-0884-467d-ae62-9193d6950191\" (UID: \"d27c9f8c-0884-467d-ae62-9193d6950191\") " Jul 07 19:03:37.384447 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.384316 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rgtvz\" (UniqueName: \"kubernetes.io/projected/d27c9f8c-0884-467d-ae62-9193d6950191-kube-api-access-rgtvz\") pod \"d27c9f8c-0884-467d-ae62-9193d6950191\" (UID: \"d27c9f8c-0884-467d-ae62-9193d6950191\") " Jul 07 19:03:37.384447 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.384319 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d27c9f8c-0884-467d-ae62-9193d6950191-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "d27c9f8c-0884-467d-ae62-9193d6950191" (UID: "d27c9f8c-0884-467d-ae62-9193d6950191"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 19:03:37.384447 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.384343 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d27c9f8c-0884-467d-ae62-9193d6950191-config-out\") pod \"d27c9f8c-0884-467d-ae62-9193d6950191\" (UID: \"d27c9f8c-0884-467d-ae62-9193d6950191\") " Jul 07 19:03:37.384899 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.384648 2563 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d27c9f8c-0884-467d-ae62-9193d6950191-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-142-134.ec2.internal\" DevicePath \"\"" Jul 07 19:03:37.384899 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.384674 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d27c9f8c-0884-467d-ae62-9193d6950191-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "d27c9f8c-0884-467d-ae62-9193d6950191" (UID: "d27c9f8c-0884-467d-ae62-9193d6950191"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 19:03:37.385630 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.385326 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d27c9f8c-0884-467d-ae62-9193d6950191-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "d27c9f8c-0884-467d-ae62-9193d6950191" (UID: "d27c9f8c-0884-467d-ae62-9193d6950191"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:03:37.388320 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.386072 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d27c9f8c-0884-467d-ae62-9193d6950191-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "d27c9f8c-0884-467d-ae62-9193d6950191" (UID: "d27c9f8c-0884-467d-ae62-9193d6950191"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 19:03:37.388320 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.386383 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d27c9f8c-0884-467d-ae62-9193d6950191-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "d27c9f8c-0884-467d-ae62-9193d6950191" (UID: "d27c9f8c-0884-467d-ae62-9193d6950191"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 19:03:37.388320 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.386626 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d27c9f8c-0884-467d-ae62-9193d6950191-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "d27c9f8c-0884-467d-ae62-9193d6950191" (UID: "d27c9f8c-0884-467d-ae62-9193d6950191"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 19:03:37.388320 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.386640 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d27c9f8c-0884-467d-ae62-9193d6950191-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "d27c9f8c-0884-467d-ae62-9193d6950191" (UID: "d27c9f8c-0884-467d-ae62-9193d6950191"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 19:03:37.389573 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.389193 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d27c9f8c-0884-467d-ae62-9193d6950191-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "d27c9f8c-0884-467d-ae62-9193d6950191" (UID: "d27c9f8c-0884-467d-ae62-9193d6950191"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 19:03:37.389573 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.389306 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d27c9f8c-0884-467d-ae62-9193d6950191-config-out" (OuterVolumeSpecName: "config-out") pod "d27c9f8c-0884-467d-ae62-9193d6950191" (UID: "d27c9f8c-0884-467d-ae62-9193d6950191"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:03:37.389573 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.389364 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d27c9f8c-0884-467d-ae62-9193d6950191-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "d27c9f8c-0884-467d-ae62-9193d6950191" (UID: "d27c9f8c-0884-467d-ae62-9193d6950191"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 19:03:37.389573 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.389415 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d27c9f8c-0884-467d-ae62-9193d6950191-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "d27c9f8c-0884-467d-ae62-9193d6950191" (UID: "d27c9f8c-0884-467d-ae62-9193d6950191"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 19:03:37.389573 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.389470 2563 scope.go:117] "RemoveContainer" containerID="e6fe0065e5a888e34565995c53a516d658ec05c20c9c006207a83fa67796b50e" Jul 07 19:03:37.389573 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.389531 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d27c9f8c-0884-467d-ae62-9193d6950191-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "d27c9f8c-0884-467d-ae62-9193d6950191" (UID: "d27c9f8c-0884-467d-ae62-9193d6950191"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 19:03:37.390589 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.390453 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d27c9f8c-0884-467d-ae62-9193d6950191-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "d27c9f8c-0884-467d-ae62-9193d6950191" (UID: "d27c9f8c-0884-467d-ae62-9193d6950191"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 19:03:37.390589 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.390515 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d27c9f8c-0884-467d-ae62-9193d6950191-config" (OuterVolumeSpecName: "config") pod "d27c9f8c-0884-467d-ae62-9193d6950191" (UID: "d27c9f8c-0884-467d-ae62-9193d6950191"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 19:03:37.390746 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.390603 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d27c9f8c-0884-467d-ae62-9193d6950191-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "d27c9f8c-0884-467d-ae62-9193d6950191" (UID: "d27c9f8c-0884-467d-ae62-9193d6950191"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 19:03:37.390746 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.390618 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d27c9f8c-0884-467d-ae62-9193d6950191-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "d27c9f8c-0884-467d-ae62-9193d6950191" (UID: "d27c9f8c-0884-467d-ae62-9193d6950191"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 19:03:37.391635 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.391609 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d27c9f8c-0884-467d-ae62-9193d6950191-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "d27c9f8c-0884-467d-ae62-9193d6950191" (UID: "d27c9f8c-0884-467d-ae62-9193d6950191"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 19:03:37.391707 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.391641 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d27c9f8c-0884-467d-ae62-9193d6950191-kube-api-access-rgtvz" (OuterVolumeSpecName: "kube-api-access-rgtvz") pod "d27c9f8c-0884-467d-ae62-9193d6950191" (UID: "d27c9f8c-0884-467d-ae62-9193d6950191"). InnerVolumeSpecName "kube-api-access-rgtvz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 19:03:37.392104 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.392087 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d27c9f8c-0884-467d-ae62-9193d6950191-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "d27c9f8c-0884-467d-ae62-9193d6950191" (UID: "d27c9f8c-0884-467d-ae62-9193d6950191"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 19:03:37.397605 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.397585 2563 scope.go:117] "RemoveContainer" containerID="a46ae0c3beca66e600238a2bd178354d061b72d7fb5267ecdfb2d685cea43af0" Jul 07 19:03:37.401388 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.401367 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d27c9f8c-0884-467d-ae62-9193d6950191-web-config" (OuterVolumeSpecName: "web-config") pod "d27c9f8c-0884-467d-ae62-9193d6950191" (UID: "d27c9f8c-0884-467d-ae62-9193d6950191"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 19:03:37.404808 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.404792 2563 scope.go:117] "RemoveContainer" containerID="f9e867a77df9e6aee0670b91ed40795b79a63a7a63fcf93801429e87dc94924c" Jul 07 19:03:37.411631 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.411610 2563 scope.go:117] "RemoveContainer" containerID="642e2e3f887dc2826bc7aa92e29de4499b4e1e873cea8fbec1f1e28f2f0e2091" Jul 07 19:03:37.411884 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:03:37.411866 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"642e2e3f887dc2826bc7aa92e29de4499b4e1e873cea8fbec1f1e28f2f0e2091\": container with ID starting with 642e2e3f887dc2826bc7aa92e29de4499b4e1e873cea8fbec1f1e28f2f0e2091 not found: ID does not exist" containerID="642e2e3f887dc2826bc7aa92e29de4499b4e1e873cea8fbec1f1e28f2f0e2091" Jul 07 19:03:37.411934 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.411893 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"642e2e3f887dc2826bc7aa92e29de4499b4e1e873cea8fbec1f1e28f2f0e2091"} err="failed to get container status \"642e2e3f887dc2826bc7aa92e29de4499b4e1e873cea8fbec1f1e28f2f0e2091\": rpc error: code = NotFound desc = could not find container \"642e2e3f887dc2826bc7aa92e29de4499b4e1e873cea8fbec1f1e28f2f0e2091\": container with ID starting with 642e2e3f887dc2826bc7aa92e29de4499b4e1e873cea8fbec1f1e28f2f0e2091 not found: ID does not exist" Jul 07 19:03:37.411934 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.411912 2563 scope.go:117] "RemoveContainer" containerID="4ca654707d5d4fcc607ae59f52a684e43360f3946743a9e402e1261500880ac7" Jul 07 19:03:37.412171 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:03:37.412151 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ca654707d5d4fcc607ae59f52a684e43360f3946743a9e402e1261500880ac7\": container with ID starting with 4ca654707d5d4fcc607ae59f52a684e43360f3946743a9e402e1261500880ac7 not found: ID does not exist" containerID="4ca654707d5d4fcc607ae59f52a684e43360f3946743a9e402e1261500880ac7" Jul 07 19:03:37.412216 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.412180 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ca654707d5d4fcc607ae59f52a684e43360f3946743a9e402e1261500880ac7"} err="failed to get container status \"4ca654707d5d4fcc607ae59f52a684e43360f3946743a9e402e1261500880ac7\": rpc error: code = NotFound desc = could not find container \"4ca654707d5d4fcc607ae59f52a684e43360f3946743a9e402e1261500880ac7\": container with ID starting with 4ca654707d5d4fcc607ae59f52a684e43360f3946743a9e402e1261500880ac7 not found: ID does not exist" Jul 07 19:03:37.412216 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.412195 2563 scope.go:117] "RemoveContainer" containerID="1472dd326ad564274c5e93960ed0abb12f890ff81d7d8a15c278498000684c10" Jul 07 19:03:37.412440 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:03:37.412420 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1472dd326ad564274c5e93960ed0abb12f890ff81d7d8a15c278498000684c10\": container with ID starting with 1472dd326ad564274c5e93960ed0abb12f890ff81d7d8a15c278498000684c10 not found: ID does not exist" containerID="1472dd326ad564274c5e93960ed0abb12f890ff81d7d8a15c278498000684c10" Jul 07 19:03:37.412504 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.412446 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1472dd326ad564274c5e93960ed0abb12f890ff81d7d8a15c278498000684c10"} err="failed to get container status \"1472dd326ad564274c5e93960ed0abb12f890ff81d7d8a15c278498000684c10\": rpc error: code = NotFound desc = could not find container \"1472dd326ad564274c5e93960ed0abb12f890ff81d7d8a15c278498000684c10\": container with ID starting with 1472dd326ad564274c5e93960ed0abb12f890ff81d7d8a15c278498000684c10 not found: ID does not exist" Jul 07 19:03:37.412504 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.412477 2563 scope.go:117] "RemoveContainer" containerID="146645ac04a6265cbafa9a51132ccfed6688ca9d523c3adf9a759a145167a3b7" Jul 07 19:03:37.412691 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:03:37.412674 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"146645ac04a6265cbafa9a51132ccfed6688ca9d523c3adf9a759a145167a3b7\": container with ID starting with 146645ac04a6265cbafa9a51132ccfed6688ca9d523c3adf9a759a145167a3b7 not found: ID does not exist" containerID="146645ac04a6265cbafa9a51132ccfed6688ca9d523c3adf9a759a145167a3b7" Jul 07 19:03:37.412734 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.412694 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"146645ac04a6265cbafa9a51132ccfed6688ca9d523c3adf9a759a145167a3b7"} err="failed to get container status \"146645ac04a6265cbafa9a51132ccfed6688ca9d523c3adf9a759a145167a3b7\": rpc error: code = NotFound desc = could not find container \"146645ac04a6265cbafa9a51132ccfed6688ca9d523c3adf9a759a145167a3b7\": container with ID starting with 146645ac04a6265cbafa9a51132ccfed6688ca9d523c3adf9a759a145167a3b7 not found: ID does not exist" Jul 07 19:03:37.412734 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.412707 2563 scope.go:117] "RemoveContainer" containerID="e6fe0065e5a888e34565995c53a516d658ec05c20c9c006207a83fa67796b50e" Jul 07 19:03:37.412975 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:03:37.412932 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e6fe0065e5a888e34565995c53a516d658ec05c20c9c006207a83fa67796b50e\": container with ID starting with e6fe0065e5a888e34565995c53a516d658ec05c20c9c006207a83fa67796b50e not found: ID does not exist" containerID="e6fe0065e5a888e34565995c53a516d658ec05c20c9c006207a83fa67796b50e" Jul 07 19:03:37.413063 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.413045 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6fe0065e5a888e34565995c53a516d658ec05c20c9c006207a83fa67796b50e"} err="failed to get container status \"e6fe0065e5a888e34565995c53a516d658ec05c20c9c006207a83fa67796b50e\": rpc error: code = NotFound desc = could not find container \"e6fe0065e5a888e34565995c53a516d658ec05c20c9c006207a83fa67796b50e\": container with ID starting with e6fe0065e5a888e34565995c53a516d658ec05c20c9c006207a83fa67796b50e not found: ID does not exist" Jul 07 19:03:37.413152 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.413065 2563 scope.go:117] "RemoveContainer" containerID="a46ae0c3beca66e600238a2bd178354d061b72d7fb5267ecdfb2d685cea43af0" Jul 07 19:03:37.413300 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:03:37.413280 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a46ae0c3beca66e600238a2bd178354d061b72d7fb5267ecdfb2d685cea43af0\": container with ID starting with a46ae0c3beca66e600238a2bd178354d061b72d7fb5267ecdfb2d685cea43af0 not found: ID does not exist" containerID="a46ae0c3beca66e600238a2bd178354d061b72d7fb5267ecdfb2d685cea43af0" Jul 07 19:03:37.413347 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.413307 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a46ae0c3beca66e600238a2bd178354d061b72d7fb5267ecdfb2d685cea43af0"} err="failed to get container status \"a46ae0c3beca66e600238a2bd178354d061b72d7fb5267ecdfb2d685cea43af0\": rpc error: code = NotFound desc = could not find container \"a46ae0c3beca66e600238a2bd178354d061b72d7fb5267ecdfb2d685cea43af0\": container with ID starting with a46ae0c3beca66e600238a2bd178354d061b72d7fb5267ecdfb2d685cea43af0 not found: ID does not exist" Jul 07 19:03:37.413347 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.413321 2563 scope.go:117] "RemoveContainer" containerID="f9e867a77df9e6aee0670b91ed40795b79a63a7a63fcf93801429e87dc94924c" Jul 07 19:03:37.413556 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:03:37.413541 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9e867a77df9e6aee0670b91ed40795b79a63a7a63fcf93801429e87dc94924c\": container with ID starting with f9e867a77df9e6aee0670b91ed40795b79a63a7a63fcf93801429e87dc94924c not found: ID does not exist" containerID="f9e867a77df9e6aee0670b91ed40795b79a63a7a63fcf93801429e87dc94924c" Jul 07 19:03:37.413607 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.413561 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9e867a77df9e6aee0670b91ed40795b79a63a7a63fcf93801429e87dc94924c"} err="failed to get container status \"f9e867a77df9e6aee0670b91ed40795b79a63a7a63fcf93801429e87dc94924c\": rpc error: code = NotFound desc = could not find container \"f9e867a77df9e6aee0670b91ed40795b79a63a7a63fcf93801429e87dc94924c\": container with ID starting with f9e867a77df9e6aee0670b91ed40795b79a63a7a63fcf93801429e87dc94924c not found: ID does not exist" Jul 07 19:03:37.413607 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.413579 2563 scope.go:117] "RemoveContainer" containerID="642e2e3f887dc2826bc7aa92e29de4499b4e1e873cea8fbec1f1e28f2f0e2091" Jul 07 19:03:37.413792 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.413776 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"642e2e3f887dc2826bc7aa92e29de4499b4e1e873cea8fbec1f1e28f2f0e2091"} err="failed to get container status \"642e2e3f887dc2826bc7aa92e29de4499b4e1e873cea8fbec1f1e28f2f0e2091\": rpc error: code = NotFound desc = could not find container \"642e2e3f887dc2826bc7aa92e29de4499b4e1e873cea8fbec1f1e28f2f0e2091\": container with ID starting with 642e2e3f887dc2826bc7aa92e29de4499b4e1e873cea8fbec1f1e28f2f0e2091 not found: ID does not exist" Jul 07 19:03:37.413834 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.413793 2563 scope.go:117] "RemoveContainer" containerID="4ca654707d5d4fcc607ae59f52a684e43360f3946743a9e402e1261500880ac7" Jul 07 19:03:37.414011 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.413996 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ca654707d5d4fcc607ae59f52a684e43360f3946743a9e402e1261500880ac7"} err="failed to get container status \"4ca654707d5d4fcc607ae59f52a684e43360f3946743a9e402e1261500880ac7\": rpc error: code = NotFound desc = could not find container \"4ca654707d5d4fcc607ae59f52a684e43360f3946743a9e402e1261500880ac7\": container with ID starting with 4ca654707d5d4fcc607ae59f52a684e43360f3946743a9e402e1261500880ac7 not found: ID does not exist" Jul 07 19:03:37.414011 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.414010 2563 scope.go:117] "RemoveContainer" containerID="1472dd326ad564274c5e93960ed0abb12f890ff81d7d8a15c278498000684c10" Jul 07 19:03:37.414252 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.414237 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1472dd326ad564274c5e93960ed0abb12f890ff81d7d8a15c278498000684c10"} err="failed to get container status \"1472dd326ad564274c5e93960ed0abb12f890ff81d7d8a15c278498000684c10\": rpc error: code = NotFound desc = could not find container \"1472dd326ad564274c5e93960ed0abb12f890ff81d7d8a15c278498000684c10\": container with ID starting with 1472dd326ad564274c5e93960ed0abb12f890ff81d7d8a15c278498000684c10 not found: ID does not exist" Jul 07 19:03:37.414297 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.414253 2563 scope.go:117] "RemoveContainer" containerID="146645ac04a6265cbafa9a51132ccfed6688ca9d523c3adf9a759a145167a3b7" Jul 07 19:03:37.414446 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.414430 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"146645ac04a6265cbafa9a51132ccfed6688ca9d523c3adf9a759a145167a3b7"} err="failed to get container status \"146645ac04a6265cbafa9a51132ccfed6688ca9d523c3adf9a759a145167a3b7\": rpc error: code = NotFound desc = could not find container \"146645ac04a6265cbafa9a51132ccfed6688ca9d523c3adf9a759a145167a3b7\": container with ID starting with 146645ac04a6265cbafa9a51132ccfed6688ca9d523c3adf9a759a145167a3b7 not found: ID does not exist" Jul 07 19:03:37.414486 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.414446 2563 scope.go:117] "RemoveContainer" containerID="e6fe0065e5a888e34565995c53a516d658ec05c20c9c006207a83fa67796b50e" Jul 07 19:03:37.414654 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.414638 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6fe0065e5a888e34565995c53a516d658ec05c20c9c006207a83fa67796b50e"} err="failed to get container status \"e6fe0065e5a888e34565995c53a516d658ec05c20c9c006207a83fa67796b50e\": rpc error: code = NotFound desc = could not find container \"e6fe0065e5a888e34565995c53a516d658ec05c20c9c006207a83fa67796b50e\": container with ID starting with e6fe0065e5a888e34565995c53a516d658ec05c20c9c006207a83fa67796b50e not found: ID does not exist" Jul 07 19:03:37.414654 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.414655 2563 scope.go:117] "RemoveContainer" containerID="a46ae0c3beca66e600238a2bd178354d061b72d7fb5267ecdfb2d685cea43af0" Jul 07 19:03:37.414850 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.414828 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a46ae0c3beca66e600238a2bd178354d061b72d7fb5267ecdfb2d685cea43af0"} err="failed to get container status \"a46ae0c3beca66e600238a2bd178354d061b72d7fb5267ecdfb2d685cea43af0\": rpc error: code = NotFound desc = could not find container \"a46ae0c3beca66e600238a2bd178354d061b72d7fb5267ecdfb2d685cea43af0\": container with ID starting with a46ae0c3beca66e600238a2bd178354d061b72d7fb5267ecdfb2d685cea43af0 not found: ID does not exist" Jul 07 19:03:37.414887 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.414851 2563 scope.go:117] "RemoveContainer" containerID="f9e867a77df9e6aee0670b91ed40795b79a63a7a63fcf93801429e87dc94924c" Jul 07 19:03:37.415074 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.415059 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9e867a77df9e6aee0670b91ed40795b79a63a7a63fcf93801429e87dc94924c"} err="failed to get container status \"f9e867a77df9e6aee0670b91ed40795b79a63a7a63fcf93801429e87dc94924c\": rpc error: code = NotFound desc = could not find container \"f9e867a77df9e6aee0670b91ed40795b79a63a7a63fcf93801429e87dc94924c\": container with ID starting with f9e867a77df9e6aee0670b91ed40795b79a63a7a63fcf93801429e87dc94924c not found: ID does not exist" Jul 07 19:03:37.415113 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.415074 2563 scope.go:117] "RemoveContainer" containerID="642e2e3f887dc2826bc7aa92e29de4499b4e1e873cea8fbec1f1e28f2f0e2091" Jul 07 19:03:37.415303 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.415283 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"642e2e3f887dc2826bc7aa92e29de4499b4e1e873cea8fbec1f1e28f2f0e2091"} err="failed to get container status \"642e2e3f887dc2826bc7aa92e29de4499b4e1e873cea8fbec1f1e28f2f0e2091\": rpc error: code = NotFound desc = could not find container \"642e2e3f887dc2826bc7aa92e29de4499b4e1e873cea8fbec1f1e28f2f0e2091\": container with ID starting with 642e2e3f887dc2826bc7aa92e29de4499b4e1e873cea8fbec1f1e28f2f0e2091 not found: ID does not exist" Jul 07 19:03:37.415352 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.415302 2563 scope.go:117] "RemoveContainer" containerID="4ca654707d5d4fcc607ae59f52a684e43360f3946743a9e402e1261500880ac7" Jul 07 19:03:37.415524 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.415503 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ca654707d5d4fcc607ae59f52a684e43360f3946743a9e402e1261500880ac7"} err="failed to get container status \"4ca654707d5d4fcc607ae59f52a684e43360f3946743a9e402e1261500880ac7\": rpc error: code = NotFound desc = could not find container \"4ca654707d5d4fcc607ae59f52a684e43360f3946743a9e402e1261500880ac7\": container with ID starting with 4ca654707d5d4fcc607ae59f52a684e43360f3946743a9e402e1261500880ac7 not found: ID does not exist" Jul 07 19:03:37.415572 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.415528 2563 scope.go:117] "RemoveContainer" containerID="1472dd326ad564274c5e93960ed0abb12f890ff81d7d8a15c278498000684c10" Jul 07 19:03:37.415793 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.415773 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1472dd326ad564274c5e93960ed0abb12f890ff81d7d8a15c278498000684c10"} err="failed to get container status \"1472dd326ad564274c5e93960ed0abb12f890ff81d7d8a15c278498000684c10\": rpc error: code = NotFound desc = could not find container \"1472dd326ad564274c5e93960ed0abb12f890ff81d7d8a15c278498000684c10\": container with ID starting with 1472dd326ad564274c5e93960ed0abb12f890ff81d7d8a15c278498000684c10 not found: ID does not exist" Jul 07 19:03:37.415793 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.415792 2563 scope.go:117] "RemoveContainer" containerID="146645ac04a6265cbafa9a51132ccfed6688ca9d523c3adf9a759a145167a3b7" Jul 07 19:03:37.416039 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.416021 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"146645ac04a6265cbafa9a51132ccfed6688ca9d523c3adf9a759a145167a3b7"} err="failed to get container status \"146645ac04a6265cbafa9a51132ccfed6688ca9d523c3adf9a759a145167a3b7\": rpc error: code = NotFound desc = could not find container \"146645ac04a6265cbafa9a51132ccfed6688ca9d523c3adf9a759a145167a3b7\": container with ID starting with 146645ac04a6265cbafa9a51132ccfed6688ca9d523c3adf9a759a145167a3b7 not found: ID does not exist" Jul 07 19:03:37.416084 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.416039 2563 scope.go:117] "RemoveContainer" containerID="e6fe0065e5a888e34565995c53a516d658ec05c20c9c006207a83fa67796b50e" Jul 07 19:03:37.416251 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.416235 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6fe0065e5a888e34565995c53a516d658ec05c20c9c006207a83fa67796b50e"} err="failed to get container status \"e6fe0065e5a888e34565995c53a516d658ec05c20c9c006207a83fa67796b50e\": rpc error: code = NotFound desc = could not find container \"e6fe0065e5a888e34565995c53a516d658ec05c20c9c006207a83fa67796b50e\": container with ID starting with e6fe0065e5a888e34565995c53a516d658ec05c20c9c006207a83fa67796b50e not found: ID does not exist" Jul 07 19:03:37.416289 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.416251 2563 scope.go:117] "RemoveContainer" containerID="a46ae0c3beca66e600238a2bd178354d061b72d7fb5267ecdfb2d685cea43af0" Jul 07 19:03:37.416479 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.416464 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a46ae0c3beca66e600238a2bd178354d061b72d7fb5267ecdfb2d685cea43af0"} err="failed to get container status \"a46ae0c3beca66e600238a2bd178354d061b72d7fb5267ecdfb2d685cea43af0\": rpc error: code = NotFound desc = could not find container \"a46ae0c3beca66e600238a2bd178354d061b72d7fb5267ecdfb2d685cea43af0\": container with ID starting with a46ae0c3beca66e600238a2bd178354d061b72d7fb5267ecdfb2d685cea43af0 not found: ID does not exist" Jul 07 19:03:37.416536 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.416479 2563 scope.go:117] "RemoveContainer" containerID="f9e867a77df9e6aee0670b91ed40795b79a63a7a63fcf93801429e87dc94924c" Jul 07 19:03:37.416692 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.416676 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9e867a77df9e6aee0670b91ed40795b79a63a7a63fcf93801429e87dc94924c"} err="failed to get container status \"f9e867a77df9e6aee0670b91ed40795b79a63a7a63fcf93801429e87dc94924c\": rpc error: code = NotFound desc = could not find container \"f9e867a77df9e6aee0670b91ed40795b79a63a7a63fcf93801429e87dc94924c\": container with ID starting with f9e867a77df9e6aee0670b91ed40795b79a63a7a63fcf93801429e87dc94924c not found: ID does not exist" Jul 07 19:03:37.416736 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.416691 2563 scope.go:117] "RemoveContainer" containerID="642e2e3f887dc2826bc7aa92e29de4499b4e1e873cea8fbec1f1e28f2f0e2091" Jul 07 19:03:37.416899 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.416883 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"642e2e3f887dc2826bc7aa92e29de4499b4e1e873cea8fbec1f1e28f2f0e2091"} err="failed to get container status \"642e2e3f887dc2826bc7aa92e29de4499b4e1e873cea8fbec1f1e28f2f0e2091\": rpc error: code = NotFound desc = could not find container \"642e2e3f887dc2826bc7aa92e29de4499b4e1e873cea8fbec1f1e28f2f0e2091\": container with ID starting with 642e2e3f887dc2826bc7aa92e29de4499b4e1e873cea8fbec1f1e28f2f0e2091 not found: ID does not exist" Jul 07 19:03:37.416937 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.416899 2563 scope.go:117] "RemoveContainer" containerID="4ca654707d5d4fcc607ae59f52a684e43360f3946743a9e402e1261500880ac7" Jul 07 19:03:37.417156 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.417139 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ca654707d5d4fcc607ae59f52a684e43360f3946743a9e402e1261500880ac7"} err="failed to get container status \"4ca654707d5d4fcc607ae59f52a684e43360f3946743a9e402e1261500880ac7\": rpc error: code = NotFound desc = could not find container \"4ca654707d5d4fcc607ae59f52a684e43360f3946743a9e402e1261500880ac7\": container with ID starting with 4ca654707d5d4fcc607ae59f52a684e43360f3946743a9e402e1261500880ac7 not found: ID does not exist" Jul 07 19:03:37.417156 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.417156 2563 scope.go:117] "RemoveContainer" containerID="1472dd326ad564274c5e93960ed0abb12f890ff81d7d8a15c278498000684c10" Jul 07 19:03:37.417362 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.417346 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1472dd326ad564274c5e93960ed0abb12f890ff81d7d8a15c278498000684c10"} err="failed to get container status \"1472dd326ad564274c5e93960ed0abb12f890ff81d7d8a15c278498000684c10\": rpc error: code = NotFound desc = could not find container \"1472dd326ad564274c5e93960ed0abb12f890ff81d7d8a15c278498000684c10\": container with ID starting with 1472dd326ad564274c5e93960ed0abb12f890ff81d7d8a15c278498000684c10 not found: ID does not exist" Jul 07 19:03:37.417408 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.417362 2563 scope.go:117] "RemoveContainer" containerID="146645ac04a6265cbafa9a51132ccfed6688ca9d523c3adf9a759a145167a3b7" Jul 07 19:03:37.417559 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.417543 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"146645ac04a6265cbafa9a51132ccfed6688ca9d523c3adf9a759a145167a3b7"} err="failed to get container status \"146645ac04a6265cbafa9a51132ccfed6688ca9d523c3adf9a759a145167a3b7\": rpc error: code = NotFound desc = could not find container \"146645ac04a6265cbafa9a51132ccfed6688ca9d523c3adf9a759a145167a3b7\": container with ID starting with 146645ac04a6265cbafa9a51132ccfed6688ca9d523c3adf9a759a145167a3b7 not found: ID does not exist" Jul 07 19:03:37.417559 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.417559 2563 scope.go:117] "RemoveContainer" containerID="e6fe0065e5a888e34565995c53a516d658ec05c20c9c006207a83fa67796b50e" Jul 07 19:03:37.417773 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.417758 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6fe0065e5a888e34565995c53a516d658ec05c20c9c006207a83fa67796b50e"} err="failed to get container status \"e6fe0065e5a888e34565995c53a516d658ec05c20c9c006207a83fa67796b50e\": rpc error: code = NotFound desc = could not find container \"e6fe0065e5a888e34565995c53a516d658ec05c20c9c006207a83fa67796b50e\": container with ID starting with e6fe0065e5a888e34565995c53a516d658ec05c20c9c006207a83fa67796b50e not found: ID does not exist" Jul 07 19:03:37.417824 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.417774 2563 scope.go:117] "RemoveContainer" containerID="a46ae0c3beca66e600238a2bd178354d061b72d7fb5267ecdfb2d685cea43af0" Jul 07 19:03:37.418019 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.418002 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a46ae0c3beca66e600238a2bd178354d061b72d7fb5267ecdfb2d685cea43af0"} err="failed to get container status \"a46ae0c3beca66e600238a2bd178354d061b72d7fb5267ecdfb2d685cea43af0\": rpc error: code = NotFound desc = could not find container \"a46ae0c3beca66e600238a2bd178354d061b72d7fb5267ecdfb2d685cea43af0\": container with ID starting with a46ae0c3beca66e600238a2bd178354d061b72d7fb5267ecdfb2d685cea43af0 not found: ID does not exist" Jul 07 19:03:37.418071 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.418021 2563 scope.go:117] "RemoveContainer" containerID="f9e867a77df9e6aee0670b91ed40795b79a63a7a63fcf93801429e87dc94924c" Jul 07 19:03:37.418252 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.418232 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9e867a77df9e6aee0670b91ed40795b79a63a7a63fcf93801429e87dc94924c"} err="failed to get container status \"f9e867a77df9e6aee0670b91ed40795b79a63a7a63fcf93801429e87dc94924c\": rpc error: code = NotFound desc = could not find container \"f9e867a77df9e6aee0670b91ed40795b79a63a7a63fcf93801429e87dc94924c\": container with ID starting with f9e867a77df9e6aee0670b91ed40795b79a63a7a63fcf93801429e87dc94924c not found: ID does not exist" Jul 07 19:03:37.418252 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.418251 2563 scope.go:117] "RemoveContainer" containerID="642e2e3f887dc2826bc7aa92e29de4499b4e1e873cea8fbec1f1e28f2f0e2091" Jul 07 19:03:37.418453 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.418438 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"642e2e3f887dc2826bc7aa92e29de4499b4e1e873cea8fbec1f1e28f2f0e2091"} err="failed to get container status \"642e2e3f887dc2826bc7aa92e29de4499b4e1e873cea8fbec1f1e28f2f0e2091\": rpc error: code = NotFound desc = could not find container \"642e2e3f887dc2826bc7aa92e29de4499b4e1e873cea8fbec1f1e28f2f0e2091\": container with ID starting with 642e2e3f887dc2826bc7aa92e29de4499b4e1e873cea8fbec1f1e28f2f0e2091 not found: ID does not exist" Jul 07 19:03:37.418453 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.418452 2563 scope.go:117] "RemoveContainer" containerID="4ca654707d5d4fcc607ae59f52a684e43360f3946743a9e402e1261500880ac7" Jul 07 19:03:37.418672 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.418656 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ca654707d5d4fcc607ae59f52a684e43360f3946743a9e402e1261500880ac7"} err="failed to get container status \"4ca654707d5d4fcc607ae59f52a684e43360f3946743a9e402e1261500880ac7\": rpc error: code = NotFound desc = could not find container \"4ca654707d5d4fcc607ae59f52a684e43360f3946743a9e402e1261500880ac7\": container with ID starting with 4ca654707d5d4fcc607ae59f52a684e43360f3946743a9e402e1261500880ac7 not found: ID does not exist" Jul 07 19:03:37.418718 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.418672 2563 scope.go:117] "RemoveContainer" containerID="1472dd326ad564274c5e93960ed0abb12f890ff81d7d8a15c278498000684c10" Jul 07 19:03:37.418884 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.418859 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1472dd326ad564274c5e93960ed0abb12f890ff81d7d8a15c278498000684c10"} err="failed to get container status \"1472dd326ad564274c5e93960ed0abb12f890ff81d7d8a15c278498000684c10\": rpc error: code = NotFound desc = could not find container \"1472dd326ad564274c5e93960ed0abb12f890ff81d7d8a15c278498000684c10\": container with ID starting with 1472dd326ad564274c5e93960ed0abb12f890ff81d7d8a15c278498000684c10 not found: ID does not exist" Jul 07 19:03:37.418884 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.418879 2563 scope.go:117] "RemoveContainer" containerID="146645ac04a6265cbafa9a51132ccfed6688ca9d523c3adf9a759a145167a3b7" Jul 07 19:03:37.419112 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.419092 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"146645ac04a6265cbafa9a51132ccfed6688ca9d523c3adf9a759a145167a3b7"} err="failed to get container status \"146645ac04a6265cbafa9a51132ccfed6688ca9d523c3adf9a759a145167a3b7\": rpc error: code = NotFound desc = could not find container \"146645ac04a6265cbafa9a51132ccfed6688ca9d523c3adf9a759a145167a3b7\": container with ID starting with 146645ac04a6265cbafa9a51132ccfed6688ca9d523c3adf9a759a145167a3b7 not found: ID does not exist" Jul 07 19:03:37.419157 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.419112 2563 scope.go:117] "RemoveContainer" containerID="e6fe0065e5a888e34565995c53a516d658ec05c20c9c006207a83fa67796b50e" Jul 07 19:03:37.419323 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.419306 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6fe0065e5a888e34565995c53a516d658ec05c20c9c006207a83fa67796b50e"} err="failed to get container status \"e6fe0065e5a888e34565995c53a516d658ec05c20c9c006207a83fa67796b50e\": rpc error: code = NotFound desc = could not find container \"e6fe0065e5a888e34565995c53a516d658ec05c20c9c006207a83fa67796b50e\": container with ID starting with e6fe0065e5a888e34565995c53a516d658ec05c20c9c006207a83fa67796b50e not found: ID does not exist" Jul 07 19:03:37.419360 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.419323 2563 scope.go:117] "RemoveContainer" containerID="a46ae0c3beca66e600238a2bd178354d061b72d7fb5267ecdfb2d685cea43af0" Jul 07 19:03:37.419530 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.419512 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a46ae0c3beca66e600238a2bd178354d061b72d7fb5267ecdfb2d685cea43af0"} err="failed to get container status \"a46ae0c3beca66e600238a2bd178354d061b72d7fb5267ecdfb2d685cea43af0\": rpc error: code = NotFound desc = could not find container \"a46ae0c3beca66e600238a2bd178354d061b72d7fb5267ecdfb2d685cea43af0\": container with ID starting with a46ae0c3beca66e600238a2bd178354d061b72d7fb5267ecdfb2d685cea43af0 not found: ID does not exist" Jul 07 19:03:37.419594 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.419530 2563 scope.go:117] "RemoveContainer" containerID="f9e867a77df9e6aee0670b91ed40795b79a63a7a63fcf93801429e87dc94924c" Jul 07 19:03:37.419724 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.419706 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9e867a77df9e6aee0670b91ed40795b79a63a7a63fcf93801429e87dc94924c"} err="failed to get container status \"f9e867a77df9e6aee0670b91ed40795b79a63a7a63fcf93801429e87dc94924c\": rpc error: code = NotFound desc = could not find container \"f9e867a77df9e6aee0670b91ed40795b79a63a7a63fcf93801429e87dc94924c\": container with ID starting with f9e867a77df9e6aee0670b91ed40795b79a63a7a63fcf93801429e87dc94924c not found: ID does not exist" Jul 07 19:03:37.419724 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.419724 2563 scope.go:117] "RemoveContainer" containerID="642e2e3f887dc2826bc7aa92e29de4499b4e1e873cea8fbec1f1e28f2f0e2091" Jul 07 19:03:37.419915 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.419897 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"642e2e3f887dc2826bc7aa92e29de4499b4e1e873cea8fbec1f1e28f2f0e2091"} err="failed to get container status \"642e2e3f887dc2826bc7aa92e29de4499b4e1e873cea8fbec1f1e28f2f0e2091\": rpc error: code = NotFound desc = could not find container \"642e2e3f887dc2826bc7aa92e29de4499b4e1e873cea8fbec1f1e28f2f0e2091\": container with ID starting with 642e2e3f887dc2826bc7aa92e29de4499b4e1e873cea8fbec1f1e28f2f0e2091 not found: ID does not exist" Jul 07 19:03:37.419991 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.419915 2563 scope.go:117] "RemoveContainer" containerID="4ca654707d5d4fcc607ae59f52a684e43360f3946743a9e402e1261500880ac7" Jul 07 19:03:37.420141 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.420125 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ca654707d5d4fcc607ae59f52a684e43360f3946743a9e402e1261500880ac7"} err="failed to get container status \"4ca654707d5d4fcc607ae59f52a684e43360f3946743a9e402e1261500880ac7\": rpc error: code = NotFound desc = could not find container \"4ca654707d5d4fcc607ae59f52a684e43360f3946743a9e402e1261500880ac7\": container with ID starting with 4ca654707d5d4fcc607ae59f52a684e43360f3946743a9e402e1261500880ac7 not found: ID does not exist" Jul 07 19:03:37.420141 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.420140 2563 scope.go:117] "RemoveContainer" containerID="1472dd326ad564274c5e93960ed0abb12f890ff81d7d8a15c278498000684c10" Jul 07 19:03:37.420350 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.420332 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1472dd326ad564274c5e93960ed0abb12f890ff81d7d8a15c278498000684c10"} err="failed to get container status \"1472dd326ad564274c5e93960ed0abb12f890ff81d7d8a15c278498000684c10\": rpc error: code = NotFound desc = could not find container \"1472dd326ad564274c5e93960ed0abb12f890ff81d7d8a15c278498000684c10\": container with ID starting with 1472dd326ad564274c5e93960ed0abb12f890ff81d7d8a15c278498000684c10 not found: ID does not exist" Jul 07 19:03:37.420401 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.420350 2563 scope.go:117] "RemoveContainer" containerID="146645ac04a6265cbafa9a51132ccfed6688ca9d523c3adf9a759a145167a3b7" Jul 07 19:03:37.420580 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.420560 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"146645ac04a6265cbafa9a51132ccfed6688ca9d523c3adf9a759a145167a3b7"} err="failed to get container status \"146645ac04a6265cbafa9a51132ccfed6688ca9d523c3adf9a759a145167a3b7\": rpc error: code = NotFound desc = could not find container \"146645ac04a6265cbafa9a51132ccfed6688ca9d523c3adf9a759a145167a3b7\": container with ID starting with 146645ac04a6265cbafa9a51132ccfed6688ca9d523c3adf9a759a145167a3b7 not found: ID does not exist" Jul 07 19:03:37.420580 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.420579 2563 scope.go:117] "RemoveContainer" containerID="e6fe0065e5a888e34565995c53a516d658ec05c20c9c006207a83fa67796b50e" Jul 07 19:03:37.420813 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.420784 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6fe0065e5a888e34565995c53a516d658ec05c20c9c006207a83fa67796b50e"} err="failed to get container status \"e6fe0065e5a888e34565995c53a516d658ec05c20c9c006207a83fa67796b50e\": rpc error: code = NotFound desc = could not find container \"e6fe0065e5a888e34565995c53a516d658ec05c20c9c006207a83fa67796b50e\": container with ID starting with e6fe0065e5a888e34565995c53a516d658ec05c20c9c006207a83fa67796b50e not found: ID does not exist" Jul 07 19:03:37.420813 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.420814 2563 scope.go:117] "RemoveContainer" containerID="a46ae0c3beca66e600238a2bd178354d061b72d7fb5267ecdfb2d685cea43af0" Jul 07 19:03:37.421101 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.421082 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a46ae0c3beca66e600238a2bd178354d061b72d7fb5267ecdfb2d685cea43af0"} err="failed to get container status \"a46ae0c3beca66e600238a2bd178354d061b72d7fb5267ecdfb2d685cea43af0\": rpc error: code = NotFound desc = could not find container \"a46ae0c3beca66e600238a2bd178354d061b72d7fb5267ecdfb2d685cea43af0\": container with ID starting with a46ae0c3beca66e600238a2bd178354d061b72d7fb5267ecdfb2d685cea43af0 not found: ID does not exist" Jul 07 19:03:37.421171 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.421101 2563 scope.go:117] "RemoveContainer" containerID="f9e867a77df9e6aee0670b91ed40795b79a63a7a63fcf93801429e87dc94924c" Jul 07 19:03:37.421309 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.421289 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9e867a77df9e6aee0670b91ed40795b79a63a7a63fcf93801429e87dc94924c"} err="failed to get container status \"f9e867a77df9e6aee0670b91ed40795b79a63a7a63fcf93801429e87dc94924c\": rpc error: code = NotFound desc = could not find container \"f9e867a77df9e6aee0670b91ed40795b79a63a7a63fcf93801429e87dc94924c\": container with ID starting with f9e867a77df9e6aee0670b91ed40795b79a63a7a63fcf93801429e87dc94924c not found: ID does not exist" Jul 07 19:03:37.485482 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.485400 2563 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d27c9f8c-0884-467d-ae62-9193d6950191-secret-kube-rbac-proxy\") on node \"ip-10-0-142-134.ec2.internal\" DevicePath \"\"" Jul 07 19:03:37.485482 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.485431 2563 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/d27c9f8c-0884-467d-ae62-9193d6950191-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-142-134.ec2.internal\" DevicePath \"\"" Jul 07 19:03:37.485482 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.485450 2563 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d27c9f8c-0884-467d-ae62-9193d6950191-configmap-metrics-client-ca\") on node \"ip-10-0-142-134.ec2.internal\" DevicePath \"\"" Jul 07 19:03:37.485482 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.485465 2563 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d27c9f8c-0884-467d-ae62-9193d6950191-web-config\") on node \"ip-10-0-142-134.ec2.internal\" DevicePath \"\"" Jul 07 19:03:37.485482 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.485482 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rgtvz\" (UniqueName: \"kubernetes.io/projected/d27c9f8c-0884-467d-ae62-9193d6950191-kube-api-access-rgtvz\") on node \"ip-10-0-142-134.ec2.internal\" DevicePath \"\"" Jul 07 19:03:37.485826 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.485496 2563 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d27c9f8c-0884-467d-ae62-9193d6950191-config-out\") on node \"ip-10-0-142-134.ec2.internal\" DevicePath \"\"" Jul 07 19:03:37.485826 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.485524 2563 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/d27c9f8c-0884-467d-ae62-9193d6950191-secret-prometheus-k8s-tls\") on node \"ip-10-0-142-134.ec2.internal\" DevicePath \"\"" Jul 07 19:03:37.485826 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.485544 2563 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/d27c9f8c-0884-467d-ae62-9193d6950191-thanos-prometheus-http-client-file\") on node \"ip-10-0-142-134.ec2.internal\" DevicePath \"\"" Jul 07 19:03:37.485826 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.485558 2563 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d27c9f8c-0884-467d-ae62-9193d6950191-secret-grpc-tls\") on node \"ip-10-0-142-134.ec2.internal\" DevicePath \"\"" Jul 07 19:03:37.485826 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.485572 2563 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/d27c9f8c-0884-467d-ae62-9193d6950191-secret-metrics-client-certs\") on node \"ip-10-0-142-134.ec2.internal\" DevicePath \"\"" Jul 07 19:03:37.485826 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.485586 2563 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/d27c9f8c-0884-467d-ae62-9193d6950191-prometheus-k8s-db\") on node \"ip-10-0-142-134.ec2.internal\" DevicePath \"\"" Jul 07 19:03:37.485826 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.485601 2563 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d27c9f8c-0884-467d-ae62-9193d6950191-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-142-134.ec2.internal\" DevicePath \"\"" Jul 07 19:03:37.485826 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.485617 2563 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/d27c9f8c-0884-467d-ae62-9193d6950191-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-142-134.ec2.internal\" DevicePath \"\"" Jul 07 19:03:37.485826 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.485633 2563 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/d27c9f8c-0884-467d-ae62-9193d6950191-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-142-134.ec2.internal\" DevicePath \"\"" Jul 07 19:03:37.485826 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.485660 2563 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/d27c9f8c-0884-467d-ae62-9193d6950191-config\") on node \"ip-10-0-142-134.ec2.internal\" DevicePath \"\"" Jul 07 19:03:37.485826 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.485684 2563 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d27c9f8c-0884-467d-ae62-9193d6950191-prometheus-trusted-ca-bundle\") on node \"ip-10-0-142-134.ec2.internal\" DevicePath \"\"" Jul 07 19:03:37.485826 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.485701 2563 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d27c9f8c-0884-467d-ae62-9193d6950191-tls-assets\") on node \"ip-10-0-142-134.ec2.internal\" DevicePath \"\"" Jul 07 19:03:37.485826 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.485716 2563 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d27c9f8c-0884-467d-ae62-9193d6950191-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-142-134.ec2.internal\" DevicePath \"\"" Jul 07 19:03:37.485826 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.485729 2563 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/d27c9f8c-0884-467d-ae62-9193d6950191-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-142-134.ec2.internal\" DevicePath \"\"" Jul 07 19:03:37.680917 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.680885 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 07 19:03:37.686924 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.686894 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 07 19:03:37.731101 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.731065 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 07 19:03:37.731377 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.731364 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d27c9f8c-0884-467d-ae62-9193d6950191" containerName="kube-rbac-proxy-thanos" Jul 07 19:03:37.731434 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.731379 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="d27c9f8c-0884-467d-ae62-9193d6950191" containerName="kube-rbac-proxy-thanos" Jul 07 19:03:37.731434 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.731391 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d27c9f8c-0884-467d-ae62-9193d6950191" containerName="kube-rbac-proxy-web" Jul 07 19:03:37.731434 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.731398 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="d27c9f8c-0884-467d-ae62-9193d6950191" containerName="kube-rbac-proxy-web" Jul 07 19:03:37.731434 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.731405 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d27c9f8c-0884-467d-ae62-9193d6950191" containerName="config-reloader" Jul 07 19:03:37.731434 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.731410 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="d27c9f8c-0884-467d-ae62-9193d6950191" containerName="config-reloader" Jul 07 19:03:37.731434 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.731421 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d27c9f8c-0884-467d-ae62-9193d6950191" containerName="kube-rbac-proxy" Jul 07 19:03:37.731434 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.731430 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="d27c9f8c-0884-467d-ae62-9193d6950191" containerName="kube-rbac-proxy" Jul 07 19:03:37.731624 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.731441 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d27c9f8c-0884-467d-ae62-9193d6950191" containerName="thanos-sidecar" Jul 07 19:03:37.731624 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.731446 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="d27c9f8c-0884-467d-ae62-9193d6950191" containerName="thanos-sidecar" Jul 07 19:03:37.731624 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.731454 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d27c9f8c-0884-467d-ae62-9193d6950191" containerName="init-config-reloader" Jul 07 19:03:37.731624 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.731459 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="d27c9f8c-0884-467d-ae62-9193d6950191" containerName="init-config-reloader" Jul 07 19:03:37.731624 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.731465 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d27c9f8c-0884-467d-ae62-9193d6950191" containerName="prometheus" Jul 07 19:03:37.731624 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.731470 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="d27c9f8c-0884-467d-ae62-9193d6950191" containerName="prometheus" Jul 07 19:03:37.731624 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.731522 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="d27c9f8c-0884-467d-ae62-9193d6950191" containerName="config-reloader" Jul 07 19:03:37.731624 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.731529 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="d27c9f8c-0884-467d-ae62-9193d6950191" containerName="kube-rbac-proxy" Jul 07 19:03:37.731624 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.731535 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="d27c9f8c-0884-467d-ae62-9193d6950191" containerName="thanos-sidecar" Jul 07 19:03:37.731624 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.731540 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="d27c9f8c-0884-467d-ae62-9193d6950191" containerName="prometheus" Jul 07 19:03:37.731624 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.731547 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="d27c9f8c-0884-467d-ae62-9193d6950191" containerName="kube-rbac-proxy-web" Jul 07 19:03:37.731624 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.731552 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="d27c9f8c-0884-467d-ae62-9193d6950191" containerName="kube-rbac-proxy-thanos" Jul 07 19:03:37.738676 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.737070 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:03:37.740427 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.740406 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 07 19:03:37.740542 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.740465 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 07 19:03:37.742078 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.742059 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 07 19:03:37.742188 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.742113 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 07 19:03:37.742388 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.742374 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-d9kr8\"" Jul 07 19:03:37.742388 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.742384 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-d3n2opql6gns4\"" Jul 07 19:03:37.747209 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.747193 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 07 19:03:37.747386 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.747369 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 07 19:03:37.754074 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.754054 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 07 19:03:37.754170 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.754080 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 07 19:03:37.754170 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.754097 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 07 19:03:37.754170 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.754087 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 07 19:03:37.764397 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.764373 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 07 19:03:37.764510 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.764421 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 07 19:03:37.765425 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.765409 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 07 19:03:37.784245 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.784216 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 07 19:03:37.878135 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.878104 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 07 19:03:37.889501 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.889469 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/fe61befe-7a98-40cf-b142-a435f8804125-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"fe61befe-7a98-40cf-b142-a435f8804125\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:03:37.889674 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.889517 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fe61befe-7a98-40cf-b142-a435f8804125-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"fe61befe-7a98-40cf-b142-a435f8804125\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:03:37.889674 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.889579 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fe61befe-7a98-40cf-b142-a435f8804125-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fe61befe-7a98-40cf-b142-a435f8804125\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:03:37.889674 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.889626 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wv4tv\" (UniqueName: \"kubernetes.io/projected/fe61befe-7a98-40cf-b142-a435f8804125-kube-api-access-wv4tv\") pod \"prometheus-k8s-0\" (UID: \"fe61befe-7a98-40cf-b142-a435f8804125\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:03:37.889860 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.889693 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/fe61befe-7a98-40cf-b142-a435f8804125-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"fe61befe-7a98-40cf-b142-a435f8804125\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:03:37.889860 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.889776 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fe61befe-7a98-40cf-b142-a435f8804125-config-out\") pod \"prometheus-k8s-0\" (UID: \"fe61befe-7a98-40cf-b142-a435f8804125\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:03:37.889860 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.889811 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/fe61befe-7a98-40cf-b142-a435f8804125-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"fe61befe-7a98-40cf-b142-a435f8804125\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:03:37.890053 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.889861 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/fe61befe-7a98-40cf-b142-a435f8804125-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"fe61befe-7a98-40cf-b142-a435f8804125\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:03:37.890053 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.889894 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fe61befe-7a98-40cf-b142-a435f8804125-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fe61befe-7a98-40cf-b142-a435f8804125\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:03:37.890053 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.889929 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/fe61befe-7a98-40cf-b142-a435f8804125-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"fe61befe-7a98-40cf-b142-a435f8804125\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:03:37.890053 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.889976 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/fe61befe-7a98-40cf-b142-a435f8804125-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"fe61befe-7a98-40cf-b142-a435f8804125\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:03:37.890053 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.890003 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/fe61befe-7a98-40cf-b142-a435f8804125-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"fe61befe-7a98-40cf-b142-a435f8804125\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:03:37.890296 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.890054 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/fe61befe-7a98-40cf-b142-a435f8804125-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"fe61befe-7a98-40cf-b142-a435f8804125\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:03:37.890296 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.890094 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fe61befe-7a98-40cf-b142-a435f8804125-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fe61befe-7a98-40cf-b142-a435f8804125\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:03:37.890296 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.890113 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fe61befe-7a98-40cf-b142-a435f8804125-web-config\") pod \"prometheus-k8s-0\" (UID: \"fe61befe-7a98-40cf-b142-a435f8804125\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:03:37.890296 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.890135 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fe61befe-7a98-40cf-b142-a435f8804125-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"fe61befe-7a98-40cf-b142-a435f8804125\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:03:37.890296 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.890157 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fe61befe-7a98-40cf-b142-a435f8804125-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"fe61befe-7a98-40cf-b142-a435f8804125\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:03:37.890296 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.890181 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/fe61befe-7a98-40cf-b142-a435f8804125-config\") pod \"prometheus-k8s-0\" (UID: \"fe61befe-7a98-40cf-b142-a435f8804125\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:03:37.890296 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.890203 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fe61befe-7a98-40cf-b142-a435f8804125-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"fe61befe-7a98-40cf-b142-a435f8804125\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:03:37.890296 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.890225 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/fe61befe-7a98-40cf-b142-a435f8804125-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"fe61befe-7a98-40cf-b142-a435f8804125\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:03:37.990685 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.990602 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/fe61befe-7a98-40cf-b142-a435f8804125-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"fe61befe-7a98-40cf-b142-a435f8804125\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:03:37.990685 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.990633 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fe61befe-7a98-40cf-b142-a435f8804125-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"fe61befe-7a98-40cf-b142-a435f8804125\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:03:37.990685 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.990654 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fe61befe-7a98-40cf-b142-a435f8804125-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fe61befe-7a98-40cf-b142-a435f8804125\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:03:37.990685 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.990671 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wv4tv\" (UniqueName: \"kubernetes.io/projected/fe61befe-7a98-40cf-b142-a435f8804125-kube-api-access-wv4tv\") pod \"prometheus-k8s-0\" (UID: \"fe61befe-7a98-40cf-b142-a435f8804125\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:03:37.991055 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.990696 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/fe61befe-7a98-40cf-b142-a435f8804125-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"fe61befe-7a98-40cf-b142-a435f8804125\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:03:37.991055 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.990739 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fe61befe-7a98-40cf-b142-a435f8804125-config-out\") pod \"prometheus-k8s-0\" (UID: \"fe61befe-7a98-40cf-b142-a435f8804125\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:03:37.991055 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.990764 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/fe61befe-7a98-40cf-b142-a435f8804125-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"fe61befe-7a98-40cf-b142-a435f8804125\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:03:37.991055 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.990797 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/fe61befe-7a98-40cf-b142-a435f8804125-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"fe61befe-7a98-40cf-b142-a435f8804125\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:03:37.991055 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.990820 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fe61befe-7a98-40cf-b142-a435f8804125-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fe61befe-7a98-40cf-b142-a435f8804125\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:03:37.991055 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.990860 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/fe61befe-7a98-40cf-b142-a435f8804125-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"fe61befe-7a98-40cf-b142-a435f8804125\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:03:37.991055 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.990886 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/fe61befe-7a98-40cf-b142-a435f8804125-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"fe61befe-7a98-40cf-b142-a435f8804125\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:03:37.991055 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.990913 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/fe61befe-7a98-40cf-b142-a435f8804125-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"fe61befe-7a98-40cf-b142-a435f8804125\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:03:37.991055 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.990970 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/fe61befe-7a98-40cf-b142-a435f8804125-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"fe61befe-7a98-40cf-b142-a435f8804125\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:03:37.991055 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.991009 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fe61befe-7a98-40cf-b142-a435f8804125-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fe61befe-7a98-40cf-b142-a435f8804125\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:03:37.991055 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.991032 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fe61befe-7a98-40cf-b142-a435f8804125-web-config\") pod \"prometheus-k8s-0\" (UID: \"fe61befe-7a98-40cf-b142-a435f8804125\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:03:37.991561 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.991059 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fe61befe-7a98-40cf-b142-a435f8804125-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"fe61befe-7a98-40cf-b142-a435f8804125\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:03:37.991561 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.991092 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fe61befe-7a98-40cf-b142-a435f8804125-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"fe61befe-7a98-40cf-b142-a435f8804125\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:03:37.991561 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.991120 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/fe61befe-7a98-40cf-b142-a435f8804125-config\") pod \"prometheus-k8s-0\" (UID: \"fe61befe-7a98-40cf-b142-a435f8804125\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:03:37.991561 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.991176 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fe61befe-7a98-40cf-b142-a435f8804125-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"fe61befe-7a98-40cf-b142-a435f8804125\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:03:37.991561 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.991216 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/fe61befe-7a98-40cf-b142-a435f8804125-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"fe61befe-7a98-40cf-b142-a435f8804125\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:03:37.991561 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.991287 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/fe61befe-7a98-40cf-b142-a435f8804125-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"fe61befe-7a98-40cf-b142-a435f8804125\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:03:37.991561 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.991440 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fe61befe-7a98-40cf-b142-a435f8804125-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fe61befe-7a98-40cf-b142-a435f8804125\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:03:37.992285 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.992258 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fe61befe-7a98-40cf-b142-a435f8804125-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fe61befe-7a98-40cf-b142-a435f8804125\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:03:37.992590 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.992568 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/fe61befe-7a98-40cf-b142-a435f8804125-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"fe61befe-7a98-40cf-b142-a435f8804125\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:03:37.993436 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.993409 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fe61befe-7a98-40cf-b142-a435f8804125-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"fe61befe-7a98-40cf-b142-a435f8804125\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:03:37.994136 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.993814 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fe61befe-7a98-40cf-b142-a435f8804125-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"fe61befe-7a98-40cf-b142-a435f8804125\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:03:37.994136 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.994061 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fe61befe-7a98-40cf-b142-a435f8804125-config-out\") pod \"prometheus-k8s-0\" (UID: \"fe61befe-7a98-40cf-b142-a435f8804125\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:03:37.994136 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.994111 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fe61befe-7a98-40cf-b142-a435f8804125-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fe61befe-7a98-40cf-b142-a435f8804125\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:03:37.994675 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.994419 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/fe61befe-7a98-40cf-b142-a435f8804125-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"fe61befe-7a98-40cf-b142-a435f8804125\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:03:37.994778 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.994710 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fe61befe-7a98-40cf-b142-a435f8804125-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"fe61befe-7a98-40cf-b142-a435f8804125\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:03:37.995249 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.994968 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/fe61befe-7a98-40cf-b142-a435f8804125-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"fe61befe-7a98-40cf-b142-a435f8804125\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:03:37.995249 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.995121 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/fe61befe-7a98-40cf-b142-a435f8804125-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"fe61befe-7a98-40cf-b142-a435f8804125\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:03:37.995886 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.995586 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/fe61befe-7a98-40cf-b142-a435f8804125-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"fe61befe-7a98-40cf-b142-a435f8804125\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:03:37.996570 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.996541 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/fe61befe-7a98-40cf-b142-a435f8804125-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"fe61befe-7a98-40cf-b142-a435f8804125\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:03:37.997338 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.997319 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fe61befe-7a98-40cf-b142-a435f8804125-web-config\") pod \"prometheus-k8s-0\" (UID: \"fe61befe-7a98-40cf-b142-a435f8804125\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:03:37.997499 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.997478 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/fe61befe-7a98-40cf-b142-a435f8804125-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"fe61befe-7a98-40cf-b142-a435f8804125\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:03:37.997692 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.997672 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fe61befe-7a98-40cf-b142-a435f8804125-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"fe61befe-7a98-40cf-b142-a435f8804125\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:03:37.997920 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.997896 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/fe61befe-7a98-40cf-b142-a435f8804125-config\") pod \"prometheus-k8s-0\" (UID: \"fe61befe-7a98-40cf-b142-a435f8804125\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:03:37.998091 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:37.998074 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/fe61befe-7a98-40cf-b142-a435f8804125-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"fe61befe-7a98-40cf-b142-a435f8804125\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:03:38.015689 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:38.015639 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wv4tv\" (UniqueName: \"kubernetes.io/projected/fe61befe-7a98-40cf-b142-a435f8804125-kube-api-access-wv4tv\") pod \"prometheus-k8s-0\" (UID: \"fe61befe-7a98-40cf-b142-a435f8804125\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:03:38.047857 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:38.047821 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:03:38.208892 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:38.208856 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 07 19:03:38.212549 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:03:38.212522 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe61befe_7a98_40cf_b142_a435f8804125.slice/crio-dad792c43b252fc75770e9ed4de0f90247160d47913ea9ef94ad5f939c6944e5 WatchSource:0}: Error finding container dad792c43b252fc75770e9ed4de0f90247160d47913ea9ef94ad5f939c6944e5: Status 404 returned error can't find the container with id dad792c43b252fc75770e9ed4de0f90247160d47913ea9ef94ad5f939c6944e5 Jul 07 19:03:38.360035 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:38.359999 2563 generic.go:356] "Generic (PLEG): container finished" podID="fe61befe-7a98-40cf-b142-a435f8804125" containerID="4b94d205613d589b2dd4d0c5b2364f33dfa5dfa36f2798eb22c8c4d268cc263e" exitCode=0 Jul 07 19:03:38.360503 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:38.360079 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fe61befe-7a98-40cf-b142-a435f8804125","Type":"ContainerDied","Data":"4b94d205613d589b2dd4d0c5b2364f33dfa5dfa36f2798eb22c8c4d268cc263e"} Jul 07 19:03:38.360503 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:38.360176 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fe61befe-7a98-40cf-b142-a435f8804125","Type":"ContainerStarted","Data":"dad792c43b252fc75770e9ed4de0f90247160d47913ea9ef94ad5f939c6944e5"} Jul 07 19:03:39.370651 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:39.370623 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fe61befe-7a98-40cf-b142-a435f8804125","Type":"ContainerStarted","Data":"0d4fb05d72517519b4747ed6897aa051d87af75652dfe00149e218d27d6adc6e"} Jul 07 19:03:39.370999 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:39.370684 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fe61befe-7a98-40cf-b142-a435f8804125","Type":"ContainerStarted","Data":"20046f1538259f7d07f2c94f91f907b76a598a0ed06bf67bfa7b4fffaee94094"} Jul 07 19:03:39.370999 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:39.370742 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fe61befe-7a98-40cf-b142-a435f8804125","Type":"ContainerStarted","Data":"85117f3294f199e72ce88194d40bf5bd630495a88f42aeea41c7dbf05734a0c7"} Jul 07 19:03:39.370999 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:39.370767 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fe61befe-7a98-40cf-b142-a435f8804125","Type":"ContainerStarted","Data":"49a6247aa402d44e909823ccd2bf1ea7fd36251e1604f1bcb6e7ff3d45aa54b1"} Jul 07 19:03:39.370999 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:39.370791 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fe61befe-7a98-40cf-b142-a435f8804125","Type":"ContainerStarted","Data":"1792cd90506013c68aada247152c803cba520ac924c23c230b447b678409a347"} Jul 07 19:03:39.370999 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:39.370833 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fe61befe-7a98-40cf-b142-a435f8804125","Type":"ContainerStarted","Data":"5a1ce0320dd68640a5d769cbc4ea47984dc74071846bb53fc3d7c55c0ddce6f1"} Jul 07 19:03:39.414708 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:39.414642 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.389177186 podStartE2EDuration="2.41462347s" totalImagesPullingTime="0s" totalInitContainerRuntime="25.446284ms" isStatefulPod=false podCreationTimestamp="2026-07-07 19:03:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:03:39.413472737 +0000 UTC m=+202.429476921" watchObservedRunningTime="2026-07-07 19:03:39.41462347 +0000 UTC m=+202.430627650" Jul 07 19:03:39.477358 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:39.477327 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d27c9f8c-0884-467d-ae62-9193d6950191" path="/var/lib/kubelet/pods/d27c9f8c-0884-467d-ae62-9193d6950191/volumes" Jul 07 19:03:40.379564 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:40.379531 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5d4848bbfb-w9dk4" event={"ID":"b43c0ae2-9e13-4beb-a30b-e2147f3d4e41","Type":"ContainerStarted","Data":"a7723743e2ede0dbcef6a9a6f3a098b6eb860fd4806f0c7ba927a641c8b9ac90"} Jul 07 19:03:40.380049 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:40.379587 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5d4848bbfb-w9dk4" event={"ID":"b43c0ae2-9e13-4beb-a30b-e2147f3d4e41","Type":"ContainerStarted","Data":"74ae86d67037a0993171c20a66d8a1fd3f6e114ff1c9888f58c3e0f613f77e13"} Jul 07 19:03:40.380049 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:40.379607 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5d4848bbfb-w9dk4" event={"ID":"b43c0ae2-9e13-4beb-a30b-e2147f3d4e41","Type":"ContainerStarted","Data":"8c1413baa4d4422c65c851a8086473446bbb29c4f1762ecebfa60127795ba97e"} Jul 07 19:03:40.408866 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:40.408764 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-5d4848bbfb-w9dk4" podStartSLOduration=2.205989732 podStartE2EDuration="4.408746929s" totalImagesPullingTime="2.202757197s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:03:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:03:40.405944356 +0000 UTC m=+203.421948537" watchObservedRunningTime="2026-07-07 19:03:40.408746929 +0000 UTC m=+203.424751110" Jul 07 19:03:41.317171 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:41.317129 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:03:43.048895 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:43.048843 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:03:52.362186 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:52.362148 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:03:52.463864 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:52.463825 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:03:52.545404 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:03:52.545364 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:04:09.560982 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:04:09.560926 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:04:30.780655 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:04:30.780620 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:04:30.880632 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:04:30.880596 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:04:38.048537 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:04:38.048480 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:04:38.063937 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:04:38.063911 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:04:38.620692 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:04:38.620659 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 07 19:04:41.264004 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:04:41.263904 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:04:46.795600 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:04:46.795558 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:04:46.881325 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:04:46.881283 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:04:58.780675 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:04:58.780635 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:04:58.853388 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:04:58.853356 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:05:02.013317 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:05:02.013283 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:05:02.195685 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:05:02.195649 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:05:17.416576 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:05:17.416550 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-frt7c_e25ab82f-72fb-4cde-89c4-0a0dd80f5feb/console-operator/2.log" Jul 07 19:05:17.419453 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:05:17.419426 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-frt7c_e25ab82f-72fb-4cde-89c4-0a0dd80f5feb/console-operator/2.log" Jul 07 19:05:17.423493 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:05:17.423467 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zzwbq_4f0c5d5c-147f-426a-965d-8886e5e79984/ovn-acl-logging/0.log" Jul 07 19:05:17.426166 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:05:17.426143 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zzwbq_4f0c5d5c-147f-426a-965d-8886e5e79984/ovn-acl-logging/0.log" Jul 07 19:05:17.427538 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:05:17.427521 2563 kubelet.go:1648] "Image garbage collection succeeded" Jul 07 19:05:20.196366 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:05:20.196331 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:05:20.269845 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:05:20.269807 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:05:20.914289 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:05:20.914246 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:05:21.003180 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:05:21.003150 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:05:24.370942 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:05:24.370902 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:05:30.670440 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:05:30.670401 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:05:30.772386 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:05:30.772342 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:05:30.872849 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:05:30.872811 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:05:48.181549 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:05:48.181512 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:05:49.549722 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:05:49.549685 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:05:52.214271 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:05:52.214234 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:05:52.311011 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:05:52.310976 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:05:53.148985 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:05:53.148925 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:05:53.254517 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:05:53.254491 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:05:54.332184 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:05:54.332148 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:05:54.439019 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:05:54.438991 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:05:56.593598 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:05:56.593565 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:05:56.943395 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:05:56.943308 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:05:58.962379 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:05:58.962348 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:05:59.034416 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:05:59.034376 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:06:04.547402 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:06:04.547370 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:06:04.918609 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:06:04.918522 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:06:05.018827 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:06:05.018793 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:06:05.174550 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:06:05.174458 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:06:05.275482 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:06:05.275442 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:06:07.322432 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:06:07.322397 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-f8xsd"] Jul 07 19:06:07.325815 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:06:07.325792 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-f8xsd" Jul 07 19:06:07.328908 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:06:07.328883 2563 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"cert-manager-dockercfg-566lq\" is forbidden: User \"system:node:ip-10-0-142-134.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"cert-manager\": no relationship found between node 'ip-10-0-142-134.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-566lq\"" type="*v1.Secret" Jul 07 19:06:07.329051 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:06:07.328916 2563 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"openshift-service-ca.crt\" is forbidden: User \"system:node:ip-10-0-142-134.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"cert-manager\": no relationship found between node 'ip-10-0-142-134.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" Jul 07 19:06:07.329275 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:06:07.329250 2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"cert-manager-55b4d954d5-f8xsd\" is forbidden: User \"system:node:ip-10-0-142-134.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"cert-manager\": no relationship found between node 'ip-10-0-142-134.ec2.internal' and this object" podUID="b81d3afc-6dd7-491b-9d96-655da9a2ade8" pod="cert-manager/cert-manager-55b4d954d5-f8xsd" Jul 07 19:06:07.329336 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:06:07.329273 2563 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"kube-root-ca.crt\" is forbidden: User \"system:node:ip-10-0-142-134.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"cert-manager\": no relationship found between node 'ip-10-0-142-134.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" Jul 07 19:06:07.338889 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:06:07.338866 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-f8xsd"] Jul 07 19:06:07.387417 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:06:07.387373 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b81d3afc-6dd7-491b-9d96-655da9a2ade8-bound-sa-token\") pod \"cert-manager-55b4d954d5-f8xsd\" (UID: \"b81d3afc-6dd7-491b-9d96-655da9a2ade8\") " pod="cert-manager/cert-manager-55b4d954d5-f8xsd" Jul 07 19:06:07.387417 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:06:07.387417 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7sc6\" (UniqueName: \"kubernetes.io/projected/b81d3afc-6dd7-491b-9d96-655da9a2ade8-kube-api-access-q7sc6\") pod \"cert-manager-55b4d954d5-f8xsd\" (UID: \"b81d3afc-6dd7-491b-9d96-655da9a2ade8\") " pod="cert-manager/cert-manager-55b4d954d5-f8xsd" Jul 07 19:06:07.488366 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:06:07.488323 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b81d3afc-6dd7-491b-9d96-655da9a2ade8-bound-sa-token\") pod \"cert-manager-55b4d954d5-f8xsd\" (UID: \"b81d3afc-6dd7-491b-9d96-655da9a2ade8\") " pod="cert-manager/cert-manager-55b4d954d5-f8xsd" Jul 07 19:06:07.488366 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:06:07.488366 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7sc6\" (UniqueName: \"kubernetes.io/projected/b81d3afc-6dd7-491b-9d96-655da9a2ade8-kube-api-access-q7sc6\") pod \"cert-manager-55b4d954d5-f8xsd\" (UID: \"b81d3afc-6dd7-491b-9d96-655da9a2ade8\") " pod="cert-manager/cert-manager-55b4d954d5-f8xsd" Jul 07 19:06:07.498728 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:06:07.498697 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b81d3afc-6dd7-491b-9d96-655da9a2ade8-bound-sa-token\") pod \"cert-manager-55b4d954d5-f8xsd\" (UID: \"b81d3afc-6dd7-491b-9d96-655da9a2ade8\") " pod="cert-manager/cert-manager-55b4d954d5-f8xsd" Jul 07 19:06:07.566102 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:06:07.566071 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-7pw62"] Jul 07 19:06:07.569363 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:06:07.569344 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-7pw62" Jul 07 19:06:07.573124 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:06:07.573069 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 07 19:06:07.573124 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:06:07.573083 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-ppg2f\"" Jul 07 19:06:07.573124 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:06:07.573069 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 07 19:06:07.584444 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:06:07.584418 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-7pw62"] Jul 07 19:06:07.589101 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:06:07.589072 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7phtp\" (UniqueName: \"kubernetes.io/projected/4a7f9314-da36-4f36-97f0-b87b49df7c92-kube-api-access-7phtp\") pod \"openshift-lws-operator-fd8ccff4c-7pw62\" (UID: \"4a7f9314-da36-4f36-97f0-b87b49df7c92\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-7pw62" Jul 07 19:06:07.589200 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:06:07.589151 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4a7f9314-da36-4f36-97f0-b87b49df7c92-tmp\") pod \"openshift-lws-operator-fd8ccff4c-7pw62\" (UID: \"4a7f9314-da36-4f36-97f0-b87b49df7c92\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-7pw62" Jul 07 19:06:07.689866 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:06:07.689832 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4a7f9314-da36-4f36-97f0-b87b49df7c92-tmp\") pod \"openshift-lws-operator-fd8ccff4c-7pw62\" (UID: \"4a7f9314-da36-4f36-97f0-b87b49df7c92\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-7pw62" Jul 07 19:06:07.690077 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:06:07.689942 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7phtp\" (UniqueName: \"kubernetes.io/projected/4a7f9314-da36-4f36-97f0-b87b49df7c92-kube-api-access-7phtp\") pod \"openshift-lws-operator-fd8ccff4c-7pw62\" (UID: \"4a7f9314-da36-4f36-97f0-b87b49df7c92\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-7pw62" Jul 07 19:06:07.690242 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:06:07.690221 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4a7f9314-da36-4f36-97f0-b87b49df7c92-tmp\") pod \"openshift-lws-operator-fd8ccff4c-7pw62\" (UID: \"4a7f9314-da36-4f36-97f0-b87b49df7c92\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-7pw62" Jul 07 19:06:07.703363 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:06:07.703326 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7phtp\" (UniqueName: \"kubernetes.io/projected/4a7f9314-da36-4f36-97f0-b87b49df7c92-kube-api-access-7phtp\") pod \"openshift-lws-operator-fd8ccff4c-7pw62\" (UID: \"4a7f9314-da36-4f36-97f0-b87b49df7c92\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-7pw62" Jul 07 19:06:07.878978 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:06:07.878865 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-7pw62" Jul 07 19:06:08.013477 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:06:08.013444 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-7pw62"] Jul 07 19:06:08.016644 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:06:08.016614 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4a7f9314_da36_4f36_97f0_b87b49df7c92.slice/crio-6450eb4ee693c75b7dec3b5fbf3895111934646ba1320e43710c5e57122901c4 WatchSource:0}: Error finding container 6450eb4ee693c75b7dec3b5fbf3895111934646ba1320e43710c5e57122901c4: Status 404 returned error can't find the container with id 6450eb4ee693c75b7dec3b5fbf3895111934646ba1320e43710c5e57122901c4 Jul 07 19:06:08.018149 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:06:08.018133 2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 07 19:06:08.496758 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:06:08.496708 2563 projected.go:291] Couldn't get configMap cert-manager/kube-root-ca.crt: failed to sync configmap cache: timed out waiting for the condition Jul 07 19:06:08.615987 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:06:08.615960 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-566lq\"" Jul 07 19:06:08.649250 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:06:08.649215 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 07 19:06:08.793853 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:06:08.793825 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 07 19:06:08.797600 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:06:08.797581 2563 projected.go:196] Error preparing data for projected volume kube-api-access-q7sc6 for pod cert-manager/cert-manager-55b4d954d5-f8xsd: failed to sync configmap cache: timed out waiting for the condition Jul 07 19:06:08.797692 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:06:08.797646 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b81d3afc-6dd7-491b-9d96-655da9a2ade8-kube-api-access-q7sc6 podName:b81d3afc-6dd7-491b-9d96-655da9a2ade8 nodeName:}" failed. No retries permitted until 2026-07-07 19:06:09.297631263 +0000 UTC m=+352.313635422 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-q7sc6" (UniqueName: "kubernetes.io/projected/b81d3afc-6dd7-491b-9d96-655da9a2ade8-kube-api-access-q7sc6") pod "cert-manager-55b4d954d5-f8xsd" (UID: "b81d3afc-6dd7-491b-9d96-655da9a2ade8") : failed to sync configmap cache: timed out waiting for the condition Jul 07 19:06:08.948287 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:06:08.948257 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-7pw62" event={"ID":"4a7f9314-da36-4f36-97f0-b87b49df7c92","Type":"ContainerStarted","Data":"6450eb4ee693c75b7dec3b5fbf3895111934646ba1320e43710c5e57122901c4"} Jul 07 19:06:09.303925 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:06:09.303888 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7sc6\" (UniqueName: \"kubernetes.io/projected/b81d3afc-6dd7-491b-9d96-655da9a2ade8-kube-api-access-q7sc6\") pod \"cert-manager-55b4d954d5-f8xsd\" (UID: \"b81d3afc-6dd7-491b-9d96-655da9a2ade8\") " pod="cert-manager/cert-manager-55b4d954d5-f8xsd" Jul 07 19:06:09.306401 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:06:09.306378 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7sc6\" (UniqueName: \"kubernetes.io/projected/b81d3afc-6dd7-491b-9d96-655da9a2ade8-kube-api-access-q7sc6\") pod \"cert-manager-55b4d954d5-f8xsd\" (UID: \"b81d3afc-6dd7-491b-9d96-655da9a2ade8\") " pod="cert-manager/cert-manager-55b4d954d5-f8xsd" Jul 07 19:06:09.445438 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:06:09.445390 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-f8xsd" Jul 07 19:06:09.581661 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:06:09.581567 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-f8xsd"] Jul 07 19:06:09.584795 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:06:09.584767 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb81d3afc_6dd7_491b_9d96_655da9a2ade8.slice/crio-12e56446c0998572f9c953d527a6abfc6b1ee851567bc7fe376e9719c20ae76e WatchSource:0}: Error finding container 12e56446c0998572f9c953d527a6abfc6b1ee851567bc7fe376e9719c20ae76e: Status 404 returned error can't find the container with id 12e56446c0998572f9c953d527a6abfc6b1ee851567bc7fe376e9719c20ae76e Jul 07 19:06:09.953832 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:06:09.953757 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-f8xsd" event={"ID":"b81d3afc-6dd7-491b-9d96-655da9a2ade8","Type":"ContainerStarted","Data":"12e56446c0998572f9c953d527a6abfc6b1ee851567bc7fe376e9719c20ae76e"} Jul 07 19:06:13.974029 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:06:13.973914 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-7pw62" event={"ID":"4a7f9314-da36-4f36-97f0-b87b49df7c92","Type":"ContainerStarted","Data":"c3ea5508195448021e0fa1ff24706d99aca0bc94e5d9529dd3894360cc443004"} Jul 07 19:06:13.975500 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:06:13.975474 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-f8xsd" event={"ID":"b81d3afc-6dd7-491b-9d96-655da9a2ade8","Type":"ContainerStarted","Data":"c7168580d68506f72d0a77d132d9e755687b625b389f683fdb5121eece080811"} Jul 07 19:06:14.000917 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:06:14.000861 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-7pw62" podStartSLOduration=1.258562437 podStartE2EDuration="7.000843988s" totalImagesPullingTime="5.742281551s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:06:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:06:13.998580116 +0000 UTC m=+357.014584298" watchObservedRunningTime="2026-07-07 19:06:14.000843988 +0000 UTC m=+357.016848169" Jul 07 19:06:14.020163 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:06:14.020109 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-f8xsd" podStartSLOduration=2.895559633 podStartE2EDuration="7.020089869s" totalImagesPullingTime="4.124530236s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:06:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:06:14.019083089 +0000 UTC m=+357.035087282" watchObservedRunningTime="2026-07-07 19:06:14.020089869 +0000 UTC m=+357.036094049" Jul 07 19:06:14.721932 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:06:14.721896 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:06:14.831631 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:06:14.831591 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:06:17.305895 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:06:17.305858 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:06:28.757486 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:06:28.757448 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:06:28.852352 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:06:28.852321 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:06:29.362878 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:06:29.362850 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:06:29.709694 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:06:29.709609 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:06:29.849581 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:06:29.849546 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:06:29.886515 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:06:29.886483 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-54cc9bdf65-hwc7h"] Jul 07 19:06:29.892388 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:06:29.892368 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-54cc9bdf65-hwc7h" Jul 07 19:06:29.894606 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:06:29.894584 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 07 19:06:29.894912 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:06:29.894892 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 07 19:06:29.895048 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:06:29.895034 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 07 19:06:29.895228 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:06:29.895210 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 07 19:06:29.897338 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:06:29.897314 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-4vlfr\"" Jul 07 19:06:29.922230 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:06:29.922203 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-54cc9bdf65-hwc7h"] Jul 07 19:06:29.993340 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:06:29.993271 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5l9pz\" (UniqueName: \"kubernetes.io/projected/db7846e2-43d5-4f0a-8cbd-3b9aa92e564a-kube-api-access-5l9pz\") pod \"opendatahub-operator-controller-manager-54cc9bdf65-hwc7h\" (UID: \"db7846e2-43d5-4f0a-8cbd-3b9aa92e564a\") " pod="opendatahub/opendatahub-operator-controller-manager-54cc9bdf65-hwc7h" Jul 07 19:06:29.993340 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:06:29.993310 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/db7846e2-43d5-4f0a-8cbd-3b9aa92e564a-webhook-cert\") pod \"opendatahub-operator-controller-manager-54cc9bdf65-hwc7h\" (UID: \"db7846e2-43d5-4f0a-8cbd-3b9aa92e564a\") " pod="opendatahub/opendatahub-operator-controller-manager-54cc9bdf65-hwc7h" Jul 07 19:06:29.993340 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:06:29.993331 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/db7846e2-43d5-4f0a-8cbd-3b9aa92e564a-apiservice-cert\") pod \"opendatahub-operator-controller-manager-54cc9bdf65-hwc7h\" (UID: \"db7846e2-43d5-4f0a-8cbd-3b9aa92e564a\") " pod="opendatahub/opendatahub-operator-controller-manager-54cc9bdf65-hwc7h" Jul 07 19:06:29.993536 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:06:29.993422 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/db7846e2-43d5-4f0a-8cbd-3b9aa92e564a-tmp\") pod \"opendatahub-operator-controller-manager-54cc9bdf65-hwc7h\" (UID: \"db7846e2-43d5-4f0a-8cbd-3b9aa92e564a\") " pod="opendatahub/opendatahub-operator-controller-manager-54cc9bdf65-hwc7h" Jul 07 19:06:29.993536 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:06:29.993471 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/db7846e2-43d5-4f0a-8cbd-3b9aa92e564a-manifests\") pod \"opendatahub-operator-controller-manager-54cc9bdf65-hwc7h\" (UID: \"db7846e2-43d5-4f0a-8cbd-3b9aa92e564a\") " pod="opendatahub/opendatahub-operator-controller-manager-54cc9bdf65-hwc7h" Jul 07 19:06:30.074183 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:06:30.074148 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:06:30.094820 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:06:30.094792 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5l9pz\" (UniqueName: \"kubernetes.io/projected/db7846e2-43d5-4f0a-8cbd-3b9aa92e564a-kube-api-access-5l9pz\") pod \"opendatahub-operator-controller-manager-54cc9bdf65-hwc7h\" (UID: \"db7846e2-43d5-4f0a-8cbd-3b9aa92e564a\") " pod="opendatahub/opendatahub-operator-controller-manager-54cc9bdf65-hwc7h" Jul 07 19:06:30.094941 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:06:30.094839 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/db7846e2-43d5-4f0a-8cbd-3b9aa92e564a-webhook-cert\") pod \"opendatahub-operator-controller-manager-54cc9bdf65-hwc7h\" (UID: \"db7846e2-43d5-4f0a-8cbd-3b9aa92e564a\") " pod="opendatahub/opendatahub-operator-controller-manager-54cc9bdf65-hwc7h" Jul 07 19:06:30.094941 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:06:30.094920 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/db7846e2-43d5-4f0a-8cbd-3b9aa92e564a-apiservice-cert\") pod \"opendatahub-operator-controller-manager-54cc9bdf65-hwc7h\" (UID: \"db7846e2-43d5-4f0a-8cbd-3b9aa92e564a\") " pod="opendatahub/opendatahub-operator-controller-manager-54cc9bdf65-hwc7h" Jul 07 19:06:30.095078 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:06:30.095004 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/db7846e2-43d5-4f0a-8cbd-3b9aa92e564a-tmp\") pod \"opendatahub-operator-controller-manager-54cc9bdf65-hwc7h\" (UID: \"db7846e2-43d5-4f0a-8cbd-3b9aa92e564a\") " pod="opendatahub/opendatahub-operator-controller-manager-54cc9bdf65-hwc7h" Jul 07 19:06:30.095078 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:06:30.095051 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/db7846e2-43d5-4f0a-8cbd-3b9aa92e564a-manifests\") pod \"opendatahub-operator-controller-manager-54cc9bdf65-hwc7h\" (UID: \"db7846e2-43d5-4f0a-8cbd-3b9aa92e564a\") " pod="opendatahub/opendatahub-operator-controller-manager-54cc9bdf65-hwc7h" Jul 07 19:06:30.095407 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:06:30.095385 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/db7846e2-43d5-4f0a-8cbd-3b9aa92e564a-manifests\") pod \"opendatahub-operator-controller-manager-54cc9bdf65-hwc7h\" (UID: \"db7846e2-43d5-4f0a-8cbd-3b9aa92e564a\") " pod="opendatahub/opendatahub-operator-controller-manager-54cc9bdf65-hwc7h" Jul 07 19:06:30.097225 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:06:30.097194 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/db7846e2-43d5-4f0a-8cbd-3b9aa92e564a-tmp\") pod \"opendatahub-operator-controller-manager-54cc9bdf65-hwc7h\" (UID: \"db7846e2-43d5-4f0a-8cbd-3b9aa92e564a\") " pod="opendatahub/opendatahub-operator-controller-manager-54cc9bdf65-hwc7h" Jul 07 19:06:30.097317 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:06:30.097247 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/db7846e2-43d5-4f0a-8cbd-3b9aa92e564a-apiservice-cert\") pod \"opendatahub-operator-controller-manager-54cc9bdf65-hwc7h\" (UID: \"db7846e2-43d5-4f0a-8cbd-3b9aa92e564a\") " pod="opendatahub/opendatahub-operator-controller-manager-54cc9bdf65-hwc7h" Jul 07 19:06:30.097317 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:06:30.097285 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/db7846e2-43d5-4f0a-8cbd-3b9aa92e564a-webhook-cert\") pod \"opendatahub-operator-controller-manager-54cc9bdf65-hwc7h\" (UID: \"db7846e2-43d5-4f0a-8cbd-3b9aa92e564a\") " pod="opendatahub/opendatahub-operator-controller-manager-54cc9bdf65-hwc7h" Jul 07 19:06:30.117048 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:06:30.117030 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5l9pz\" (UniqueName: \"kubernetes.io/projected/db7846e2-43d5-4f0a-8cbd-3b9aa92e564a-kube-api-access-5l9pz\") pod \"opendatahub-operator-controller-manager-54cc9bdf65-hwc7h\" (UID: \"db7846e2-43d5-4f0a-8cbd-3b9aa92e564a\") " pod="opendatahub/opendatahub-operator-controller-manager-54cc9bdf65-hwc7h" Jul 07 19:06:30.203517 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:06:30.203481 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:06:30.204278 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:06:30.204260 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-54cc9bdf65-hwc7h" Jul 07 19:06:30.314217 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:06:30.314181 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:06:30.349311 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:06:30.349289 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-54cc9bdf65-hwc7h"] Jul 07 19:06:30.351994 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:06:30.351944 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb7846e2_43d5_4f0a_8cbd_3b9aa92e564a.slice/crio-8ed3cb59863630031cc424ca6c8008562ac787c2a67e74f44a6c79f7cf5151c4 WatchSource:0}: Error finding container 8ed3cb59863630031cc424ca6c8008562ac787c2a67e74f44a6c79f7cf5151c4: Status 404 returned error can't find the container with id 8ed3cb59863630031cc424ca6c8008562ac787c2a67e74f44a6c79f7cf5151c4 Jul 07 19:06:30.408885 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:06:30.408844 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:06:30.510323 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:06:30.510237 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:06:30.609801 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:06:30.609771 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:06:30.711476 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:06:30.711438 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:06:31.046049 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:06:31.045973 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-54cc9bdf65-hwc7h" event={"ID":"db7846e2-43d5-4f0a-8cbd-3b9aa92e564a","Type":"ContainerStarted","Data":"8ed3cb59863630031cc424ca6c8008562ac787c2a67e74f44a6c79f7cf5151c4"} Jul 07 19:06:33.516378 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:06:33.516346 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:06:33.616903 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:06:33.616864 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:06:34.062216 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:06:34.062182 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-54cc9bdf65-hwc7h" event={"ID":"db7846e2-43d5-4f0a-8cbd-3b9aa92e564a","Type":"ContainerStarted","Data":"b797e546c50ab0a3426f2b7fc5b1f3b4260dfdeb9b81b9f2b42df894f0f472ef"} Jul 07 19:06:35.069136 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:06:35.069098 2563 generic.go:356] "Generic (PLEG): container finished" podID="db7846e2-43d5-4f0a-8cbd-3b9aa92e564a" containerID="b797e546c50ab0a3426f2b7fc5b1f3b4260dfdeb9b81b9f2b42df894f0f472ef" exitCode=0 Jul 07 19:06:35.069550 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:06:35.069186 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-54cc9bdf65-hwc7h" event={"ID":"db7846e2-43d5-4f0a-8cbd-3b9aa92e564a","Type":"ContainerDied","Data":"b797e546c50ab0a3426f2b7fc5b1f3b4260dfdeb9b81b9f2b42df894f0f472ef"} Jul 07 19:06:36.076039 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:06:36.076001 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-54cc9bdf65-hwc7h" event={"ID":"db7846e2-43d5-4f0a-8cbd-3b9aa92e564a","Type":"ContainerStarted","Data":"734917a0c405ec3d312fad454744c65885cb8e538e9631b4da4f37bad0e457eb"} Jul 07 19:06:36.076530 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:06:36.076123 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-54cc9bdf65-hwc7h" Jul 07 19:06:36.150765 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:06:36.150714 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-54cc9bdf65-hwc7h" podStartSLOduration=2.452490459 podStartE2EDuration="7.15069225s" totalImagesPullingTime="3.668610502s" totalInitContainerRuntime="1.029591289s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:06:29 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:06:36.150526602 +0000 UTC m=+379.166530783" watchObservedRunningTime="2026-07-07 19:06:36.15069225 +0000 UTC m=+379.166696431" Jul 07 19:06:40.314261 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:06:40.314215 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:06:40.399491 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:06:40.399450 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:06:40.478277 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:06:40.478241 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:06:47.083587 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:06:47.083554 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-54cc9bdf65-hwc7h" Jul 07 19:06:49.185006 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:06:49.184969 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:06:49.285159 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:06:49.285114 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:06:49.709297 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:06:49.709254 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:06:49.866089 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:06:49.866051 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:06:50.135399 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:06:50.135365 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:06:50.244367 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:06:50.244323 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:06:50.267437 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:06:50.267406 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-5f4bfc584b-jcd55"] Jul 07 19:06:50.270825 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:06:50.270809 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5f4bfc584b-jcd55" Jul 07 19:06:50.272884 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:06:50.272857 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qdp5x\" (UniqueName: \"kubernetes.io/projected/c288a2e8-a2a9-4125-a538-552328cff0a9-kube-api-access-qdp5x\") pod \"lws-controller-manager-5f4bfc584b-jcd55\" (UID: \"c288a2e8-a2a9-4125-a538-552328cff0a9\") " pod="openshift-lws-operator/lws-controller-manager-5f4bfc584b-jcd55" Jul 07 19:06:50.273028 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:06:50.272912 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/c288a2e8-a2a9-4125-a538-552328cff0a9-metrics-cert\") pod \"lws-controller-manager-5f4bfc584b-jcd55\" (UID: \"c288a2e8-a2a9-4125-a538-552328cff0a9\") " pod="openshift-lws-operator/lws-controller-manager-5f4bfc584b-jcd55" Jul 07 19:06:50.273028 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:06:50.273014 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/c288a2e8-a2a9-4125-a538-552328cff0a9-manager-config\") pod \"lws-controller-manager-5f4bfc584b-jcd55\" (UID: \"c288a2e8-a2a9-4125-a538-552328cff0a9\") " pod="openshift-lws-operator/lws-controller-manager-5f4bfc584b-jcd55" Jul 07 19:06:50.273114 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:06:50.273043 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c288a2e8-a2a9-4125-a538-552328cff0a9-cert\") pod \"lws-controller-manager-5f4bfc584b-jcd55\" (UID: \"c288a2e8-a2a9-4125-a538-552328cff0a9\") " pod="openshift-lws-operator/lws-controller-manager-5f4bfc584b-jcd55" Jul 07 19:06:50.273706 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:06:50.273689 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-fvnrv\"" Jul 07 19:06:50.273810 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:06:50.273792 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 07 19:06:50.273873 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:06:50.273821 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 07 19:06:50.273930 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:06:50.273915 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 07 19:06:50.284685 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:06:50.284663 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5f4bfc584b-jcd55"] Jul 07 19:06:50.362567 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:06:50.362530 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:06:50.374078 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:06:50.374047 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/c288a2e8-a2a9-4125-a538-552328cff0a9-metrics-cert\") pod \"lws-controller-manager-5f4bfc584b-jcd55\" (UID: \"c288a2e8-a2a9-4125-a538-552328cff0a9\") " pod="openshift-lws-operator/lws-controller-manager-5f4bfc584b-jcd55" Jul 07 19:06:50.374247 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:06:50.374113 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/c288a2e8-a2a9-4125-a538-552328cff0a9-manager-config\") pod \"lws-controller-manager-5f4bfc584b-jcd55\" (UID: \"c288a2e8-a2a9-4125-a538-552328cff0a9\") " pod="openshift-lws-operator/lws-controller-manager-5f4bfc584b-jcd55" Jul 07 19:06:50.374247 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:06:50.374146 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c288a2e8-a2a9-4125-a538-552328cff0a9-cert\") pod \"lws-controller-manager-5f4bfc584b-jcd55\" (UID: \"c288a2e8-a2a9-4125-a538-552328cff0a9\") " pod="openshift-lws-operator/lws-controller-manager-5f4bfc584b-jcd55" Jul 07 19:06:50.374247 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:06:50.374203 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdp5x\" (UniqueName: \"kubernetes.io/projected/c288a2e8-a2a9-4125-a538-552328cff0a9-kube-api-access-qdp5x\") pod \"lws-controller-manager-5f4bfc584b-jcd55\" (UID: \"c288a2e8-a2a9-4125-a538-552328cff0a9\") " pod="openshift-lws-operator/lws-controller-manager-5f4bfc584b-jcd55" Jul 07 19:06:50.374890 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:06:50.374866 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/c288a2e8-a2a9-4125-a538-552328cff0a9-manager-config\") pod \"lws-controller-manager-5f4bfc584b-jcd55\" (UID: \"c288a2e8-a2a9-4125-a538-552328cff0a9\") " pod="openshift-lws-operator/lws-controller-manager-5f4bfc584b-jcd55" Jul 07 19:06:50.376733 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:06:50.376708 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/c288a2e8-a2a9-4125-a538-552328cff0a9-metrics-cert\") pod \"lws-controller-manager-5f4bfc584b-jcd55\" (UID: \"c288a2e8-a2a9-4125-a538-552328cff0a9\") " pod="openshift-lws-operator/lws-controller-manager-5f4bfc584b-jcd55" Jul 07 19:06:50.376978 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:06:50.376941 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c288a2e8-a2a9-4125-a538-552328cff0a9-cert\") pod \"lws-controller-manager-5f4bfc584b-jcd55\" (UID: \"c288a2e8-a2a9-4125-a538-552328cff0a9\") " pod="openshift-lws-operator/lws-controller-manager-5f4bfc584b-jcd55" Jul 07 19:06:50.406203 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:06:50.406134 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qdp5x\" (UniqueName: \"kubernetes.io/projected/c288a2e8-a2a9-4125-a538-552328cff0a9-kube-api-access-qdp5x\") pod \"lws-controller-manager-5f4bfc584b-jcd55\" (UID: \"c288a2e8-a2a9-4125-a538-552328cff0a9\") " pod="openshift-lws-operator/lws-controller-manager-5f4bfc584b-jcd55" Jul 07 19:06:50.467151 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:06:50.467117 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:06:50.579667 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:06:50.579627 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5f4bfc584b-jcd55" Jul 07 19:06:50.735710 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:06:50.735680 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5f4bfc584b-jcd55"] Jul 07 19:06:50.737728 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:06:50.737698 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc288a2e8_a2a9_4125_a538_552328cff0a9.slice/crio-308cd23a36922038a55a0ef9f19c23ad950963ff760e9c5d3b90b6bf2a7d3b08 WatchSource:0}: Error finding container 308cd23a36922038a55a0ef9f19c23ad950963ff760e9c5d3b90b6bf2a7d3b08: Status 404 returned error can't find the container with id 308cd23a36922038a55a0ef9f19c23ad950963ff760e9c5d3b90b6bf2a7d3b08 Jul 07 19:06:50.794106 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:06:50.794078 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:06:51.151426 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:06:51.151382 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5f4bfc584b-jcd55" event={"ID":"c288a2e8-a2a9-4125-a538-552328cff0a9","Type":"ContainerStarted","Data":"308cd23a36922038a55a0ef9f19c23ad950963ff760e9c5d3b90b6bf2a7d3b08"} Jul 07 19:06:51.420788 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:06:51.420652 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:06:51.613678 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:06:51.613644 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:06:52.090461 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:06:52.090422 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:06:52.198901 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:06:52.198864 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:06:52.280078 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:06:52.280045 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:06:53.163631 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:06:53.163596 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5f4bfc584b-jcd55" event={"ID":"c288a2e8-a2a9-4125-a538-552328cff0a9","Type":"ContainerStarted","Data":"df49f11608647f75d09e94c92ec87b9848ab8e8ed6f4a138189bea8c45e5beef"} Jul 07 19:06:53.164044 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:06:53.163723 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-5f4bfc584b-jcd55" Jul 07 19:06:53.184868 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:06:53.184814 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-5f4bfc584b-jcd55" podStartSLOduration=1.818710002 podStartE2EDuration="3.184797488s" totalImagesPullingTime="1.366087486s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:06:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:06:53.182654843 +0000 UTC m=+396.198659024" watchObservedRunningTime="2026-07-07 19:06:53.184797488 +0000 UTC m=+396.200801668" Jul 07 19:06:53.843420 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:06:53.843385 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:06:56.350170 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:06:56.350125 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:06:56.451960 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:06:56.451928 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:06:56.621652 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:06:56.621561 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-85fb56d756f9m98"] Jul 07 19:06:56.625004 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:06:56.624983 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-85fb56d756f9m98" Jul 07 19:06:56.628917 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:06:56.628891 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 07 19:06:56.629122 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:06:56.629100 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-xjspm\"" Jul 07 19:06:56.629227 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:06:56.629194 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 07 19:06:56.629227 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:06:56.629110 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 07 19:06:56.640290 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:06:56.640264 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-85fb56d756f9m98"] Jul 07 19:06:56.661521 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:06:56.661487 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:06:56.702810 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:06:56.702771 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-554689689dtjt6l"] Jul 07 19:06:56.706074 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:06:56.706055 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-554689689dtjt6l" Jul 07 19:06:56.720915 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:06:56.720885 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-554689689dtjt6l"] Jul 07 19:06:56.728393 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:06:56.728360 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f241d30a-cf63-4c57-8115-decfdd34120a-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-85fb56d756f9m98\" (UID: \"f241d30a-cf63-4c57-8115-decfdd34120a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-85fb56d756f9m98" Jul 07 19:06:56.728567 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:06:56.728414 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a0d31c0-5fe7-4d3e-9ee3-8266857f6931-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-554689689dtjt6l\" (UID: \"1a0d31c0-5fe7-4d3e-9ee3-8266857f6931\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-554689689dtjt6l" Jul 07 19:06:56.728567 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:06:56.728475 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lkd59\" (UniqueName: \"kubernetes.io/projected/1a0d31c0-5fe7-4d3e-9ee3-8266857f6931-kube-api-access-lkd59\") pod \"data-science-gateway-data-science-gateway-class-554689689dtjt6l\" (UID: \"1a0d31c0-5fe7-4d3e-9ee3-8266857f6931\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-554689689dtjt6l" Jul 07 19:06:56.728567 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:06:56.728531 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f241d30a-cf63-4c57-8115-decfdd34120a-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-85fb56d756f9m98\" (UID: \"f241d30a-cf63-4c57-8115-decfdd34120a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-85fb56d756f9m98" Jul 07 19:06:56.728567 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:06:56.728558 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a0d31c0-5fe7-4d3e-9ee3-8266857f6931-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-554689689dtjt6l\" (UID: \"1a0d31c0-5fe7-4d3e-9ee3-8266857f6931\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-554689689dtjt6l" Jul 07 19:06:56.728714 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:06:56.728584 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f241d30a-cf63-4c57-8115-decfdd34120a-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-85fb56d756f9m98\" (UID: \"f241d30a-cf63-4c57-8115-decfdd34120a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-85fb56d756f9m98" Jul 07 19:06:56.728714 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:06:56.728609 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a0d31c0-5fe7-4d3e-9ee3-8266857f6931-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-554689689dtjt6l\" (UID: \"1a0d31c0-5fe7-4d3e-9ee3-8266857f6931\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-554689689dtjt6l" Jul 07 19:06:56.728714 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:06:56.728679 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a0d31c0-5fe7-4d3e-9ee3-8266857f6931-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-554689689dtjt6l\" (UID: \"1a0d31c0-5fe7-4d3e-9ee3-8266857f6931\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-554689689dtjt6l" Jul 07 19:06:56.728816 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:06:56.728709 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f241d30a-cf63-4c57-8115-decfdd34120a-istio-token\") pod \"data-science-gateway-data-science-gateway-class-85fb56d756f9m98\" (UID: \"f241d30a-cf63-4c57-8115-decfdd34120a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-85fb56d756f9m98" Jul 07 19:06:56.728816 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:06:56.728746 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f241d30a-cf63-4c57-8115-decfdd34120a-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-85fb56d756f9m98\" (UID: \"f241d30a-cf63-4c57-8115-decfdd34120a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-85fb56d756f9m98" Jul 07 19:06:56.728816 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:06:56.728790 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a0d31c0-5fe7-4d3e-9ee3-8266857f6931-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-554689689dtjt6l\" (UID: \"1a0d31c0-5fe7-4d3e-9ee3-8266857f6931\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-554689689dtjt6l" Jul 07 19:06:56.728915 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:06:56.728815 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f241d30a-cf63-4c57-8115-decfdd34120a-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-85fb56d756f9m98\" (UID: \"f241d30a-cf63-4c57-8115-decfdd34120a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-85fb56d756f9m98" Jul 07 19:06:56.728915 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:06:56.728842 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f241d30a-cf63-4c57-8115-decfdd34120a-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-85fb56d756f9m98\" (UID: \"f241d30a-cf63-4c57-8115-decfdd34120a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-85fb56d756f9m98" Jul 07 19:06:56.728915 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:06:56.728861 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f241d30a-cf63-4c57-8115-decfdd34120a-istio-data\") pod \"data-science-gateway-data-science-gateway-class-85fb56d756f9m98\" (UID: \"f241d30a-cf63-4c57-8115-decfdd34120a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-85fb56d756f9m98" Jul 07 19:06:56.728915 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:06:56.728881 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a0d31c0-5fe7-4d3e-9ee3-8266857f6931-istio-data\") pod \"data-science-gateway-data-science-gateway-class-554689689dtjt6l\" (UID: \"1a0d31c0-5fe7-4d3e-9ee3-8266857f6931\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-554689689dtjt6l" Jul 07 19:06:56.728915 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:06:56.728896 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a0d31c0-5fe7-4d3e-9ee3-8266857f6931-istio-token\") pod \"data-science-gateway-data-science-gateway-class-554689689dtjt6l\" (UID: \"1a0d31c0-5fe7-4d3e-9ee3-8266857f6931\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-554689689dtjt6l" Jul 07 19:06:56.729261 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:06:56.728918 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a0d31c0-5fe7-4d3e-9ee3-8266857f6931-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-554689689dtjt6l\" (UID: \"1a0d31c0-5fe7-4d3e-9ee3-8266857f6931\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-554689689dtjt6l" Jul 07 19:06:56.729261 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:06:56.728947 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5467\" (UniqueName: \"kubernetes.io/projected/f241d30a-cf63-4c57-8115-decfdd34120a-kube-api-access-k5467\") pod \"data-science-gateway-data-science-gateway-class-85fb56d756f9m98\" (UID: \"f241d30a-cf63-4c57-8115-decfdd34120a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-85fb56d756f9m98" Jul 07 19:06:56.829700 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:06:56.829654 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f241d30a-cf63-4c57-8115-decfdd34120a-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-85fb56d756f9m98\" (UID: \"f241d30a-cf63-4c57-8115-decfdd34120a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-85fb56d756f9m98" Jul 07 19:06:56.829700 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:06:56.829704 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a0d31c0-5fe7-4d3e-9ee3-8266857f6931-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-554689689dtjt6l\" (UID: \"1a0d31c0-5fe7-4d3e-9ee3-8266857f6931\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-554689689dtjt6l" Jul 07 19:06:56.829935 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:06:56.829795 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkd59\" (UniqueName: \"kubernetes.io/projected/1a0d31c0-5fe7-4d3e-9ee3-8266857f6931-kube-api-access-lkd59\") pod \"data-science-gateway-data-science-gateway-class-554689689dtjt6l\" (UID: \"1a0d31c0-5fe7-4d3e-9ee3-8266857f6931\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-554689689dtjt6l" Jul 07 19:06:56.829935 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:06:56.829836 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f241d30a-cf63-4c57-8115-decfdd34120a-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-85fb56d756f9m98\" (UID: \"f241d30a-cf63-4c57-8115-decfdd34120a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-85fb56d756f9m98" Jul 07 19:06:56.829935 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:06:56.829853 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a0d31c0-5fe7-4d3e-9ee3-8266857f6931-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-554689689dtjt6l\" (UID: \"1a0d31c0-5fe7-4d3e-9ee3-8266857f6931\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-554689689dtjt6l" Jul 07 19:06:56.829935 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:06:56.829877 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f241d30a-cf63-4c57-8115-decfdd34120a-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-85fb56d756f9m98\" (UID: \"f241d30a-cf63-4c57-8115-decfdd34120a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-85fb56d756f9m98" Jul 07 19:06:56.829935 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:06:56.829901 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a0d31c0-5fe7-4d3e-9ee3-8266857f6931-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-554689689dtjt6l\" (UID: \"1a0d31c0-5fe7-4d3e-9ee3-8266857f6931\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-554689689dtjt6l" Jul 07 19:06:56.830210 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:06:56.829942 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a0d31c0-5fe7-4d3e-9ee3-8266857f6931-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-554689689dtjt6l\" (UID: \"1a0d31c0-5fe7-4d3e-9ee3-8266857f6931\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-554689689dtjt6l" Jul 07 19:06:56.830210 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:06:56.830001 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f241d30a-cf63-4c57-8115-decfdd34120a-istio-token\") pod \"data-science-gateway-data-science-gateway-class-85fb56d756f9m98\" (UID: \"f241d30a-cf63-4c57-8115-decfdd34120a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-85fb56d756f9m98" Jul 07 19:06:56.830210 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:06:56.830045 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f241d30a-cf63-4c57-8115-decfdd34120a-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-85fb56d756f9m98\" (UID: \"f241d30a-cf63-4c57-8115-decfdd34120a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-85fb56d756f9m98" Jul 07 19:06:56.830210 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:06:56.830074 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f241d30a-cf63-4c57-8115-decfdd34120a-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-85fb56d756f9m98\" (UID: \"f241d30a-cf63-4c57-8115-decfdd34120a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-85fb56d756f9m98" Jul 07 19:06:56.830210 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:06:56.830086 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a0d31c0-5fe7-4d3e-9ee3-8266857f6931-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-554689689dtjt6l\" (UID: \"1a0d31c0-5fe7-4d3e-9ee3-8266857f6931\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-554689689dtjt6l" Jul 07 19:06:56.830210 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:06:56.830118 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f241d30a-cf63-4c57-8115-decfdd34120a-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-85fb56d756f9m98\" (UID: \"f241d30a-cf63-4c57-8115-decfdd34120a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-85fb56d756f9m98" Jul 07 19:06:56.830210 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:06:56.830147 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f241d30a-cf63-4c57-8115-decfdd34120a-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-85fb56d756f9m98\" (UID: \"f241d30a-cf63-4c57-8115-decfdd34120a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-85fb56d756f9m98" Jul 07 19:06:56.830210 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:06:56.830174 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f241d30a-cf63-4c57-8115-decfdd34120a-istio-data\") pod \"data-science-gateway-data-science-gateway-class-85fb56d756f9m98\" (UID: \"f241d30a-cf63-4c57-8115-decfdd34120a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-85fb56d756f9m98" Jul 07 19:06:56.830210 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:06:56.830202 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a0d31c0-5fe7-4d3e-9ee3-8266857f6931-istio-data\") pod \"data-science-gateway-data-science-gateway-class-554689689dtjt6l\" (UID: \"1a0d31c0-5fe7-4d3e-9ee3-8266857f6931\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-554689689dtjt6l" Jul 07 19:06:56.830685 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:06:56.830230 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a0d31c0-5fe7-4d3e-9ee3-8266857f6931-istio-token\") pod \"data-science-gateway-data-science-gateway-class-554689689dtjt6l\" (UID: \"1a0d31c0-5fe7-4d3e-9ee3-8266857f6931\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-554689689dtjt6l" Jul 07 19:06:56.830685 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:06:56.830255 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a0d31c0-5fe7-4d3e-9ee3-8266857f6931-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-554689689dtjt6l\" (UID: \"1a0d31c0-5fe7-4d3e-9ee3-8266857f6931\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-554689689dtjt6l" Jul 07 19:06:56.830685 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:06:56.830299 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5467\" (UniqueName: \"kubernetes.io/projected/f241d30a-cf63-4c57-8115-decfdd34120a-kube-api-access-k5467\") pod \"data-science-gateway-data-science-gateway-class-85fb56d756f9m98\" (UID: \"f241d30a-cf63-4c57-8115-decfdd34120a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-85fb56d756f9m98" Jul 07 19:06:56.830685 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:06:56.830324 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a0d31c0-5fe7-4d3e-9ee3-8266857f6931-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-554689689dtjt6l\" (UID: \"1a0d31c0-5fe7-4d3e-9ee3-8266857f6931\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-554689689dtjt6l" Jul 07 19:06:56.830685 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:06:56.830356 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a0d31c0-5fe7-4d3e-9ee3-8266857f6931-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-554689689dtjt6l\" (UID: \"1a0d31c0-5fe7-4d3e-9ee3-8266857f6931\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-554689689dtjt6l" Jul 07 19:06:56.830685 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:06:56.830623 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a0d31c0-5fe7-4d3e-9ee3-8266857f6931-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-554689689dtjt6l\" (UID: \"1a0d31c0-5fe7-4d3e-9ee3-8266857f6931\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-554689689dtjt6l" Jul 07 19:06:56.830685 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:06:56.830649 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f241d30a-cf63-4c57-8115-decfdd34120a-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-85fb56d756f9m98\" (UID: \"f241d30a-cf63-4c57-8115-decfdd34120a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-85fb56d756f9m98" Jul 07 19:06:56.831092 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:06:56.830839 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f241d30a-cf63-4c57-8115-decfdd34120a-istio-data\") pod \"data-science-gateway-data-science-gateway-class-85fb56d756f9m98\" (UID: \"f241d30a-cf63-4c57-8115-decfdd34120a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-85fb56d756f9m98" Jul 07 19:06:56.831092 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:06:56.830899 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a0d31c0-5fe7-4d3e-9ee3-8266857f6931-istio-data\") pod \"data-science-gateway-data-science-gateway-class-554689689dtjt6l\" (UID: \"1a0d31c0-5fe7-4d3e-9ee3-8266857f6931\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-554689689dtjt6l" Jul 07 19:06:56.831092 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:06:56.830924 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f241d30a-cf63-4c57-8115-decfdd34120a-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-85fb56d756f9m98\" (UID: \"f241d30a-cf63-4c57-8115-decfdd34120a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-85fb56d756f9m98" Jul 07 19:06:56.831251 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:06:56.831089 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f241d30a-cf63-4c57-8115-decfdd34120a-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-85fb56d756f9m98\" (UID: \"f241d30a-cf63-4c57-8115-decfdd34120a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-85fb56d756f9m98" Jul 07 19:06:56.831251 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:06:56.831121 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a0d31c0-5fe7-4d3e-9ee3-8266857f6931-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-554689689dtjt6l\" (UID: \"1a0d31c0-5fe7-4d3e-9ee3-8266857f6931\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-554689689dtjt6l" Jul 07 19:06:56.832346 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:06:56.832321 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a0d31c0-5fe7-4d3e-9ee3-8266857f6931-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-554689689dtjt6l\" (UID: \"1a0d31c0-5fe7-4d3e-9ee3-8266857f6931\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-554689689dtjt6l" Jul 07 19:06:56.832790 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:06:56.832770 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a0d31c0-5fe7-4d3e-9ee3-8266857f6931-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-554689689dtjt6l\" (UID: \"1a0d31c0-5fe7-4d3e-9ee3-8266857f6931\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-554689689dtjt6l" Jul 07 19:06:56.832922 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:06:56.832903 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f241d30a-cf63-4c57-8115-decfdd34120a-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-85fb56d756f9m98\" (UID: \"f241d30a-cf63-4c57-8115-decfdd34120a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-85fb56d756f9m98" Jul 07 19:06:56.833204 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:06:56.833190 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f241d30a-cf63-4c57-8115-decfdd34120a-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-85fb56d756f9m98\" (UID: \"f241d30a-cf63-4c57-8115-decfdd34120a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-85fb56d756f9m98" Jul 07 19:06:56.844488 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:06:56.844457 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5467\" (UniqueName: \"kubernetes.io/projected/f241d30a-cf63-4c57-8115-decfdd34120a-kube-api-access-k5467\") pod \"data-science-gateway-data-science-gateway-class-85fb56d756f9m98\" (UID: \"f241d30a-cf63-4c57-8115-decfdd34120a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-85fb56d756f9m98" Jul 07 19:06:56.844991 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:06:56.844943 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkd59\" (UniqueName: \"kubernetes.io/projected/1a0d31c0-5fe7-4d3e-9ee3-8266857f6931-kube-api-access-lkd59\") pod \"data-science-gateway-data-science-gateway-class-554689689dtjt6l\" (UID: \"1a0d31c0-5fe7-4d3e-9ee3-8266857f6931\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-554689689dtjt6l" Jul 07 19:06:56.847788 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:06:56.847760 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a0d31c0-5fe7-4d3e-9ee3-8266857f6931-istio-token\") pod \"data-science-gateway-data-science-gateway-class-554689689dtjt6l\" (UID: \"1a0d31c0-5fe7-4d3e-9ee3-8266857f6931\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-554689689dtjt6l" Jul 07 19:06:56.850159 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:06:56.850134 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f241d30a-cf63-4c57-8115-decfdd34120a-istio-token\") pod \"data-science-gateway-data-science-gateway-class-85fb56d756f9m98\" (UID: \"f241d30a-cf63-4c57-8115-decfdd34120a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-85fb56d756f9m98" Jul 07 19:06:56.936275 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:06:56.936178 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-85fb56d756f9m98" Jul 07 19:06:56.991169 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:06:56.991132 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:06:57.016340 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:06:57.016304 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-554689689dtjt6l" Jul 07 19:06:57.092067 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:06:57.092021 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:06:57.121269 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:06:57.121237 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-85fb56d756f9m98"] Jul 07 19:06:57.133430 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:06:57.133398 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf241d30a_cf63_4c57_8115_decfdd34120a.slice/crio-42bcd1a249b31eecdf274d43c0dd5545290b75cf73ea6fadb0e020e0d20128a2 WatchSource:0}: Error finding container 42bcd1a249b31eecdf274d43c0dd5545290b75cf73ea6fadb0e020e0d20128a2: Status 404 returned error can't find the container with id 42bcd1a249b31eecdf274d43c0dd5545290b75cf73ea6fadb0e020e0d20128a2 Jul 07 19:06:57.181834 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:06:57.181801 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-85fb56d756f9m98" event={"ID":"f241d30a-cf63-4c57-8115-decfdd34120a","Type":"ContainerStarted","Data":"42bcd1a249b31eecdf274d43c0dd5545290b75cf73ea6fadb0e020e0d20128a2"} Jul 07 19:06:57.194036 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:06:57.194012 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-554689689dtjt6l"] Jul 07 19:06:57.195781 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:06:57.195756 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a0d31c0_5fe7_4d3e_9ee3_8266857f6931.slice/crio-56f4b7aa9a0bd7576af5cddf6485a1639d88198912e55b420e7549726bce79a9 WatchSource:0}: Error finding container 56f4b7aa9a0bd7576af5cddf6485a1639d88198912e55b420e7549726bce79a9: Status 404 returned error can't find the container with id 56f4b7aa9a0bd7576af5cddf6485a1639d88198912e55b420e7549726bce79a9 Jul 07 19:06:58.192667 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:06:58.192616 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-554689689dtjt6l" event={"ID":"1a0d31c0-5fe7-4d3e-9ee3-8266857f6931","Type":"ContainerStarted","Data":"56f4b7aa9a0bd7576af5cddf6485a1639d88198912e55b420e7549726bce79a9"} Jul 07 19:07:00.306224 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:07:00.306181 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 07 19:07:00.306641 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:07:00.306259 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 07 19:07:00.306641 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:07:00.306288 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 07 19:07:00.313104 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:07:00.313078 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 07 19:07:00.313192 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:07:00.313135 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 07 19:07:00.313192 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:07:00.313161 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 07 19:07:01.209215 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:07:01.209175 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-85fb56d756f9m98" event={"ID":"f241d30a-cf63-4c57-8115-decfdd34120a","Type":"ContainerStarted","Data":"0eceffbeeacc3c14724d1c1d1120b39855e07dc10f1bd9f85804c432c2df1a0c"} Jul 07 19:07:01.210608 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:07:01.210582 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-554689689dtjt6l" event={"ID":"1a0d31c0-5fe7-4d3e-9ee3-8266857f6931","Type":"ContainerStarted","Data":"8c5890d57ac60df508955ef5f60d7d3276002eb9631d3a8d801b91617f8c9d08"} Jul 07 19:07:01.308293 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:07:01.308244 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-554689689dtjt6l" podStartSLOduration=2.199953743 podStartE2EDuration="5.308228101s" totalImagesPullingTime="3.108274358s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:06:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:07:01.307598992 +0000 UTC m=+404.323603173" watchObservedRunningTime="2026-07-07 19:07:01.308228101 +0000 UTC m=+404.324232282" Jul 07 19:07:01.308665 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:07:01.308598 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-85fb56d756f9m98" podStartSLOduration=2.131128012 podStartE2EDuration="5.308590546s" totalImagesPullingTime="3.177462534s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:06:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:07:01.258105038 +0000 UTC m=+404.274109218" watchObservedRunningTime="2026-07-07 19:07:01.308590546 +0000 UTC m=+404.324594726" Jul 07 19:07:01.936776 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:07:01.936739 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-85fb56d756f9m98" Jul 07 19:07:01.938033 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:07:01.938005 2563 patch_prober.go:28] interesting pod/data-science-gateway-data-science-gateway-class-85fb56d756f9m98 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.25:15021/healthz/ready\": dial tcp 10.132.0.25:15021: connect: connection refused" start-of-body= Jul 07 19:07:01.938149 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:07:01.938056 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-85fb56d756f9m98" podUID="f241d30a-cf63-4c57-8115-decfdd34120a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.25:15021/healthz/ready\": dial tcp 10.132.0.25:15021: connect: connection refused" Jul 07 19:07:02.016623 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:07:02.016590 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-554689689dtjt6l" Jul 07 19:07:02.021209 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:07:02.021182 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-554689689dtjt6l" Jul 07 19:07:02.215033 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:07:02.214928 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-554689689dtjt6l" Jul 07 19:07:02.215894 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:07:02.215874 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-554689689dtjt6l" Jul 07 19:07:02.374501 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:07:02.374468 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-85fb56d756f9m98"] Jul 07 19:07:02.937134 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:07:02.937093 2563 patch_prober.go:28] interesting pod/data-science-gateway-data-science-gateway-class-85fb56d756f9m98 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.25:15021/healthz/ready\": dial tcp 10.132.0.25:15021: connect: connection refused" start-of-body= Jul 07 19:07:02.937321 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:07:02.937151 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-85fb56d756f9m98" podUID="f241d30a-cf63-4c57-8115-decfdd34120a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.25:15021/healthz/ready\": dial tcp 10.132.0.25:15021: connect: connection refused" Jul 07 19:07:03.235916 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:07:03.235837 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:07:03.354250 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:07:03.354204 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:07:03.368919 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:07:03.368888 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-zl9w5"] Jul 07 19:07:03.377826 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:07:03.377801 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-zl9w5" Jul 07 19:07:03.389692 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:07:03.389665 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Jul 07 19:07:03.389692 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:07:03.389687 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-hmxlc\"" Jul 07 19:07:03.409663 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:07:03.409630 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-zl9w5"] Jul 07 19:07:03.495825 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:07:03.495739 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f71dae8a-03c2-4f96-ad16-895ff28cd975-cert\") pod \"kserve-controller-manager-d6f4b65b8-zl9w5\" (UID: \"f71dae8a-03c2-4f96-ad16-895ff28cd975\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-zl9w5" Jul 07 19:07:03.495825 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:07:03.495813 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g96ng\" (UniqueName: \"kubernetes.io/projected/f71dae8a-03c2-4f96-ad16-895ff28cd975-kube-api-access-g96ng\") pod \"kserve-controller-manager-d6f4b65b8-zl9w5\" (UID: \"f71dae8a-03c2-4f96-ad16-895ff28cd975\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-zl9w5" Jul 07 19:07:03.583465 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:07:03.583425 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:07:03.597345 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:07:03.597317 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g96ng\" (UniqueName: \"kubernetes.io/projected/f71dae8a-03c2-4f96-ad16-895ff28cd975-kube-api-access-g96ng\") pod \"kserve-controller-manager-d6f4b65b8-zl9w5\" (UID: \"f71dae8a-03c2-4f96-ad16-895ff28cd975\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-zl9w5" Jul 07 19:07:03.597497 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:07:03.597378 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f71dae8a-03c2-4f96-ad16-895ff28cd975-cert\") pod \"kserve-controller-manager-d6f4b65b8-zl9w5\" (UID: \"f71dae8a-03c2-4f96-ad16-895ff28cd975\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-zl9w5" Jul 07 19:07:03.599846 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:07:03.599820 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f71dae8a-03c2-4f96-ad16-895ff28cd975-cert\") pod \"kserve-controller-manager-d6f4b65b8-zl9w5\" (UID: \"f71dae8a-03c2-4f96-ad16-895ff28cd975\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-zl9w5" Jul 07 19:07:03.608378 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:07:03.608343 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g96ng\" (UniqueName: \"kubernetes.io/projected/f71dae8a-03c2-4f96-ad16-895ff28cd975-kube-api-access-g96ng\") pod \"kserve-controller-manager-d6f4b65b8-zl9w5\" (UID: \"f71dae8a-03c2-4f96-ad16-895ff28cd975\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-zl9w5" Jul 07 19:07:03.665470 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:07:03.665431 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:07:03.688490 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:07:03.688454 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-zl9w5" Jul 07 19:07:03.842245 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:07:03.842215 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-zl9w5"] Jul 07 19:07:03.844238 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:07:03.844207 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf71dae8a_03c2_4f96_ad16_895ff28cd975.slice/crio-4cbf2daaf06c357cf6e9a972e68f26547084bc5d90e6d16b145e80a0d7594854 WatchSource:0}: Error finding container 4cbf2daaf06c357cf6e9a972e68f26547084bc5d90e6d16b145e80a0d7594854: Status 404 returned error can't find the container with id 4cbf2daaf06c357cf6e9a972e68f26547084bc5d90e6d16b145e80a0d7594854 Jul 07 19:07:03.936877 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:07:03.936839 2563 patch_prober.go:28] interesting pod/data-science-gateway-data-science-gateway-class-85fb56d756f9m98 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.25:15021/healthz/ready\": dial tcp 10.132.0.25:15021: connect: connection refused" start-of-body= Jul 07 19:07:03.937092 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:07:03.936909 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-85fb56d756f9m98" podUID="f241d30a-cf63-4c57-8115-decfdd34120a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.25:15021/healthz/ready\": dial tcp 10.132.0.25:15021: connect: connection refused" Jul 07 19:07:04.170591 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:07:04.170561 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-5f4bfc584b-jcd55" Jul 07 19:07:04.224891 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:07:04.224849 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-zl9w5" event={"ID":"f71dae8a-03c2-4f96-ad16-895ff28cd975","Type":"ContainerStarted","Data":"4cbf2daaf06c357cf6e9a972e68f26547084bc5d90e6d16b145e80a0d7594854"} Jul 07 19:07:04.225089 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:07:04.224943 2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-85fb56d756f9m98" podUID="f241d30a-cf63-4c57-8115-decfdd34120a" containerName="istio-proxy" containerID="cri-o://0eceffbeeacc3c14724d1c1d1120b39855e07dc10f1bd9f85804c432c2df1a0c" gracePeriod=30 Jul 07 19:07:04.744825 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:07:04.744778 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:07:04.857842 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:07:04.857803 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:07:04.963915 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:07:04.963881 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:07:05.985270 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:07:05.985236 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:07:06.101426 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:07:06.101387 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:07:07.240780 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:07:07.240741 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-zl9w5" event={"ID":"f71dae8a-03c2-4f96-ad16-895ff28cd975","Type":"ContainerStarted","Data":"2cb6eebfcb98e1c32ba3827fce37d493542d3981c4f45a7c153943ab851967bc"} Jul 07 19:07:07.241311 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:07:07.240857 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-zl9w5" Jul 07 19:07:07.273882 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:07:07.273831 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-zl9w5" podStartSLOduration=1.495879911 podStartE2EDuration="4.273812635s" totalImagesPullingTime="2.777932724s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:07:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:07:07.271290439 +0000 UTC m=+410.287294621" watchObservedRunningTime="2026-07-07 19:07:07.273812635 +0000 UTC m=+410.289816815" Jul 07 19:07:09.485857 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:07:09.485795 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-85fb56d756f9m98" Jul 07 19:07:09.555938 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:07:09.555906 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f241d30a-cf63-4c57-8115-decfdd34120a-istiod-ca-cert\") pod \"f241d30a-cf63-4c57-8115-decfdd34120a\" (UID: \"f241d30a-cf63-4c57-8115-decfdd34120a\") " Jul 07 19:07:09.556157 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:07:09.555970 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f241d30a-cf63-4c57-8115-decfdd34120a-istio-data\") pod \"f241d30a-cf63-4c57-8115-decfdd34120a\" (UID: \"f241d30a-cf63-4c57-8115-decfdd34120a\") " Jul 07 19:07:09.556157 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:07:09.555997 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f241d30a-cf63-4c57-8115-decfdd34120a-istio-envoy\") pod \"f241d30a-cf63-4c57-8115-decfdd34120a\" (UID: \"f241d30a-cf63-4c57-8115-decfdd34120a\") " Jul 07 19:07:09.556157 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:07:09.556015 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f241d30a-cf63-4c57-8115-decfdd34120a-istio-podinfo\") pod \"f241d30a-cf63-4c57-8115-decfdd34120a\" (UID: \"f241d30a-cf63-4c57-8115-decfdd34120a\") " Jul 07 19:07:09.556157 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:07:09.556050 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k5467\" (UniqueName: \"kubernetes.io/projected/f241d30a-cf63-4c57-8115-decfdd34120a-kube-api-access-k5467\") pod \"f241d30a-cf63-4c57-8115-decfdd34120a\" (UID: \"f241d30a-cf63-4c57-8115-decfdd34120a\") " Jul 07 19:07:09.556157 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:07:09.556084 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f241d30a-cf63-4c57-8115-decfdd34120a-credential-socket\") pod \"f241d30a-cf63-4c57-8115-decfdd34120a\" (UID: \"f241d30a-cf63-4c57-8115-decfdd34120a\") " Jul 07 19:07:09.556157 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:07:09.556134 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f241d30a-cf63-4c57-8115-decfdd34120a-workload-certs\") pod \"f241d30a-cf63-4c57-8115-decfdd34120a\" (UID: \"f241d30a-cf63-4c57-8115-decfdd34120a\") " Jul 07 19:07:09.556473 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:07:09.556182 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f241d30a-cf63-4c57-8115-decfdd34120a-istio-token\") pod \"f241d30a-cf63-4c57-8115-decfdd34120a\" (UID: \"f241d30a-cf63-4c57-8115-decfdd34120a\") " Jul 07 19:07:09.556473 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:07:09.556206 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f241d30a-cf63-4c57-8115-decfdd34120a-workload-socket\") pod \"f241d30a-cf63-4c57-8115-decfdd34120a\" (UID: \"f241d30a-cf63-4c57-8115-decfdd34120a\") " Jul 07 19:07:09.556473 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:07:09.556421 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f241d30a-cf63-4c57-8115-decfdd34120a-istio-data" (OuterVolumeSpecName: "istio-data") pod "f241d30a-cf63-4c57-8115-decfdd34120a" (UID: "f241d30a-cf63-4c57-8115-decfdd34120a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:07:09.556626 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:07:09.556473 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f241d30a-cf63-4c57-8115-decfdd34120a-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "f241d30a-cf63-4c57-8115-decfdd34120a" (UID: "f241d30a-cf63-4c57-8115-decfdd34120a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 19:07:09.556681 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:07:09.556638 2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f241d30a-cf63-4c57-8115-decfdd34120a-istiod-ca-cert\") on node \"ip-10-0-142-134.ec2.internal\" DevicePath \"\"" Jul 07 19:07:09.556681 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:07:09.556659 2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f241d30a-cf63-4c57-8115-decfdd34120a-istio-data\") on node \"ip-10-0-142-134.ec2.internal\" DevicePath \"\"" Jul 07 19:07:09.556681 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:07:09.556649 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f241d30a-cf63-4c57-8115-decfdd34120a-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "f241d30a-cf63-4c57-8115-decfdd34120a" (UID: "f241d30a-cf63-4c57-8115-decfdd34120a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:07:09.556798 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:07:09.556684 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f241d30a-cf63-4c57-8115-decfdd34120a-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "f241d30a-cf63-4c57-8115-decfdd34120a" (UID: "f241d30a-cf63-4c57-8115-decfdd34120a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:07:09.556798 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:07:09.556774 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f241d30a-cf63-4c57-8115-decfdd34120a-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "f241d30a-cf63-4c57-8115-decfdd34120a" (UID: "f241d30a-cf63-4c57-8115-decfdd34120a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:07:09.558549 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:07:09.558526 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f241d30a-cf63-4c57-8115-decfdd34120a-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "f241d30a-cf63-4c57-8115-decfdd34120a" (UID: "f241d30a-cf63-4c57-8115-decfdd34120a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:07:09.558666 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:07:09.558549 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f241d30a-cf63-4c57-8115-decfdd34120a-istio-token" (OuterVolumeSpecName: "istio-token") pod "f241d30a-cf63-4c57-8115-decfdd34120a" (UID: "f241d30a-cf63-4c57-8115-decfdd34120a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 19:07:09.558666 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:07:09.558645 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f241d30a-cf63-4c57-8115-decfdd34120a-kube-api-access-k5467" (OuterVolumeSpecName: "kube-api-access-k5467") pod "f241d30a-cf63-4c57-8115-decfdd34120a" (UID: "f241d30a-cf63-4c57-8115-decfdd34120a"). InnerVolumeSpecName "kube-api-access-k5467". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 19:07:09.558736 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:07:09.558648 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f241d30a-cf63-4c57-8115-decfdd34120a-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "f241d30a-cf63-4c57-8115-decfdd34120a" (UID: "f241d30a-cf63-4c57-8115-decfdd34120a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 07 19:07:09.657364 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:07:09.657326 2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f241d30a-cf63-4c57-8115-decfdd34120a-istio-envoy\") on node \"ip-10-0-142-134.ec2.internal\" DevicePath \"\"" Jul 07 19:07:09.657364 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:07:09.657356 2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f241d30a-cf63-4c57-8115-decfdd34120a-istio-podinfo\") on node \"ip-10-0-142-134.ec2.internal\" DevicePath \"\"" Jul 07 19:07:09.657364 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:07:09.657367 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k5467\" (UniqueName: \"kubernetes.io/projected/f241d30a-cf63-4c57-8115-decfdd34120a-kube-api-access-k5467\") on node \"ip-10-0-142-134.ec2.internal\" DevicePath \"\"" Jul 07 19:07:09.657599 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:07:09.657376 2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f241d30a-cf63-4c57-8115-decfdd34120a-credential-socket\") on node \"ip-10-0-142-134.ec2.internal\" DevicePath \"\"" Jul 07 19:07:09.657599 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:07:09.657386 2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f241d30a-cf63-4c57-8115-decfdd34120a-workload-certs\") on node \"ip-10-0-142-134.ec2.internal\" DevicePath \"\"" Jul 07 19:07:09.657599 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:07:09.657394 2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f241d30a-cf63-4c57-8115-decfdd34120a-istio-token\") on node \"ip-10-0-142-134.ec2.internal\" DevicePath \"\"" Jul 07 19:07:09.657599 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:07:09.657402 2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f241d30a-cf63-4c57-8115-decfdd34120a-workload-socket\") on node \"ip-10-0-142-134.ec2.internal\" DevicePath \"\"" Jul 07 19:07:10.255757 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:07:10.255721 2563 generic.go:356] "Generic (PLEG): container finished" podID="f241d30a-cf63-4c57-8115-decfdd34120a" containerID="0eceffbeeacc3c14724d1c1d1120b39855e07dc10f1bd9f85804c432c2df1a0c" exitCode=0 Jul 07 19:07:10.255988 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:07:10.255776 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-85fb56d756f9m98" Jul 07 19:07:10.255988 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:07:10.255808 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-85fb56d756f9m98" event={"ID":"f241d30a-cf63-4c57-8115-decfdd34120a","Type":"ContainerDied","Data":"0eceffbeeacc3c14724d1c1d1120b39855e07dc10f1bd9f85804c432c2df1a0c"} Jul 07 19:07:10.255988 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:07:10.255860 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-85fb56d756f9m98" event={"ID":"f241d30a-cf63-4c57-8115-decfdd34120a","Type":"ContainerDied","Data":"42bcd1a249b31eecdf274d43c0dd5545290b75cf73ea6fadb0e020e0d20128a2"} Jul 07 19:07:10.255988 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:07:10.255885 2563 scope.go:117] "RemoveContainer" containerID="0eceffbeeacc3c14724d1c1d1120b39855e07dc10f1bd9f85804c432c2df1a0c" Jul 07 19:07:10.266549 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:07:10.266511 2563 scope.go:117] "RemoveContainer" containerID="0eceffbeeacc3c14724d1c1d1120b39855e07dc10f1bd9f85804c432c2df1a0c" Jul 07 19:07:10.266892 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:07:10.266873 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0eceffbeeacc3c14724d1c1d1120b39855e07dc10f1bd9f85804c432c2df1a0c\": container with ID starting with 0eceffbeeacc3c14724d1c1d1120b39855e07dc10f1bd9f85804c432c2df1a0c not found: ID does not exist" containerID="0eceffbeeacc3c14724d1c1d1120b39855e07dc10f1bd9f85804c432c2df1a0c" Jul 07 19:07:10.266994 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:07:10.266899 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0eceffbeeacc3c14724d1c1d1120b39855e07dc10f1bd9f85804c432c2df1a0c"} err="failed to get container status \"0eceffbeeacc3c14724d1c1d1120b39855e07dc10f1bd9f85804c432c2df1a0c\": rpc error: code = NotFound desc = could not find container \"0eceffbeeacc3c14724d1c1d1120b39855e07dc10f1bd9f85804c432c2df1a0c\": container with ID starting with 0eceffbeeacc3c14724d1c1d1120b39855e07dc10f1bd9f85804c432c2df1a0c not found: ID does not exist" Jul 07 19:07:10.300785 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:07:10.300753 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-85fb56d756f9m98"] Jul 07 19:07:10.307205 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:07:10.307178 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-85fb56d756f9m98"] Jul 07 19:07:11.476487 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:07:11.476454 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f241d30a-cf63-4c57-8115-decfdd34120a" path="/var/lib/kubelet/pods/f241d30a-cf63-4c57-8115-decfdd34120a/volumes" Jul 07 19:07:38.250142 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:07:38.250107 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-zl9w5" Jul 07 19:07:50.690542 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:07:50.690458 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:07:50.821060 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:07:50.821011 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:07:50.876779 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:07:50.876744 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-msh5h"] Jul 07 19:07:50.877141 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:07:50.877128 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f241d30a-cf63-4c57-8115-decfdd34120a" containerName="istio-proxy" Jul 07 19:07:50.877194 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:07:50.877143 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="f241d30a-cf63-4c57-8115-decfdd34120a" containerName="istio-proxy" Jul 07 19:07:50.877230 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:07:50.877208 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="f241d30a-cf63-4c57-8115-decfdd34120a" containerName="istio-proxy" Jul 07 19:07:50.882507 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:07:50.882491 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-msh5h" Jul 07 19:07:50.885172 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:07:50.885146 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 07 19:07:50.885334 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:07:50.885159 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-rbvgj\"" Jul 07 19:07:50.885334 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:07:50.885297 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 07 19:07:50.896927 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:07:50.896899 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-msh5h"] Jul 07 19:07:50.934414 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:07:50.934376 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jxv9w\" (UniqueName: \"kubernetes.io/projected/b51ae0b9-d4e7-43c8-a1ad-4041f88b0eea-kube-api-access-jxv9w\") pod \"kuadrant-operator-catalog-msh5h\" (UID: \"b51ae0b9-d4e7-43c8-a1ad-4041f88b0eea\") " pod="kuadrant-system/kuadrant-operator-catalog-msh5h" Jul 07 19:07:50.955270 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:07:50.955179 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:07:51.035364 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:07:51.035329 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxv9w\" (UniqueName: \"kubernetes.io/projected/b51ae0b9-d4e7-43c8-a1ad-4041f88b0eea-kube-api-access-jxv9w\") pod \"kuadrant-operator-catalog-msh5h\" (UID: \"b51ae0b9-d4e7-43c8-a1ad-4041f88b0eea\") " pod="kuadrant-system/kuadrant-operator-catalog-msh5h" Jul 07 19:07:51.046488 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:07:51.046463 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxv9w\" (UniqueName: \"kubernetes.io/projected/b51ae0b9-d4e7-43c8-a1ad-4041f88b0eea-kube-api-access-jxv9w\") pod \"kuadrant-operator-catalog-msh5h\" (UID: \"b51ae0b9-d4e7-43c8-a1ad-4041f88b0eea\") " pod="kuadrant-system/kuadrant-operator-catalog-msh5h" Jul 07 19:07:51.180765 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:07:51.180723 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-msh5h"] Jul 07 19:07:51.181089 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:07:51.181061 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-msh5h" Jul 07 19:07:51.316419 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:07:51.316386 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-msh5h"] Jul 07 19:07:51.318961 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:07:51.318914 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb51ae0b9_d4e7_43c8_a1ad_4041f88b0eea.slice/crio-c4ccff215555f71f7b35e457a731d6f4231bcd59cd27a1e84e331d4343fd83b1 WatchSource:0}: Error finding container c4ccff215555f71f7b35e457a731d6f4231bcd59cd27a1e84e331d4343fd83b1: Status 404 returned error can't find the container with id c4ccff215555f71f7b35e457a731d6f4231bcd59cd27a1e84e331d4343fd83b1 Jul 07 19:07:51.392598 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:07:51.392567 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-wfgtm"] Jul 07 19:07:51.398050 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:07:51.398033 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-wfgtm" Jul 07 19:07:51.407585 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:07:51.407560 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-wfgtm"] Jul 07 19:07:51.434930 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:07:51.434892 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-msh5h" event={"ID":"b51ae0b9-d4e7-43c8-a1ad-4041f88b0eea","Type":"ContainerStarted","Data":"c4ccff215555f71f7b35e457a731d6f4231bcd59cd27a1e84e331d4343fd83b1"} Jul 07 19:07:51.540740 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:07:51.540711 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wtt7p\" (UniqueName: \"kubernetes.io/projected/658ee1a1-f9fa-4fb6-ab7f-8acdab870d80-kube-api-access-wtt7p\") pod \"kuadrant-operator-catalog-wfgtm\" (UID: \"658ee1a1-f9fa-4fb6-ab7f-8acdab870d80\") " pod="kuadrant-system/kuadrant-operator-catalog-wfgtm" Jul 07 19:07:51.641707 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:07:51.641666 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wtt7p\" (UniqueName: \"kubernetes.io/projected/658ee1a1-f9fa-4fb6-ab7f-8acdab870d80-kube-api-access-wtt7p\") pod \"kuadrant-operator-catalog-wfgtm\" (UID: \"658ee1a1-f9fa-4fb6-ab7f-8acdab870d80\") " pod="kuadrant-system/kuadrant-operator-catalog-wfgtm" Jul 07 19:07:51.651410 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:07:51.651373 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wtt7p\" (UniqueName: \"kubernetes.io/projected/658ee1a1-f9fa-4fb6-ab7f-8acdab870d80-kube-api-access-wtt7p\") pod \"kuadrant-operator-catalog-wfgtm\" (UID: \"658ee1a1-f9fa-4fb6-ab7f-8acdab870d80\") " pod="kuadrant-system/kuadrant-operator-catalog-wfgtm" Jul 07 19:07:51.708060 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:07:51.708022 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-wfgtm" Jul 07 19:07:51.875783 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:07:51.875756 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-wfgtm"] Jul 07 19:07:51.878431 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:07:51.878405 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod658ee1a1_f9fa_4fb6_ab7f_8acdab870d80.slice/crio-4c406c04fad420466a7000da6565e17ba160028d8778c2a9e29a75c791581164 WatchSource:0}: Error finding container 4c406c04fad420466a7000da6565e17ba160028d8778c2a9e29a75c791581164: Status 404 returned error can't find the container with id 4c406c04fad420466a7000da6565e17ba160028d8778c2a9e29a75c791581164 Jul 07 19:07:52.441586 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:07:52.441552 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-wfgtm" event={"ID":"658ee1a1-f9fa-4fb6-ab7f-8acdab870d80","Type":"ContainerStarted","Data":"4c406c04fad420466a7000da6565e17ba160028d8778c2a9e29a75c791581164"} Jul 07 19:07:54.460066 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:07:54.460028 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-msh5h" event={"ID":"b51ae0b9-d4e7-43c8-a1ad-4041f88b0eea","Type":"ContainerStarted","Data":"4c696c8748ea8f56427e693019bf08c31de3abf051d208a2f6e5cf7c445993bc"} Jul 07 19:07:54.460531 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:07:54.460111 2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-msh5h" podUID="b51ae0b9-d4e7-43c8-a1ad-4041f88b0eea" containerName="registry-server" containerID="cri-o://4c696c8748ea8f56427e693019bf08c31de3abf051d208a2f6e5cf7c445993bc" gracePeriod=2 Jul 07 19:07:54.461587 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:07:54.461556 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-wfgtm" event={"ID":"658ee1a1-f9fa-4fb6-ab7f-8acdab870d80","Type":"ContainerStarted","Data":"7c731fe12dff53780b4ca2cd3ec91dd84396c4679443c1bbb601ff56c5b435ba"} Jul 07 19:07:54.476309 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:07:54.476252 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-msh5h" podStartSLOduration=1.7901928489999999 podStartE2EDuration="4.476222822s" totalImagesPullingTime="2.686029973s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:07:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:07:54.475686299 +0000 UTC m=+457.491690483" watchObservedRunningTime="2026-07-07 19:07:54.476222822 +0000 UTC m=+457.492227001" Jul 07 19:07:54.492244 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:07:54.492196 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-wfgtm" podStartSLOduration=1.369761309 podStartE2EDuration="3.492180907s" totalImagesPullingTime="2.122419598s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:07:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:07:54.491915645 +0000 UTC m=+457.507919828" watchObservedRunningTime="2026-07-07 19:07:54.492180907 +0000 UTC m=+457.508185087" Jul 07 19:07:54.712726 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:07:54.712659 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-msh5h" Jul 07 19:07:54.771540 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:07:54.771504 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jxv9w\" (UniqueName: \"kubernetes.io/projected/b51ae0b9-d4e7-43c8-a1ad-4041f88b0eea-kube-api-access-jxv9w\") pod \"b51ae0b9-d4e7-43c8-a1ad-4041f88b0eea\" (UID: \"b51ae0b9-d4e7-43c8-a1ad-4041f88b0eea\") " Jul 07 19:07:54.773641 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:07:54.773613 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b51ae0b9-d4e7-43c8-a1ad-4041f88b0eea-kube-api-access-jxv9w" (OuterVolumeSpecName: "kube-api-access-jxv9w") pod "b51ae0b9-d4e7-43c8-a1ad-4041f88b0eea" (UID: "b51ae0b9-d4e7-43c8-a1ad-4041f88b0eea"). InnerVolumeSpecName "kube-api-access-jxv9w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 19:07:54.872501 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:07:54.872458 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jxv9w\" (UniqueName: \"kubernetes.io/projected/b51ae0b9-d4e7-43c8-a1ad-4041f88b0eea-kube-api-access-jxv9w\") on node \"ip-10-0-142-134.ec2.internal\" DevicePath \"\"" Jul 07 19:07:55.467529 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:07:55.467491 2563 generic.go:356] "Generic (PLEG): container finished" podID="b51ae0b9-d4e7-43c8-a1ad-4041f88b0eea" containerID="4c696c8748ea8f56427e693019bf08c31de3abf051d208a2f6e5cf7c445993bc" exitCode=0 Jul 07 19:07:55.467913 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:07:55.467541 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-msh5h" Jul 07 19:07:55.467913 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:07:55.467580 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-msh5h" event={"ID":"b51ae0b9-d4e7-43c8-a1ad-4041f88b0eea","Type":"ContainerDied","Data":"4c696c8748ea8f56427e693019bf08c31de3abf051d208a2f6e5cf7c445993bc"} Jul 07 19:07:55.467913 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:07:55.467628 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-msh5h" event={"ID":"b51ae0b9-d4e7-43c8-a1ad-4041f88b0eea","Type":"ContainerDied","Data":"c4ccff215555f71f7b35e457a731d6f4231bcd59cd27a1e84e331d4343fd83b1"} Jul 07 19:07:55.467913 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:07:55.467648 2563 scope.go:117] "RemoveContainer" containerID="4c696c8748ea8f56427e693019bf08c31de3abf051d208a2f6e5cf7c445993bc" Jul 07 19:07:55.476590 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:07:55.476569 2563 scope.go:117] "RemoveContainer" containerID="4c696c8748ea8f56427e693019bf08c31de3abf051d208a2f6e5cf7c445993bc" Jul 07 19:07:55.476983 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:07:55.476933 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c696c8748ea8f56427e693019bf08c31de3abf051d208a2f6e5cf7c445993bc\": container with ID starting with 4c696c8748ea8f56427e693019bf08c31de3abf051d208a2f6e5cf7c445993bc not found: ID does not exist" containerID="4c696c8748ea8f56427e693019bf08c31de3abf051d208a2f6e5cf7c445993bc" Jul 07 19:07:55.477074 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:07:55.476991 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c696c8748ea8f56427e693019bf08c31de3abf051d208a2f6e5cf7c445993bc"} err="failed to get container status \"4c696c8748ea8f56427e693019bf08c31de3abf051d208a2f6e5cf7c445993bc\": rpc error: code = NotFound desc = could not find container \"4c696c8748ea8f56427e693019bf08c31de3abf051d208a2f6e5cf7c445993bc\": container with ID starting with 4c696c8748ea8f56427e693019bf08c31de3abf051d208a2f6e5cf7c445993bc not found: ID does not exist" Jul 07 19:07:55.489758 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:07:55.489732 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-msh5h"] Jul 07 19:07:55.494038 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:07:55.494018 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-msh5h"] Jul 07 19:07:57.475887 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:07:57.475852 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b51ae0b9-d4e7-43c8-a1ad-4041f88b0eea" path="/var/lib/kubelet/pods/b51ae0b9-d4e7-43c8-a1ad-4041f88b0eea/volumes" Jul 07 19:08:01.709089 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:08:01.709050 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-wfgtm" Jul 07 19:08:01.709498 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:08:01.709151 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-wfgtm" Jul 07 19:08:01.731787 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:08:01.731751 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-wfgtm" Jul 07 19:08:02.382320 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:08:02.382293 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:08:02.466781 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:08:02.466750 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:08:02.520334 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:08:02.520310 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-wfgtm" Jul 07 19:08:14.201412 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:08:14.201371 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:08:17.447308 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:08:17.447278 2563 scope.go:117] "RemoveContainer" containerID="b3ac1a1f7e51d666caae9cbc9ad675059e92224b8c30da56be03b2ee983d4f80" Jul 07 19:08:17.893603 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:08:17.893569 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-tl2qp"] Jul 07 19:08:17.893940 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:08:17.893926 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b51ae0b9-d4e7-43c8-a1ad-4041f88b0eea" containerName="registry-server" Jul 07 19:08:17.894006 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:08:17.893941 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="b51ae0b9-d4e7-43c8-a1ad-4041f88b0eea" containerName="registry-server" Jul 07 19:08:17.894044 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:08:17.894026 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="b51ae0b9-d4e7-43c8-a1ad-4041f88b0eea" containerName="registry-server" Jul 07 19:08:17.901837 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:08:17.901817 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-tl2qp" Jul 07 19:08:17.907628 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:08:17.907605 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-jwvmn\"" Jul 07 19:08:17.921399 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:08:17.921375 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-tl2qp"] Jul 07 19:08:17.972500 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:08:17.972463 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxcg4\" (UniqueName: \"kubernetes.io/projected/7ca86793-5227-42be-84de-b06e8680755a-kube-api-access-xxcg4\") pod \"limitador-operator-controller-manager-859d7b78bd-tl2qp\" (UID: \"7ca86793-5227-42be-84de-b06e8680755a\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-tl2qp" Jul 07 19:08:18.073674 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:08:18.073645 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxcg4\" (UniqueName: \"kubernetes.io/projected/7ca86793-5227-42be-84de-b06e8680755a-kube-api-access-xxcg4\") pod \"limitador-operator-controller-manager-859d7b78bd-tl2qp\" (UID: \"7ca86793-5227-42be-84de-b06e8680755a\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-tl2qp" Jul 07 19:08:18.098396 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:08:18.098372 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxcg4\" (UniqueName: \"kubernetes.io/projected/7ca86793-5227-42be-84de-b06e8680755a-kube-api-access-xxcg4\") pod \"limitador-operator-controller-manager-859d7b78bd-tl2qp\" (UID: \"7ca86793-5227-42be-84de-b06e8680755a\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-tl2qp" Jul 07 19:08:18.211718 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:08:18.211632 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-tl2qp" Jul 07 19:08:18.354563 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:08:18.354529 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-tl2qp"] Jul 07 19:08:18.357766 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:08:18.357741 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ca86793_5227_42be_84de_b06e8680755a.slice/crio-c05f9cf1543dcb1de0ef7fc82b9e0ba5de095be8de265e1ec8a2325279fc34a4 WatchSource:0}: Error finding container c05f9cf1543dcb1de0ef7fc82b9e0ba5de095be8de265e1ec8a2325279fc34a4: Status 404 returned error can't find the container with id c05f9cf1543dcb1de0ef7fc82b9e0ba5de095be8de265e1ec8a2325279fc34a4 Jul 07 19:08:18.570856 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:08:18.570821 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-tl2qp" event={"ID":"7ca86793-5227-42be-84de-b06e8680755a","Type":"ContainerStarted","Data":"c05f9cf1543dcb1de0ef7fc82b9e0ba5de095be8de265e1ec8a2325279fc34a4"} Jul 07 19:08:18.826274 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:08:18.826177 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:08:20.584075 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:08:20.584039 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-tl2qp" event={"ID":"7ca86793-5227-42be-84de-b06e8680755a","Type":"ContainerStarted","Data":"5a443707c5421a45d008547efaf2e17c791f13921eed0a7a48d2d7ae0598059f"} Jul 07 19:08:20.584567 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:08:20.584110 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-tl2qp" Jul 07 19:08:20.605040 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:08:20.604990 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-tl2qp" podStartSLOduration=2.059734742 podStartE2EDuration="3.604975171s" totalImagesPullingTime="1.545240429s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:08:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:08:20.603289525 +0000 UTC m=+483.619293705" watchObservedRunningTime="2026-07-07 19:08:20.604975171 +0000 UTC m=+483.620979344" Jul 07 19:08:25.690333 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:08:25.690296 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:08:27.645549 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:08:27.645519 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-rhd4d"] Jul 07 19:08:27.649738 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:08:27.649717 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-rhd4d" Jul 07 19:08:27.652471 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:08:27.652445 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-td48p\"" Jul 07 19:08:27.652557 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:08:27.652498 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Jul 07 19:08:27.667124 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:08:27.667099 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-rhd4d"] Jul 07 19:08:27.767405 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:08:27.767376 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdvdd\" (UniqueName: \"kubernetes.io/projected/86db6a78-52ad-429e-a6df-2a623f06bf6b-kube-api-access-xdvdd\") pod \"dns-operator-controller-manager-864dd78c4c-rhd4d\" (UID: \"86db6a78-52ad-429e-a6df-2a623f06bf6b\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-rhd4d" Jul 07 19:08:27.868346 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:08:27.868307 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdvdd\" (UniqueName: \"kubernetes.io/projected/86db6a78-52ad-429e-a6df-2a623f06bf6b-kube-api-access-xdvdd\") pod \"dns-operator-controller-manager-864dd78c4c-rhd4d\" (UID: \"86db6a78-52ad-429e-a6df-2a623f06bf6b\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-rhd4d" Jul 07 19:08:27.885099 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:08:27.885070 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdvdd\" (UniqueName: \"kubernetes.io/projected/86db6a78-52ad-429e-a6df-2a623f06bf6b-kube-api-access-xdvdd\") pod \"dns-operator-controller-manager-864dd78c4c-rhd4d\" (UID: \"86db6a78-52ad-429e-a6df-2a623f06bf6b\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-rhd4d" Jul 07 19:08:27.959591 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:08:27.959510 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-rhd4d" Jul 07 19:08:28.114750 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:08:28.114714 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-rhd4d"] Jul 07 19:08:28.117925 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:08:28.117898 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86db6a78_52ad_429e_a6df_2a623f06bf6b.slice/crio-ac96a6af966f0ef718aa0e6f8d1a71331bc47f967bdeb8fa9a7d29e59b444f73 WatchSource:0}: Error finding container ac96a6af966f0ef718aa0e6f8d1a71331bc47f967bdeb8fa9a7d29e59b444f73: Status 404 returned error can't find the container with id ac96a6af966f0ef718aa0e6f8d1a71331bc47f967bdeb8fa9a7d29e59b444f73 Jul 07 19:08:28.209460 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:08:28.209425 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:08:28.621373 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:08:28.621334 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-rhd4d" event={"ID":"86db6a78-52ad-429e-a6df-2a623f06bf6b","Type":"ContainerStarted","Data":"ac96a6af966f0ef718aa0e6f8d1a71331bc47f967bdeb8fa9a7d29e59b444f73"} Jul 07 19:08:29.044355 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:08:29.044319 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:08:29.550947 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:08:29.550907 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:08:30.632204 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:08:30.632163 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-rhd4d" event={"ID":"86db6a78-52ad-429e-a6df-2a623f06bf6b","Type":"ContainerStarted","Data":"859d3c7699911927801b425fa02f9b226055130541bf0384703559b5d31f0782"} Jul 07 19:08:30.632625 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:08:30.632304 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-rhd4d" Jul 07 19:08:30.665456 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:08:30.665392 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-rhd4d" podStartSLOduration=1.348108791 podStartE2EDuration="3.665367553s" totalImagesPullingTime="2.317258762s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:08:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:08:30.660826543 +0000 UTC m=+493.676830726" watchObservedRunningTime="2026-07-07 19:08:30.665367553 +0000 UTC m=+493.681371750" Jul 07 19:08:30.940022 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:08:30.939984 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:08:31.590810 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:08:31.590779 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-tl2qp" Jul 07 19:08:31.604416 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:08:31.604385 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:08:31.753916 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:08:31.753884 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:08:31.901264 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:08:31.901186 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-2tvsm"] Jul 07 19:08:31.904657 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:08:31.904641 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-2tvsm" Jul 07 19:08:31.906626 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:08:31.906608 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-czfgj\"" Jul 07 19:08:31.915865 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:08:31.915846 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-2tvsm"] Jul 07 19:08:32.011345 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:08:32.011308 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5mrz\" (UniqueName: \"kubernetes.io/projected/f242cd57-28f2-4e5b-917b-e47794e9b26e-kube-api-access-k5mrz\") pod \"authorino-operator-75cc8d5cb7-2tvsm\" (UID: \"f242cd57-28f2-4e5b-917b-e47794e9b26e\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-2tvsm" Jul 07 19:08:32.112548 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:08:32.112513 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5mrz\" (UniqueName: \"kubernetes.io/projected/f242cd57-28f2-4e5b-917b-e47794e9b26e-kube-api-access-k5mrz\") pod \"authorino-operator-75cc8d5cb7-2tvsm\" (UID: \"f242cd57-28f2-4e5b-917b-e47794e9b26e\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-2tvsm" Jul 07 19:08:32.122579 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:08:32.122548 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5mrz\" (UniqueName: \"kubernetes.io/projected/f242cd57-28f2-4e5b-917b-e47794e9b26e-kube-api-access-k5mrz\") pod \"authorino-operator-75cc8d5cb7-2tvsm\" (UID: \"f242cd57-28f2-4e5b-917b-e47794e9b26e\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-2tvsm" Jul 07 19:08:32.214995 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:08:32.214885 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-2tvsm" Jul 07 19:08:32.355429 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:08:32.355393 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-2tvsm"] Jul 07 19:08:32.358150 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:08:32.358119 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf242cd57_28f2_4e5b_917b_e47794e9b26e.slice/crio-d197791f7f314fc94ca959f4ade0a66b390b12c9d5b86425f4a73f3617dc7d1d WatchSource:0}: Error finding container d197791f7f314fc94ca959f4ade0a66b390b12c9d5b86425f4a73f3617dc7d1d: Status 404 returned error can't find the container with id d197791f7f314fc94ca959f4ade0a66b390b12c9d5b86425f4a73f3617dc7d1d Jul 07 19:08:32.643777 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:08:32.643740 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-2tvsm" event={"ID":"f242cd57-28f2-4e5b-917b-e47794e9b26e","Type":"ContainerStarted","Data":"d197791f7f314fc94ca959f4ade0a66b390b12c9d5b86425f4a73f3617dc7d1d"} Jul 07 19:08:32.745029 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:08:32.744984 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:08:34.290814 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:08:34.290770 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:08:34.405384 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:08:34.405353 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:08:34.655501 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:08:34.655416 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-2tvsm" event={"ID":"f242cd57-28f2-4e5b-917b-e47794e9b26e","Type":"ContainerStarted","Data":"71128d8e13ed384a682e86b99d7b5b19b4808151abc90b77525f173df3ed5e22"} Jul 07 19:08:34.655650 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:08:34.655568 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-2tvsm" Jul 07 19:08:35.304499 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:08:35.304469 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:08:35.417507 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:08:35.417464 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:08:38.095529 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:08:38.095493 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:08:38.278453 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:08:38.278416 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:08:41.640618 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:08:41.640588 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-rhd4d" Jul 07 19:08:41.673668 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:08:41.673605 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-75cc8d5cb7-2tvsm" podStartSLOduration=8.571574524 podStartE2EDuration="10.673586175s" totalImagesPullingTime="2.102011651s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:08:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:08:34.675907852 +0000 UTC m=+497.691912035" watchObservedRunningTime="2026-07-07 19:08:41.673586175 +0000 UTC m=+504.689590356" Jul 07 19:08:42.096788 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:08:42.096754 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-tl2qp"] Jul 07 19:08:42.097010 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:08:42.096944 2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-tl2qp" podUID="7ca86793-5227-42be-84de-b06e8680755a" containerName="manager" containerID="cri-o://5a443707c5421a45d008547efaf2e17c791f13921eed0a7a48d2d7ae0598059f" gracePeriod=2 Jul 07 19:08:42.110048 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:08:42.110018 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-tl2qp"] Jul 07 19:08:42.131450 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:08:42.131423 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-q9d5b"] Jul 07 19:08:42.131809 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:08:42.131794 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ca86793-5227-42be-84de-b06e8680755a" containerName="manager" Jul 07 19:08:42.131872 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:08:42.131810 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ca86793-5227-42be-84de-b06e8680755a" containerName="manager" Jul 07 19:08:42.131928 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:08:42.131875 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="7ca86793-5227-42be-84de-b06e8680755a" containerName="manager" Jul 07 19:08:42.134798 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:08:42.134778 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-q9d5b" Jul 07 19:08:42.148809 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:08:42.148777 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-q9d5b"] Jul 07 19:08:42.174357 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:08:42.174325 2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-tl2qp\" is forbidden: User \"system:node:ip-10-0-142-134.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-134.ec2.internal' and this object" podUID="7ca86793-5227-42be-84de-b06e8680755a" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-tl2qp" Jul 07 19:08:42.199393 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:08:42.199368 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8r7k\" (UniqueName: \"kubernetes.io/projected/1eb6d982-857c-49eb-a0ca-f489029bcc87-kube-api-access-b8r7k\") pod \"limitador-operator-controller-manager-859d7b78bd-q9d5b\" (UID: \"1eb6d982-857c-49eb-a0ca-f489029bcc87\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-q9d5b" Jul 07 19:08:42.300519 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:08:42.300477 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8r7k\" (UniqueName: \"kubernetes.io/projected/1eb6d982-857c-49eb-a0ca-f489029bcc87-kube-api-access-b8r7k\") pod \"limitador-operator-controller-manager-859d7b78bd-q9d5b\" (UID: \"1eb6d982-857c-49eb-a0ca-f489029bcc87\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-q9d5b" Jul 07 19:08:42.327837 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:08:42.327808 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8r7k\" (UniqueName: \"kubernetes.io/projected/1eb6d982-857c-49eb-a0ca-f489029bcc87-kube-api-access-b8r7k\") pod \"limitador-operator-controller-manager-859d7b78bd-q9d5b\" (UID: \"1eb6d982-857c-49eb-a0ca-f489029bcc87\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-q9d5b" Jul 07 19:08:42.343864 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:08:42.343845 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-tl2qp" Jul 07 19:08:42.345382 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:08:42.345358 2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-tl2qp\" is forbidden: User \"system:node:ip-10-0-142-134.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-134.ec2.internal' and this object" podUID="7ca86793-5227-42be-84de-b06e8680755a" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-tl2qp" Jul 07 19:08:42.477016 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:08:42.476893 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-q9d5b" Jul 07 19:08:42.502366 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:08:42.502336 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xxcg4\" (UniqueName: \"kubernetes.io/projected/7ca86793-5227-42be-84de-b06e8680755a-kube-api-access-xxcg4\") pod \"7ca86793-5227-42be-84de-b06e8680755a\" (UID: \"7ca86793-5227-42be-84de-b06e8680755a\") " Jul 07 19:08:42.504657 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:08:42.504619 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ca86793-5227-42be-84de-b06e8680755a-kube-api-access-xxcg4" (OuterVolumeSpecName: "kube-api-access-xxcg4") pod "7ca86793-5227-42be-84de-b06e8680755a" (UID: "7ca86793-5227-42be-84de-b06e8680755a"). InnerVolumeSpecName "kube-api-access-xxcg4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 19:08:42.603972 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:08:42.603926 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xxcg4\" (UniqueName: \"kubernetes.io/projected/7ca86793-5227-42be-84de-b06e8680755a-kube-api-access-xxcg4\") on node \"ip-10-0-142-134.ec2.internal\" DevicePath \"\"" Jul 07 19:08:42.625298 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:08:42.625201 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-q9d5b"] Jul 07 19:08:42.629009 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:08:42.628983 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1eb6d982_857c_49eb_a0ca_f489029bcc87.slice/crio-58059c5b2fffb5ce139e56b058b5f8cd21f8dcef3a952375fa93a3f4b47b2da3 WatchSource:0}: Error finding container 58059c5b2fffb5ce139e56b058b5f8cd21f8dcef3a952375fa93a3f4b47b2da3: Status 404 returned error can't find the container with id 58059c5b2fffb5ce139e56b058b5f8cd21f8dcef3a952375fa93a3f4b47b2da3 Jul 07 19:08:42.696006 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:08:42.695976 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-q9d5b" event={"ID":"1eb6d982-857c-49eb-a0ca-f489029bcc87","Type":"ContainerStarted","Data":"58059c5b2fffb5ce139e56b058b5f8cd21f8dcef3a952375fa93a3f4b47b2da3"} Jul 07 19:08:42.697457 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:08:42.697432 2563 generic.go:356] "Generic (PLEG): container finished" podID="7ca86793-5227-42be-84de-b06e8680755a" containerID="5a443707c5421a45d008547efaf2e17c791f13921eed0a7a48d2d7ae0598059f" exitCode=0 Jul 07 19:08:42.697565 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:08:42.697477 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-tl2qp" Jul 07 19:08:42.697565 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:08:42.697522 2563 scope.go:117] "RemoveContainer" containerID="5a443707c5421a45d008547efaf2e17c791f13921eed0a7a48d2d7ae0598059f" Jul 07 19:08:42.699444 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:08:42.699412 2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-tl2qp\" is forbidden: User \"system:node:ip-10-0-142-134.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-134.ec2.internal' and this object" podUID="7ca86793-5227-42be-84de-b06e8680755a" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-tl2qp" Jul 07 19:08:42.705872 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:08:42.705858 2563 scope.go:117] "RemoveContainer" containerID="5a443707c5421a45d008547efaf2e17c791f13921eed0a7a48d2d7ae0598059f" Jul 07 19:08:42.706152 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:08:42.706127 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a443707c5421a45d008547efaf2e17c791f13921eed0a7a48d2d7ae0598059f\": container with ID starting with 5a443707c5421a45d008547efaf2e17c791f13921eed0a7a48d2d7ae0598059f not found: ID does not exist" containerID="5a443707c5421a45d008547efaf2e17c791f13921eed0a7a48d2d7ae0598059f" Jul 07 19:08:42.706209 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:08:42.706166 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a443707c5421a45d008547efaf2e17c791f13921eed0a7a48d2d7ae0598059f"} err="failed to get container status \"5a443707c5421a45d008547efaf2e17c791f13921eed0a7a48d2d7ae0598059f\": rpc error: code = NotFound desc = could not find container \"5a443707c5421a45d008547efaf2e17c791f13921eed0a7a48d2d7ae0598059f\": container with ID starting with 5a443707c5421a45d008547efaf2e17c791f13921eed0a7a48d2d7ae0598059f not found: ID does not exist" Jul 07 19:08:42.707471 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:08:42.707447 2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-tl2qp\" is forbidden: User \"system:node:ip-10-0-142-134.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-134.ec2.internal' and this object" podUID="7ca86793-5227-42be-84de-b06e8680755a" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-tl2qp" Jul 07 19:08:43.480300 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:08:43.480216 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7ca86793-5227-42be-84de-b06e8680755a" path="/var/lib/kubelet/pods/7ca86793-5227-42be-84de-b06e8680755a/volumes" Jul 07 19:08:43.703604 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:08:43.703570 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-q9d5b" event={"ID":"1eb6d982-857c-49eb-a0ca-f489029bcc87","Type":"ContainerStarted","Data":"559250b95c53617cadffeac3ec56c7217846153303d1c54450f9b046dd978fdb"} Jul 07 19:08:43.704082 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:08:43.703685 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-q9d5b" Jul 07 19:08:45.414521 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:08:45.414490 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:08:45.519027 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:08:45.518985 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:08:45.662558 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:08:45.662532 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-2tvsm" Jul 07 19:08:45.688632 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:08:45.688507 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-q9d5b" podStartSLOduration=3.688485864 podStartE2EDuration="3.688485864s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:08:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:08:43.742765642 +0000 UTC m=+506.758769826" watchObservedRunningTime="2026-07-07 19:08:45.688485864 +0000 UTC m=+508.704490046" Jul 07 19:08:45.738423 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:08:45.738372 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:08:45.926338 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:08:45.926300 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:08:52.221283 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:08:52.221248 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:08:52.320090 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:08:52.320056 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:08:54.711540 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:08:54.711506 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-q9d5b" Jul 07 19:08:54.846238 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:08:54.846205 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:08:54.946024 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:08:54.945986 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:08:57.020839 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:08:57.020801 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:08:57.130772 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:08:57.130744 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:09:03.525054 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:09:03.525011 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:09:03.616405 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:09:03.616366 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:09:05.080762 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:09:05.080724 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:09:05.163038 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:09:05.163005 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:09:05.989345 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:09:05.989305 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:09:06.092933 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:09:06.092900 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:09:06.191763 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:09:06.191729 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:09:10.035535 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:09:10.035450 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:09:10.135920 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:09:10.135884 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:09:11.876176 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:09:11.876143 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-74949dcd89-rph7z"] Jul 07 19:09:11.879642 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:09:11.879610 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-74949dcd89-rph7z" Jul 07 19:09:11.883421 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:09:11.883394 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-kmr4f\"" Jul 07 19:09:11.899011 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:09:11.898980 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-74949dcd89-rph7z"] Jul 07 19:09:11.939346 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:09:11.939311 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:09:11.968621 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:09:11.968581 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9e6513c-1aa8-40da-9c2c-a0ba5ea11b69-istio-envoy\") pod \"maas-default-gateway-openshift-default-74949dcd89-rph7z\" (UID: \"f9e6513c-1aa8-40da-9c2c-a0ba5ea11b69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74949dcd89-rph7z" Jul 07 19:09:11.968789 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:09:11.968638 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9e6513c-1aa8-40da-9c2c-a0ba5ea11b69-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-74949dcd89-rph7z\" (UID: \"f9e6513c-1aa8-40da-9c2c-a0ba5ea11b69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74949dcd89-rph7z" Jul 07 19:09:11.968789 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:09:11.968701 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9e6513c-1aa8-40da-9c2c-a0ba5ea11b69-istio-data\") pod \"maas-default-gateway-openshift-default-74949dcd89-rph7z\" (UID: \"f9e6513c-1aa8-40da-9c2c-a0ba5ea11b69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74949dcd89-rph7z" Jul 07 19:09:11.968789 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:09:11.968744 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9e6513c-1aa8-40da-9c2c-a0ba5ea11b69-credential-socket\") pod \"maas-default-gateway-openshift-default-74949dcd89-rph7z\" (UID: \"f9e6513c-1aa8-40da-9c2c-a0ba5ea11b69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74949dcd89-rph7z" Jul 07 19:09:11.968789 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:09:11.968776 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9e6513c-1aa8-40da-9c2c-a0ba5ea11b69-workload-socket\") pod \"maas-default-gateway-openshift-default-74949dcd89-rph7z\" (UID: \"f9e6513c-1aa8-40da-9c2c-a0ba5ea11b69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74949dcd89-rph7z" Jul 07 19:09:11.968928 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:09:11.968791 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9e6513c-1aa8-40da-9c2c-a0ba5ea11b69-istio-podinfo\") pod \"maas-default-gateway-openshift-default-74949dcd89-rph7z\" (UID: \"f9e6513c-1aa8-40da-9c2c-a0ba5ea11b69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74949dcd89-rph7z" Jul 07 19:09:11.968928 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:09:11.968824 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9e6513c-1aa8-40da-9c2c-a0ba5ea11b69-workload-certs\") pod \"maas-default-gateway-openshift-default-74949dcd89-rph7z\" (UID: \"f9e6513c-1aa8-40da-9c2c-a0ba5ea11b69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74949dcd89-rph7z" Jul 07 19:09:11.968928 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:09:11.968838 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9e6513c-1aa8-40da-9c2c-a0ba5ea11b69-istio-token\") pod \"maas-default-gateway-openshift-default-74949dcd89-rph7z\" (UID: \"f9e6513c-1aa8-40da-9c2c-a0ba5ea11b69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74949dcd89-rph7z" Jul 07 19:09:11.968928 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:09:11.968858 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dzm6t\" (UniqueName: \"kubernetes.io/projected/f9e6513c-1aa8-40da-9c2c-a0ba5ea11b69-kube-api-access-dzm6t\") pod \"maas-default-gateway-openshift-default-74949dcd89-rph7z\" (UID: \"f9e6513c-1aa8-40da-9c2c-a0ba5ea11b69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74949dcd89-rph7z" Jul 07 19:09:12.069908 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:09:12.069875 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9e6513c-1aa8-40da-9c2c-a0ba5ea11b69-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-74949dcd89-rph7z\" (UID: \"f9e6513c-1aa8-40da-9c2c-a0ba5ea11b69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74949dcd89-rph7z" Jul 07 19:09:12.069908 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:09:12.069909 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9e6513c-1aa8-40da-9c2c-a0ba5ea11b69-istio-data\") pod \"maas-default-gateway-openshift-default-74949dcd89-rph7z\" (UID: \"f9e6513c-1aa8-40da-9c2c-a0ba5ea11b69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74949dcd89-rph7z" Jul 07 19:09:12.070182 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:09:12.070023 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9e6513c-1aa8-40da-9c2c-a0ba5ea11b69-credential-socket\") pod \"maas-default-gateway-openshift-default-74949dcd89-rph7z\" (UID: \"f9e6513c-1aa8-40da-9c2c-a0ba5ea11b69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74949dcd89-rph7z" Jul 07 19:09:12.070182 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:09:12.070095 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9e6513c-1aa8-40da-9c2c-a0ba5ea11b69-workload-socket\") pod \"maas-default-gateway-openshift-default-74949dcd89-rph7z\" (UID: \"f9e6513c-1aa8-40da-9c2c-a0ba5ea11b69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74949dcd89-rph7z" Jul 07 19:09:12.070182 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:09:12.070123 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9e6513c-1aa8-40da-9c2c-a0ba5ea11b69-istio-podinfo\") pod \"maas-default-gateway-openshift-default-74949dcd89-rph7z\" (UID: \"f9e6513c-1aa8-40da-9c2c-a0ba5ea11b69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74949dcd89-rph7z" Jul 07 19:09:12.070182 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:09:12.070151 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9e6513c-1aa8-40da-9c2c-a0ba5ea11b69-workload-certs\") pod \"maas-default-gateway-openshift-default-74949dcd89-rph7z\" (UID: \"f9e6513c-1aa8-40da-9c2c-a0ba5ea11b69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74949dcd89-rph7z" Jul 07 19:09:12.070182 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:09:12.070178 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9e6513c-1aa8-40da-9c2c-a0ba5ea11b69-istio-token\") pod \"maas-default-gateway-openshift-default-74949dcd89-rph7z\" (UID: \"f9e6513c-1aa8-40da-9c2c-a0ba5ea11b69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74949dcd89-rph7z" Jul 07 19:09:12.070455 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:09:12.070217 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzm6t\" (UniqueName: \"kubernetes.io/projected/f9e6513c-1aa8-40da-9c2c-a0ba5ea11b69-kube-api-access-dzm6t\") pod \"maas-default-gateway-openshift-default-74949dcd89-rph7z\" (UID: \"f9e6513c-1aa8-40da-9c2c-a0ba5ea11b69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74949dcd89-rph7z" Jul 07 19:09:12.070455 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:09:12.070304 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9e6513c-1aa8-40da-9c2c-a0ba5ea11b69-istio-envoy\") pod \"maas-default-gateway-openshift-default-74949dcd89-rph7z\" (UID: \"f9e6513c-1aa8-40da-9c2c-a0ba5ea11b69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74949dcd89-rph7z" Jul 07 19:09:12.070455 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:09:12.070319 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9e6513c-1aa8-40da-9c2c-a0ba5ea11b69-istio-data\") pod \"maas-default-gateway-openshift-default-74949dcd89-rph7z\" (UID: \"f9e6513c-1aa8-40da-9c2c-a0ba5ea11b69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74949dcd89-rph7z" Jul 07 19:09:12.070455 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:09:12.070390 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9e6513c-1aa8-40da-9c2c-a0ba5ea11b69-credential-socket\") pod \"maas-default-gateway-openshift-default-74949dcd89-rph7z\" (UID: \"f9e6513c-1aa8-40da-9c2c-a0ba5ea11b69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74949dcd89-rph7z" Jul 07 19:09:12.070659 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:09:12.070477 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9e6513c-1aa8-40da-9c2c-a0ba5ea11b69-workload-socket\") pod \"maas-default-gateway-openshift-default-74949dcd89-rph7z\" (UID: \"f9e6513c-1aa8-40da-9c2c-a0ba5ea11b69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74949dcd89-rph7z" Jul 07 19:09:12.070659 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:09:12.070524 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9e6513c-1aa8-40da-9c2c-a0ba5ea11b69-workload-certs\") pod \"maas-default-gateway-openshift-default-74949dcd89-rph7z\" (UID: \"f9e6513c-1aa8-40da-9c2c-a0ba5ea11b69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74949dcd89-rph7z" Jul 07 19:09:12.070774 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:09:12.070748 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9e6513c-1aa8-40da-9c2c-a0ba5ea11b69-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-74949dcd89-rph7z\" (UID: \"f9e6513c-1aa8-40da-9c2c-a0ba5ea11b69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74949dcd89-rph7z" Jul 07 19:09:12.072583 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:09:12.072558 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9e6513c-1aa8-40da-9c2c-a0ba5ea11b69-istio-envoy\") pod \"maas-default-gateway-openshift-default-74949dcd89-rph7z\" (UID: \"f9e6513c-1aa8-40da-9c2c-a0ba5ea11b69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74949dcd89-rph7z" Jul 07 19:09:12.072803 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:09:12.072786 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9e6513c-1aa8-40da-9c2c-a0ba5ea11b69-istio-podinfo\") pod \"maas-default-gateway-openshift-default-74949dcd89-rph7z\" (UID: \"f9e6513c-1aa8-40da-9c2c-a0ba5ea11b69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74949dcd89-rph7z" Jul 07 19:09:12.084074 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:09:12.084047 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9e6513c-1aa8-40da-9c2c-a0ba5ea11b69-istio-token\") pod \"maas-default-gateway-openshift-default-74949dcd89-rph7z\" (UID: \"f9e6513c-1aa8-40da-9c2c-a0ba5ea11b69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74949dcd89-rph7z" Jul 07 19:09:12.084208 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:09:12.084188 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzm6t\" (UniqueName: \"kubernetes.io/projected/f9e6513c-1aa8-40da-9c2c-a0ba5ea11b69-kube-api-access-dzm6t\") pod \"maas-default-gateway-openshift-default-74949dcd89-rph7z\" (UID: \"f9e6513c-1aa8-40da-9c2c-a0ba5ea11b69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74949dcd89-rph7z" Jul 07 19:09:12.166288 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:09:12.166201 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:09:12.189779 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:09:12.189755 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-74949dcd89-rph7z" Jul 07 19:09:12.326190 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:09:12.326134 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-74949dcd89-rph7z"] Jul 07 19:09:12.328548 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:09:12.328521 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9e6513c_1aa8_40da_9c2c_a0ba5ea11b69.slice/crio-aa2fd9162a054a253b0cc14ff6e8e34c4fabccce7e9ba473db47ea42629fbc33 WatchSource:0}: Error finding container aa2fd9162a054a253b0cc14ff6e8e34c4fabccce7e9ba473db47ea42629fbc33: Status 404 returned error can't find the container with id aa2fd9162a054a253b0cc14ff6e8e34c4fabccce7e9ba473db47ea42629fbc33 Jul 07 19:09:12.330553 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:09:12.330522 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 07 19:09:12.330626 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:09:12.330585 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 07 19:09:12.330626 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:09:12.330614 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 07 19:09:12.677653 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:09:12.677610 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:09:12.784075 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:09:12.784035 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:09:12.843798 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:09:12.843762 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-74949dcd89-rph7z" event={"ID":"f9e6513c-1aa8-40da-9c2c-a0ba5ea11b69","Type":"ContainerStarted","Data":"aa38968de72dd26d2a7e4c3aa5277c8932d706a2791b8e5c8017e9481335f702"} Jul 07 19:09:12.843945 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:09:12.843823 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-74949dcd89-rph7z" event={"ID":"f9e6513c-1aa8-40da-9c2c-a0ba5ea11b69","Type":"ContainerStarted","Data":"aa2fd9162a054a253b0cc14ff6e8e34c4fabccce7e9ba473db47ea42629fbc33"} Jul 07 19:09:12.867412 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:09:12.867362 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-74949dcd89-rph7z" podStartSLOduration=1.867344935 podStartE2EDuration="1.867344935s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:09:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:09:12.86564136 +0000 UTC m=+535.881645541" watchObservedRunningTime="2026-07-07 19:09:12.867344935 +0000 UTC m=+535.883349116" Jul 07 19:09:13.190652 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:09:13.190611 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-74949dcd89-rph7z" Jul 07 19:09:13.195806 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:09:13.195779 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-74949dcd89-rph7z" Jul 07 19:09:13.848943 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:09:13.848905 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-74949dcd89-rph7z" Jul 07 19:09:13.849845 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:09:13.849827 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-74949dcd89-rph7z" Jul 07 19:09:14.035173 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:09:14.035140 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:09:14.133975 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:09:14.133874 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:09:14.240101 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:09:14.240072 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:09:15.026909 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:09:15.026874 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:09:15.139621 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:09:15.139587 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:09:15.657879 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:09:15.657831 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:09:15.764740 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:09:15.764711 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:09:22.515086 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:09:22.515043 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:09:22.617476 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:09:22.617445 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:09:29.095908 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:09:29.095873 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:09:29.199500 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:09:29.199463 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:09:35.535839 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:09:35.535783 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:09:46.933105 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:09:46.933057 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:09:47.152363 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:09:47.152323 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:09:53.700175 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:09:53.700124 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:09:53.864376 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:09:53.864337 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:09:54.031428 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:09:54.031396 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:09:54.541510 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:09:54.541475 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:09:56.544203 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:09:56.544171 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:09:56.744268 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:09:56.744216 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:09:58.616845 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:09:58.616808 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:09:58.743212 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:09:58.743176 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:10:17.449473 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:10:17.449442 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-frt7c_e25ab82f-72fb-4cde-89c4-0a0dd80f5feb/console-operator/2.log" Jul 07 19:10:17.449941 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:10:17.449449 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-frt7c_e25ab82f-72fb-4cde-89c4-0a0dd80f5feb/console-operator/2.log" Jul 07 19:10:17.455567 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:10:17.455547 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zzwbq_4f0c5d5c-147f-426a-965d-8886e5e79984/ovn-acl-logging/0.log" Jul 07 19:10:17.455567 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:10:17.455559 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zzwbq_4f0c5d5c-147f-426a-965d-8886e5e79984/ovn-acl-logging/0.log" Jul 07 19:10:25.963445 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:10:25.963400 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:10:26.037344 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:10:26.037308 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:10:31.731241 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:10:31.731183 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:10:32.113841 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:10:32.113798 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:10:32.235646 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:10:32.235603 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:10:32.636502 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:10:32.636456 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:10:32.741462 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:10:32.741409 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:10:40.284141 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:10:40.284108 2563 generic.go:356] "Generic (PLEG): container finished" podID="f9e6513c-1aa8-40da-9c2c-a0ba5ea11b69" containerID="aa38968de72dd26d2a7e4c3aa5277c8932d706a2791b8e5c8017e9481335f702" exitCode=0 Jul 07 19:10:40.284576 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:10:40.284193 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-74949dcd89-rph7z" event={"ID":"f9e6513c-1aa8-40da-9c2c-a0ba5ea11b69","Type":"ContainerDied","Data":"aa38968de72dd26d2a7e4c3aa5277c8932d706a2791b8e5c8017e9481335f702"} Jul 07 19:10:40.284621 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:10:40.284579 2563 scope.go:117] "RemoveContainer" containerID="aa38968de72dd26d2a7e4c3aa5277c8932d706a2791b8e5c8017e9481335f702" Jul 07 19:10:40.285536 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:10:40.285511 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 07 19:10:40.285628 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:10:40.285572 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 07 19:10:40.285628 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:10:40.285604 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 07 19:10:41.290769 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:10:41.290731 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-74949dcd89-rph7z" event={"ID":"f9e6513c-1aa8-40da-9c2c-a0ba5ea11b69","Type":"ContainerStarted","Data":"ee3c6669f857154e7e0d953b21632f905e591a3a4947f3596d62da3e37666ee9"} Jul 07 19:10:41.785993 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:10:41.785945 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:10:41.886492 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:10:41.886454 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:10:42.190069 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:10:42.189972 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-74949dcd89-rph7z" Jul 07 19:10:42.191164 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:10:42.191136 2563 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-74949dcd89-rph7z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" start-of-body= Jul 07 19:10:42.191306 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:10:42.191203 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-74949dcd89-rph7z" podUID="f9e6513c-1aa8-40da-9c2c-a0ba5ea11b69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" Jul 07 19:10:43.191058 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:10:43.191024 2563 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-74949dcd89-rph7z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" start-of-body= Jul 07 19:10:43.191488 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:10:43.191079 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-74949dcd89-rph7z" podUID="f9e6513c-1aa8-40da-9c2c-a0ba5ea11b69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" Jul 07 19:10:43.849603 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:10:43.849566 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-74949dcd89-rph7z" Jul 07 19:10:44.191211 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:10:44.191118 2563 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-74949dcd89-rph7z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" start-of-body= Jul 07 19:10:44.191211 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:10:44.191192 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-74949dcd89-rph7z" podUID="f9e6513c-1aa8-40da-9c2c-a0ba5ea11b69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" Jul 07 19:10:45.194046 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:10:45.194013 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-74949dcd89-rph7z" Jul 07 19:10:45.194927 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:10:45.194911 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-74949dcd89-rph7z" Jul 07 19:10:59.501234 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:10:59.501195 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:11:00.758495 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:11:00.758456 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:11:02.334830 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:11:02.334797 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-c622d"] Jul 07 19:11:02.338370 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:11:02.338348 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-c622d" Jul 07 19:11:02.341550 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:11:02.341491 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:11:02.341829 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:11:02.341813 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 07 19:11:02.342078 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:11:02.342064 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-n4slz\"" Jul 07 19:11:02.343811 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:11:02.343797 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Jul 07 19:11:02.344019 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:11:02.344006 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 07 19:11:02.353540 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:11:02.353511 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-c622d"] Jul 07 19:11:02.471390 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:11:02.471354 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d5d65f82-5aca-41e4-aa46-0457819c4664-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-c622d\" (UID: \"d5d65f82-5aca-41e4-aa46-0457819c4664\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-c622d" Jul 07 19:11:02.471564 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:11:02.471402 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d5d65f82-5aca-41e4-aa46-0457819c4664-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-c622d\" (UID: \"d5d65f82-5aca-41e4-aa46-0457819c4664\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-c622d" Jul 07 19:11:02.471564 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:11:02.471452 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vt7fs\" (UniqueName: \"kubernetes.io/projected/d5d65f82-5aca-41e4-aa46-0457819c4664-kube-api-access-vt7fs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-c622d\" (UID: \"d5d65f82-5aca-41e4-aa46-0457819c4664\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-c622d" Jul 07 19:11:02.471564 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:11:02.471539 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d5d65f82-5aca-41e4-aa46-0457819c4664-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-c622d\" (UID: \"d5d65f82-5aca-41e4-aa46-0457819c4664\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-c622d" Jul 07 19:11:02.471673 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:11:02.471577 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d5d65f82-5aca-41e4-aa46-0457819c4664-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-c622d\" (UID: \"d5d65f82-5aca-41e4-aa46-0457819c4664\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-c622d" Jul 07 19:11:02.471673 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:11:02.471615 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d5d65f82-5aca-41e4-aa46-0457819c4664-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-c622d\" (UID: \"d5d65f82-5aca-41e4-aa46-0457819c4664\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-c622d" Jul 07 19:11:02.572457 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:11:02.572411 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d5d65f82-5aca-41e4-aa46-0457819c4664-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-c622d\" (UID: \"d5d65f82-5aca-41e4-aa46-0457819c4664\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-c622d" Jul 07 19:11:02.572651 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:11:02.572471 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d5d65f82-5aca-41e4-aa46-0457819c4664-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-c622d\" (UID: \"d5d65f82-5aca-41e4-aa46-0457819c4664\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-c622d" Jul 07 19:11:02.572651 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:11:02.572505 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d5d65f82-5aca-41e4-aa46-0457819c4664-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-c622d\" (UID: \"d5d65f82-5aca-41e4-aa46-0457819c4664\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-c622d" Jul 07 19:11:02.572651 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:11:02.572545 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d5d65f82-5aca-41e4-aa46-0457819c4664-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-c622d\" (UID: \"d5d65f82-5aca-41e4-aa46-0457819c4664\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-c622d" Jul 07 19:11:02.572651 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:11:02.572571 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d5d65f82-5aca-41e4-aa46-0457819c4664-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-c622d\" (UID: \"d5d65f82-5aca-41e4-aa46-0457819c4664\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-c622d" Jul 07 19:11:02.572651 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:11:02.572598 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vt7fs\" (UniqueName: \"kubernetes.io/projected/d5d65f82-5aca-41e4-aa46-0457819c4664-kube-api-access-vt7fs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-c622d\" (UID: \"d5d65f82-5aca-41e4-aa46-0457819c4664\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-c622d" Jul 07 19:11:02.572925 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:11:02.572899 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d5d65f82-5aca-41e4-aa46-0457819c4664-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-c622d\" (UID: \"d5d65f82-5aca-41e4-aa46-0457819c4664\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-c622d" Jul 07 19:11:02.573027 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:11:02.572936 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d5d65f82-5aca-41e4-aa46-0457819c4664-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-c622d\" (UID: \"d5d65f82-5aca-41e4-aa46-0457819c4664\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-c622d" Jul 07 19:11:02.573027 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:11:02.572988 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d5d65f82-5aca-41e4-aa46-0457819c4664-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-c622d\" (UID: \"d5d65f82-5aca-41e4-aa46-0457819c4664\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-c622d" Jul 07 19:11:02.574965 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:11:02.574932 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d5d65f82-5aca-41e4-aa46-0457819c4664-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-c622d\" (UID: \"d5d65f82-5aca-41e4-aa46-0457819c4664\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-c622d" Jul 07 19:11:02.575244 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:11:02.575224 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d5d65f82-5aca-41e4-aa46-0457819c4664-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-c622d\" (UID: \"d5d65f82-5aca-41e4-aa46-0457819c4664\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-c622d" Jul 07 19:11:02.583112 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:11:02.583086 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vt7fs\" (UniqueName: \"kubernetes.io/projected/d5d65f82-5aca-41e4-aa46-0457819c4664-kube-api-access-vt7fs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-c622d\" (UID: \"d5d65f82-5aca-41e4-aa46-0457819c4664\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-c622d" Jul 07 19:11:02.650652 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:11:02.650558 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-c622d" Jul 07 19:11:02.796463 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:11:02.796426 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-c622d"] Jul 07 19:11:02.797910 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:11:02.797882 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5d65f82_5aca_41e4_aa46_0457819c4664.slice/crio-b75c1d6b28e9eebedd9ed85835c5f4d6e0df739ae36f8b566712ac781d5e9a54 WatchSource:0}: Error finding container b75c1d6b28e9eebedd9ed85835c5f4d6e0df739ae36f8b566712ac781d5e9a54: Status 404 returned error can't find the container with id b75c1d6b28e9eebedd9ed85835c5f4d6e0df739ae36f8b566712ac781d5e9a54 Jul 07 19:11:03.400922 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:11:03.400887 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-c622d" event={"ID":"d5d65f82-5aca-41e4-aa46-0457819c4664","Type":"ContainerStarted","Data":"b75c1d6b28e9eebedd9ed85835c5f4d6e0df739ae36f8b566712ac781d5e9a54"} Jul 07 19:11:04.410395 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:11:04.410357 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-c622d" event={"ID":"d5d65f82-5aca-41e4-aa46-0457819c4664","Type":"ContainerStarted","Data":"a54534678341604e6dd24398037fd1b0f889b8d7d80c8f9ab13530950973f16d"} Jul 07 19:11:04.430256 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:11:04.430184 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-c622d" podStartSLOduration=1.389981766 podStartE2EDuration="2.430167066s" totalImagesPullingTime="1.0401853s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:11:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:11:04.430046805 +0000 UTC m=+647.446050987" watchObservedRunningTime="2026-07-07 19:11:04.430167066 +0000 UTC m=+647.446171246" Jul 07 19:11:12.651263 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:11:12.651224 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-c622d" Jul 07 19:11:12.651750 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:11:12.651302 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-c622d" Jul 07 19:11:12.663850 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:11:12.663813 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-c622d" Jul 07 19:11:13.469822 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:11:13.469791 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-c622d" Jul 07 19:11:14.509486 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:11:14.509449 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:11:26.703844 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:11:26.703813 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:11:30.748725 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:11:30.748690 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db8mhk2"] Jul 07 19:11:30.752329 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:11:30.752307 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db8mhk2" Jul 07 19:11:30.754332 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:11:30.754308 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Jul 07 19:11:30.763064 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:11:30.763034 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db8mhk2"] Jul 07 19:11:30.786264 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:11:30.786230 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:11:30.830334 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:11:30.830292 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/354ac1be-b1da-46ba-9eac-57e9cdcd7469-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db8mhk2\" (UID: \"354ac1be-b1da-46ba-9eac-57e9cdcd7469\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db8mhk2" Jul 07 19:11:30.830533 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:11:30.830352 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/354ac1be-b1da-46ba-9eac-57e9cdcd7469-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db8mhk2\" (UID: \"354ac1be-b1da-46ba-9eac-57e9cdcd7469\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db8mhk2" Jul 07 19:11:30.830533 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:11:30.830449 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/354ac1be-b1da-46ba-9eac-57e9cdcd7469-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db8mhk2\" (UID: \"354ac1be-b1da-46ba-9eac-57e9cdcd7469\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db8mhk2" Jul 07 19:11:30.830533 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:11:30.830481 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/354ac1be-b1da-46ba-9eac-57e9cdcd7469-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db8mhk2\" (UID: \"354ac1be-b1da-46ba-9eac-57e9cdcd7469\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db8mhk2" Jul 07 19:11:30.830533 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:11:30.830525 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f56xc\" (UniqueName: \"kubernetes.io/projected/354ac1be-b1da-46ba-9eac-57e9cdcd7469-kube-api-access-f56xc\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db8mhk2\" (UID: \"354ac1be-b1da-46ba-9eac-57e9cdcd7469\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db8mhk2" Jul 07 19:11:30.830695 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:11:30.830590 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/354ac1be-b1da-46ba-9eac-57e9cdcd7469-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db8mhk2\" (UID: \"354ac1be-b1da-46ba-9eac-57e9cdcd7469\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db8mhk2" Jul 07 19:11:30.931168 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:11:30.931124 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/354ac1be-b1da-46ba-9eac-57e9cdcd7469-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db8mhk2\" (UID: \"354ac1be-b1da-46ba-9eac-57e9cdcd7469\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db8mhk2" Jul 07 19:11:30.931367 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:11:30.931183 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f56xc\" (UniqueName: \"kubernetes.io/projected/354ac1be-b1da-46ba-9eac-57e9cdcd7469-kube-api-access-f56xc\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db8mhk2\" (UID: \"354ac1be-b1da-46ba-9eac-57e9cdcd7469\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db8mhk2" Jul 07 19:11:30.931367 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:11:30.931232 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/354ac1be-b1da-46ba-9eac-57e9cdcd7469-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db8mhk2\" (UID: \"354ac1be-b1da-46ba-9eac-57e9cdcd7469\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db8mhk2" Jul 07 19:11:30.931367 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:11:30.931276 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/354ac1be-b1da-46ba-9eac-57e9cdcd7469-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db8mhk2\" (UID: \"354ac1be-b1da-46ba-9eac-57e9cdcd7469\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db8mhk2" Jul 07 19:11:30.931367 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:11:30.931323 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/354ac1be-b1da-46ba-9eac-57e9cdcd7469-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db8mhk2\" (UID: \"354ac1be-b1da-46ba-9eac-57e9cdcd7469\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db8mhk2" Jul 07 19:11:30.931601 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:11:30.931369 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/354ac1be-b1da-46ba-9eac-57e9cdcd7469-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db8mhk2\" (UID: \"354ac1be-b1da-46ba-9eac-57e9cdcd7469\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db8mhk2" Jul 07 19:11:30.931882 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:11:30.931860 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/354ac1be-b1da-46ba-9eac-57e9cdcd7469-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db8mhk2\" (UID: \"354ac1be-b1da-46ba-9eac-57e9cdcd7469\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db8mhk2" Jul 07 19:11:30.932016 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:11:30.931933 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/354ac1be-b1da-46ba-9eac-57e9cdcd7469-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db8mhk2\" (UID: \"354ac1be-b1da-46ba-9eac-57e9cdcd7469\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db8mhk2" Jul 07 19:11:30.932217 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:11:30.932195 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/354ac1be-b1da-46ba-9eac-57e9cdcd7469-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db8mhk2\" (UID: \"354ac1be-b1da-46ba-9eac-57e9cdcd7469\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db8mhk2" Jul 07 19:11:30.935026 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:11:30.935000 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/354ac1be-b1da-46ba-9eac-57e9cdcd7469-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db8mhk2\" (UID: \"354ac1be-b1da-46ba-9eac-57e9cdcd7469\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db8mhk2" Jul 07 19:11:30.935381 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:11:30.935353 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/354ac1be-b1da-46ba-9eac-57e9cdcd7469-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db8mhk2\" (UID: \"354ac1be-b1da-46ba-9eac-57e9cdcd7469\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db8mhk2" Jul 07 19:11:30.940571 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:11:30.940545 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f56xc\" (UniqueName: \"kubernetes.io/projected/354ac1be-b1da-46ba-9eac-57e9cdcd7469-kube-api-access-f56xc\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db8mhk2\" (UID: \"354ac1be-b1da-46ba-9eac-57e9cdcd7469\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db8mhk2" Jul 07 19:11:31.062544 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:11:31.062506 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db8mhk2" Jul 07 19:11:31.197994 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:11:31.197964 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db8mhk2"] Jul 07 19:11:31.200088 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:11:31.200057 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod354ac1be_b1da_46ba_9eac_57e9cdcd7469.slice/crio-758b15866a648f5945ac005f179f19c9f762ea3300b64e89e7bf53dce4ac9fde WatchSource:0}: Error finding container 758b15866a648f5945ac005f179f19c9f762ea3300b64e89e7bf53dce4ac9fde: Status 404 returned error can't find the container with id 758b15866a648f5945ac005f179f19c9f762ea3300b64e89e7bf53dce4ac9fde Jul 07 19:11:31.201482 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:11:31.201466 2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 07 19:11:31.539547 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:11:31.539517 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db8mhk2" event={"ID":"354ac1be-b1da-46ba-9eac-57e9cdcd7469","Type":"ContainerStarted","Data":"2a56e417f350016976b9db1a5058d32844a4f4771dd9f4f35d1536e93f92c545"} Jul 07 19:11:31.539734 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:11:31.539575 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db8mhk2" event={"ID":"354ac1be-b1da-46ba-9eac-57e9cdcd7469","Type":"ContainerStarted","Data":"758b15866a648f5945ac005f179f19c9f762ea3300b64e89e7bf53dce4ac9fde"} Jul 07 19:11:31.559967 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:11:31.559904 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db8mhk2" podStartSLOduration=1.324258097 podStartE2EDuration="1.559886637s" totalImagesPullingTime="235.62854ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:11:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:11:31.55785258 +0000 UTC m=+674.573856758" watchObservedRunningTime="2026-07-07 19:11:31.559886637 +0000 UTC m=+674.575890818" Jul 07 19:11:41.063021 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:11:41.062976 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db8mhk2" Jul 07 19:11:41.063565 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:11:41.063048 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db8mhk2" Jul 07 19:11:41.075978 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:11:41.075912 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db8mhk2" Jul 07 19:11:41.601190 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:11:41.601161 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db8mhk2" Jul 07 19:11:43.187570 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:11:43.187524 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:11:48.454700 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:11:48.454666 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:11:52.355364 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:11:52.355324 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-cg9mj"] Jul 07 19:11:52.359158 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:11:52.359129 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-cg9mj" Jul 07 19:11:52.361387 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:11:52.361361 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Jul 07 19:11:52.370284 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:11:52.370256 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-cg9mj"] Jul 07 19:11:52.401272 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:11:52.401237 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:11:52.431490 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:11:52.431459 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aee2eeec-b78e-481e-ad56-7577bda25769-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-cg9mj\" (UID: \"aee2eeec-b78e-481e-ad56-7577bda25769\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-cg9mj" Jul 07 19:11:52.431631 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:11:52.431518 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aee2eeec-b78e-481e-ad56-7577bda25769-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-cg9mj\" (UID: \"aee2eeec-b78e-481e-ad56-7577bda25769\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-cg9mj" Jul 07 19:11:52.431631 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:11:52.431583 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aee2eeec-b78e-481e-ad56-7577bda25769-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-cg9mj\" (UID: \"aee2eeec-b78e-481e-ad56-7577bda25769\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-cg9mj" Jul 07 19:11:52.431758 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:11:52.431693 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aee2eeec-b78e-481e-ad56-7577bda25769-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-cg9mj\" (UID: \"aee2eeec-b78e-481e-ad56-7577bda25769\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-cg9mj" Jul 07 19:11:52.431758 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:11:52.431725 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aee2eeec-b78e-481e-ad56-7577bda25769-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-cg9mj\" (UID: \"aee2eeec-b78e-481e-ad56-7577bda25769\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-cg9mj" Jul 07 19:11:52.431867 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:11:52.431756 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nq6vl\" (UniqueName: \"kubernetes.io/projected/aee2eeec-b78e-481e-ad56-7577bda25769-kube-api-access-nq6vl\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-cg9mj\" (UID: \"aee2eeec-b78e-481e-ad56-7577bda25769\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-cg9mj" Jul 07 19:11:52.532278 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:11:52.532245 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aee2eeec-b78e-481e-ad56-7577bda25769-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-cg9mj\" (UID: \"aee2eeec-b78e-481e-ad56-7577bda25769\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-cg9mj" Jul 07 19:11:52.532278 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:11:52.532285 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aee2eeec-b78e-481e-ad56-7577bda25769-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-cg9mj\" (UID: \"aee2eeec-b78e-481e-ad56-7577bda25769\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-cg9mj" Jul 07 19:11:52.532528 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:11:52.532346 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aee2eeec-b78e-481e-ad56-7577bda25769-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-cg9mj\" (UID: \"aee2eeec-b78e-481e-ad56-7577bda25769\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-cg9mj" Jul 07 19:11:52.532528 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:11:52.532365 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aee2eeec-b78e-481e-ad56-7577bda25769-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-cg9mj\" (UID: \"aee2eeec-b78e-481e-ad56-7577bda25769\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-cg9mj" Jul 07 19:11:52.532528 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:11:52.532390 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nq6vl\" (UniqueName: \"kubernetes.io/projected/aee2eeec-b78e-481e-ad56-7577bda25769-kube-api-access-nq6vl\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-cg9mj\" (UID: \"aee2eeec-b78e-481e-ad56-7577bda25769\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-cg9mj" Jul 07 19:11:52.532528 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:11:52.532469 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aee2eeec-b78e-481e-ad56-7577bda25769-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-cg9mj\" (UID: \"aee2eeec-b78e-481e-ad56-7577bda25769\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-cg9mj" Jul 07 19:11:52.532741 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:11:52.532659 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aee2eeec-b78e-481e-ad56-7577bda25769-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-cg9mj\" (UID: \"aee2eeec-b78e-481e-ad56-7577bda25769\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-cg9mj" Jul 07 19:11:52.532741 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:11:52.532724 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aee2eeec-b78e-481e-ad56-7577bda25769-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-cg9mj\" (UID: \"aee2eeec-b78e-481e-ad56-7577bda25769\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-cg9mj" Jul 07 19:11:52.532846 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:11:52.532832 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aee2eeec-b78e-481e-ad56-7577bda25769-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-cg9mj\" (UID: \"aee2eeec-b78e-481e-ad56-7577bda25769\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-cg9mj" Jul 07 19:11:52.534695 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:11:52.534667 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aee2eeec-b78e-481e-ad56-7577bda25769-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-cg9mj\" (UID: \"aee2eeec-b78e-481e-ad56-7577bda25769\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-cg9mj" Jul 07 19:11:52.534863 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:11:52.534846 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aee2eeec-b78e-481e-ad56-7577bda25769-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-cg9mj\" (UID: \"aee2eeec-b78e-481e-ad56-7577bda25769\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-cg9mj" Jul 07 19:11:52.541026 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:11:52.541002 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nq6vl\" (UniqueName: \"kubernetes.io/projected/aee2eeec-b78e-481e-ad56-7577bda25769-kube-api-access-nq6vl\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-cg9mj\" (UID: \"aee2eeec-b78e-481e-ad56-7577bda25769\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-cg9mj" Jul 07 19:11:52.669836 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:11:52.669752 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-cg9mj" Jul 07 19:11:52.808644 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:11:52.808615 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-cg9mj"] Jul 07 19:11:52.811136 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:11:52.811108 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podaee2eeec_b78e_481e_ad56_7577bda25769.slice/crio-8193a80f8bdd66be72a5443ecde41a592798dbcf0e5dd9c40f377258c05c4755 WatchSource:0}: Error finding container 8193a80f8bdd66be72a5443ecde41a592798dbcf0e5dd9c40f377258c05c4755: Status 404 returned error can't find the container with id 8193a80f8bdd66be72a5443ecde41a592798dbcf0e5dd9c40f377258c05c4755 Jul 07 19:11:53.650745 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:11:53.650709 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-cg9mj" event={"ID":"aee2eeec-b78e-481e-ad56-7577bda25769","Type":"ContainerStarted","Data":"338bab5c492a50e61b04cf0bd1c6d07dd63a851931e7cc80ea4c18d4d6bbc7c6"} Jul 07 19:11:53.651170 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:11:53.650756 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-cg9mj" event={"ID":"aee2eeec-b78e-481e-ad56-7577bda25769","Type":"ContainerStarted","Data":"8193a80f8bdd66be72a5443ecde41a592798dbcf0e5dd9c40f377258c05c4755"} Jul 07 19:11:53.673154 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:11:53.673101 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-cg9mj" podStartSLOduration=1.451139069 podStartE2EDuration="1.6730855s" totalImagesPullingTime="221.946431ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:11:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:11:53.670130495 +0000 UTC m=+696.686134677" watchObservedRunningTime="2026-07-07 19:11:53.6730855 +0000 UTC m=+696.689089681" Jul 07 19:12:02.670099 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:12:02.670062 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-cg9mj" Jul 07 19:12:02.670563 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:12:02.670122 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-cg9mj" Jul 07 19:12:02.682775 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:12:02.682740 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-cg9mj" Jul 07 19:12:02.706200 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:12:02.706168 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-cg9mj" Jul 07 19:12:23.915981 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:12:23.915869 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:12:23.992361 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:12:23.992324 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:12:31.470584 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:12:31.470535 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:12:50.614408 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:12:50.614372 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:12:50.714509 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:12:50.714447 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:12:51.752168 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:12:51.752105 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:12:51.865462 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:12:51.865410 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:14:02.433931 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:14:02.433866 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:14:04.407340 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:14:04.407300 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:14:26.019231 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:14:26.019198 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:14:26.092155 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:14:26.092120 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:14:43.037310 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:14:43.037269 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:14:43.136333 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:14:43.136304 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:14:43.255042 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:14:43.255008 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:14:49.762413 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:14:49.762368 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:14:49.868760 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:14:49.868710 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:14:49.988890 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:14:49.988859 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:14:50.131995 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:14:50.131967 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:14:56.820172 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:14:56.820127 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:14:56.892468 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:14:56.892439 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:15:17.482724 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:15:17.482641 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-frt7c_e25ab82f-72fb-4cde-89c4-0a0dd80f5feb/console-operator/2.log" Jul 07 19:15:17.483390 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:15:17.483372 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-frt7c_e25ab82f-72fb-4cde-89c4-0a0dd80f5feb/console-operator/2.log" Jul 07 19:15:17.488930 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:15:17.488908 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zzwbq_4f0c5d5c-147f-426a-965d-8886e5e79984/ovn-acl-logging/0.log" Jul 07 19:15:17.489588 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:15:17.489568 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zzwbq_4f0c5d5c-147f-426a-965d-8886e5e79984/ovn-acl-logging/0.log" Jul 07 19:17:07.813467 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:17:07.813435 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:19:05.045136 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:19:05.045080 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:19:05.135250 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:19:05.135221 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:20:10.052163 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:20:10.052128 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:20:10.146338 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:20:10.146304 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:20:17.514089 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:20:17.514059 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-frt7c_e25ab82f-72fb-4cde-89c4-0a0dd80f5feb/console-operator/2.log" Jul 07 19:20:17.517704 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:20:17.517682 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-frt7c_e25ab82f-72fb-4cde-89c4-0a0dd80f5feb/console-operator/2.log" Jul 07 19:20:17.520274 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:20:17.520256 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zzwbq_4f0c5d5c-147f-426a-965d-8886e5e79984/ovn-acl-logging/0.log" Jul 07 19:20:17.523899 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:20:17.523882 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zzwbq_4f0c5d5c-147f-426a-965d-8886e5e79984/ovn-acl-logging/0.log" Jul 07 19:22:12.594564 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:22:12.594533 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:25:17.546154 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:25:17.546119 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-frt7c_e25ab82f-72fb-4cde-89c4-0a0dd80f5feb/console-operator/2.log" Jul 07 19:25:17.550462 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:25:17.550440 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-frt7c_e25ab82f-72fb-4cde-89c4-0a0dd80f5feb/console-operator/2.log" Jul 07 19:25:17.552836 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:25:17.552817 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zzwbq_4f0c5d5c-147f-426a-965d-8886e5e79984/ovn-acl-logging/0.log" Jul 07 19:25:17.557178 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:25:17.557163 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zzwbq_4f0c5d5c-147f-426a-965d-8886e5e79984/ovn-acl-logging/0.log" Jul 07 19:25:48.719014 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:25:48.718917 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:25:55.901483 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:25:55.901439 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:26:04.209932 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:04.209896 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:26:09.282108 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:09.282071 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:26:12.113459 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:12.113431 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:26:12.212745 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:12.212711 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:26:12.585217 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:12.585184 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:26:20.659719 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:20.659679 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:26:20.943090 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:20.943005 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:26:20.963697 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:20.963661 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-736468f2-openshift-default-578644dff-c98hp"] Jul 07 19:26:20.967278 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:20.967260 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-736468f2-openshift-default-578644dff-c98hp" Jul 07 19:26:20.969606 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:20.969582 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-736468f2-openshift-default-dockercfg-kwbn5\"" Jul 07 19:26:21.030546 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:21.030511 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-736468f2-openshift-default-578644dff-c98hp"] Jul 07 19:26:21.067012 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:21.066938 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1fc689f2-8ddd-4596-a9fb-acfbc45b9fee-istiod-ca-cert\") pod \"e2e-ait-736468f2-openshift-default-578644dff-c98hp\" (UID: \"1fc689f2-8ddd-4596-a9fb-acfbc45b9fee\") " pod="openshift-ingress/e2e-ait-736468f2-openshift-default-578644dff-c98hp" Jul 07 19:26:21.067197 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:21.067041 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fc689f2-8ddd-4596-a9fb-acfbc45b9fee-workload-socket\") pod \"e2e-ait-736468f2-openshift-default-578644dff-c98hp\" (UID: \"1fc689f2-8ddd-4596-a9fb-acfbc45b9fee\") " pod="openshift-ingress/e2e-ait-736468f2-openshift-default-578644dff-c98hp" Jul 07 19:26:21.067197 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:21.067122 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1fc689f2-8ddd-4596-a9fb-acfbc45b9fee-workload-certs\") pod \"e2e-ait-736468f2-openshift-default-578644dff-c98hp\" (UID: \"1fc689f2-8ddd-4596-a9fb-acfbc45b9fee\") " pod="openshift-ingress/e2e-ait-736468f2-openshift-default-578644dff-c98hp" Jul 07 19:26:21.067197 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:21.067177 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f67zw\" (UniqueName: \"kubernetes.io/projected/1fc689f2-8ddd-4596-a9fb-acfbc45b9fee-kube-api-access-f67zw\") pod \"e2e-ait-736468f2-openshift-default-578644dff-c98hp\" (UID: \"1fc689f2-8ddd-4596-a9fb-acfbc45b9fee\") " pod="openshift-ingress/e2e-ait-736468f2-openshift-default-578644dff-c98hp" Jul 07 19:26:21.067331 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:21.067210 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1fc689f2-8ddd-4596-a9fb-acfbc45b9fee-istio-podinfo\") pod \"e2e-ait-736468f2-openshift-default-578644dff-c98hp\" (UID: \"1fc689f2-8ddd-4596-a9fb-acfbc45b9fee\") " pod="openshift-ingress/e2e-ait-736468f2-openshift-default-578644dff-c98hp" Jul 07 19:26:21.067331 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:21.067252 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fc689f2-8ddd-4596-a9fb-acfbc45b9fee-credential-socket\") pod \"e2e-ait-736468f2-openshift-default-578644dff-c98hp\" (UID: \"1fc689f2-8ddd-4596-a9fb-acfbc45b9fee\") " pod="openshift-ingress/e2e-ait-736468f2-openshift-default-578644dff-c98hp" Jul 07 19:26:21.067331 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:21.067271 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1fc689f2-8ddd-4596-a9fb-acfbc45b9fee-istio-data\") pod \"e2e-ait-736468f2-openshift-default-578644dff-c98hp\" (UID: \"1fc689f2-8ddd-4596-a9fb-acfbc45b9fee\") " pod="openshift-ingress/e2e-ait-736468f2-openshift-default-578644dff-c98hp" Jul 07 19:26:21.067477 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:21.067329 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1fc689f2-8ddd-4596-a9fb-acfbc45b9fee-istio-token\") pod \"e2e-ait-736468f2-openshift-default-578644dff-c98hp\" (UID: \"1fc689f2-8ddd-4596-a9fb-acfbc45b9fee\") " pod="openshift-ingress/e2e-ait-736468f2-openshift-default-578644dff-c98hp" Jul 07 19:26:21.067477 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:21.067385 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1fc689f2-8ddd-4596-a9fb-acfbc45b9fee-istio-envoy\") pod \"e2e-ait-736468f2-openshift-default-578644dff-c98hp\" (UID: \"1fc689f2-8ddd-4596-a9fb-acfbc45b9fee\") " pod="openshift-ingress/e2e-ait-736468f2-openshift-default-578644dff-c98hp" Jul 07 19:26:21.127209 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:21.127179 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:26:21.168256 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:21.168222 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1fc689f2-8ddd-4596-a9fb-acfbc45b9fee-workload-certs\") pod \"e2e-ait-736468f2-openshift-default-578644dff-c98hp\" (UID: \"1fc689f2-8ddd-4596-a9fb-acfbc45b9fee\") " pod="openshift-ingress/e2e-ait-736468f2-openshift-default-578644dff-c98hp" Jul 07 19:26:21.168431 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:21.168283 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f67zw\" (UniqueName: \"kubernetes.io/projected/1fc689f2-8ddd-4596-a9fb-acfbc45b9fee-kube-api-access-f67zw\") pod \"e2e-ait-736468f2-openshift-default-578644dff-c98hp\" (UID: \"1fc689f2-8ddd-4596-a9fb-acfbc45b9fee\") " pod="openshift-ingress/e2e-ait-736468f2-openshift-default-578644dff-c98hp" Jul 07 19:26:21.168431 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:21.168330 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1fc689f2-8ddd-4596-a9fb-acfbc45b9fee-istio-podinfo\") pod \"e2e-ait-736468f2-openshift-default-578644dff-c98hp\" (UID: \"1fc689f2-8ddd-4596-a9fb-acfbc45b9fee\") " pod="openshift-ingress/e2e-ait-736468f2-openshift-default-578644dff-c98hp" Jul 07 19:26:21.168431 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:21.168385 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fc689f2-8ddd-4596-a9fb-acfbc45b9fee-credential-socket\") pod \"e2e-ait-736468f2-openshift-default-578644dff-c98hp\" (UID: \"1fc689f2-8ddd-4596-a9fb-acfbc45b9fee\") " pod="openshift-ingress/e2e-ait-736468f2-openshift-default-578644dff-c98hp" Jul 07 19:26:21.168564 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:21.168500 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1fc689f2-8ddd-4596-a9fb-acfbc45b9fee-istio-data\") pod \"e2e-ait-736468f2-openshift-default-578644dff-c98hp\" (UID: \"1fc689f2-8ddd-4596-a9fb-acfbc45b9fee\") " pod="openshift-ingress/e2e-ait-736468f2-openshift-default-578644dff-c98hp" Jul 07 19:26:21.168640 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:21.168563 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1fc689f2-8ddd-4596-a9fb-acfbc45b9fee-istio-token\") pod \"e2e-ait-736468f2-openshift-default-578644dff-c98hp\" (UID: \"1fc689f2-8ddd-4596-a9fb-acfbc45b9fee\") " pod="openshift-ingress/e2e-ait-736468f2-openshift-default-578644dff-c98hp" Jul 07 19:26:21.168640 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:21.168588 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1fc689f2-8ddd-4596-a9fb-acfbc45b9fee-istio-envoy\") pod \"e2e-ait-736468f2-openshift-default-578644dff-c98hp\" (UID: \"1fc689f2-8ddd-4596-a9fb-acfbc45b9fee\") " pod="openshift-ingress/e2e-ait-736468f2-openshift-default-578644dff-c98hp" Jul 07 19:26:21.168640 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:21.168631 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1fc689f2-8ddd-4596-a9fb-acfbc45b9fee-istiod-ca-cert\") pod \"e2e-ait-736468f2-openshift-default-578644dff-c98hp\" (UID: \"1fc689f2-8ddd-4596-a9fb-acfbc45b9fee\") " pod="openshift-ingress/e2e-ait-736468f2-openshift-default-578644dff-c98hp" Jul 07 19:26:21.168810 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:21.168655 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fc689f2-8ddd-4596-a9fb-acfbc45b9fee-workload-socket\") pod \"e2e-ait-736468f2-openshift-default-578644dff-c98hp\" (UID: \"1fc689f2-8ddd-4596-a9fb-acfbc45b9fee\") " pod="openshift-ingress/e2e-ait-736468f2-openshift-default-578644dff-c98hp" Jul 07 19:26:21.168810 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:21.168735 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1fc689f2-8ddd-4596-a9fb-acfbc45b9fee-workload-certs\") pod \"e2e-ait-736468f2-openshift-default-578644dff-c98hp\" (UID: \"1fc689f2-8ddd-4596-a9fb-acfbc45b9fee\") " pod="openshift-ingress/e2e-ait-736468f2-openshift-default-578644dff-c98hp" Jul 07 19:26:21.168810 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:21.168737 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fc689f2-8ddd-4596-a9fb-acfbc45b9fee-credential-socket\") pod \"e2e-ait-736468f2-openshift-default-578644dff-c98hp\" (UID: \"1fc689f2-8ddd-4596-a9fb-acfbc45b9fee\") " pod="openshift-ingress/e2e-ait-736468f2-openshift-default-578644dff-c98hp" Jul 07 19:26:21.169021 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:21.168891 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1fc689f2-8ddd-4596-a9fb-acfbc45b9fee-istio-data\") pod \"e2e-ait-736468f2-openshift-default-578644dff-c98hp\" (UID: \"1fc689f2-8ddd-4596-a9fb-acfbc45b9fee\") " pod="openshift-ingress/e2e-ait-736468f2-openshift-default-578644dff-c98hp" Jul 07 19:26:21.169118 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:21.169093 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fc689f2-8ddd-4596-a9fb-acfbc45b9fee-workload-socket\") pod \"e2e-ait-736468f2-openshift-default-578644dff-c98hp\" (UID: \"1fc689f2-8ddd-4596-a9fb-acfbc45b9fee\") " pod="openshift-ingress/e2e-ait-736468f2-openshift-default-578644dff-c98hp" Jul 07 19:26:21.169405 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:21.169385 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1fc689f2-8ddd-4596-a9fb-acfbc45b9fee-istiod-ca-cert\") pod \"e2e-ait-736468f2-openshift-default-578644dff-c98hp\" (UID: \"1fc689f2-8ddd-4596-a9fb-acfbc45b9fee\") " pod="openshift-ingress/e2e-ait-736468f2-openshift-default-578644dff-c98hp" Jul 07 19:26:21.171026 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:21.170993 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1fc689f2-8ddd-4596-a9fb-acfbc45b9fee-istio-envoy\") pod \"e2e-ait-736468f2-openshift-default-578644dff-c98hp\" (UID: \"1fc689f2-8ddd-4596-a9fb-acfbc45b9fee\") " pod="openshift-ingress/e2e-ait-736468f2-openshift-default-578644dff-c98hp" Jul 07 19:26:21.171162 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:21.171137 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1fc689f2-8ddd-4596-a9fb-acfbc45b9fee-istio-podinfo\") pod \"e2e-ait-736468f2-openshift-default-578644dff-c98hp\" (UID: \"1fc689f2-8ddd-4596-a9fb-acfbc45b9fee\") " pod="openshift-ingress/e2e-ait-736468f2-openshift-default-578644dff-c98hp" Jul 07 19:26:21.179774 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:21.179750 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1fc689f2-8ddd-4596-a9fb-acfbc45b9fee-istio-token\") pod \"e2e-ait-736468f2-openshift-default-578644dff-c98hp\" (UID: \"1fc689f2-8ddd-4596-a9fb-acfbc45b9fee\") " pod="openshift-ingress/e2e-ait-736468f2-openshift-default-578644dff-c98hp" Jul 07 19:26:21.180728 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:21.180707 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f67zw\" (UniqueName: \"kubernetes.io/projected/1fc689f2-8ddd-4596-a9fb-acfbc45b9fee-kube-api-access-f67zw\") pod \"e2e-ait-736468f2-openshift-default-578644dff-c98hp\" (UID: \"1fc689f2-8ddd-4596-a9fb-acfbc45b9fee\") " pod="openshift-ingress/e2e-ait-736468f2-openshift-default-578644dff-c98hp" Jul 07 19:26:21.279030 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:21.278926 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-736468f2-openshift-default-578644dff-c98hp" Jul 07 19:26:21.428402 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:21.428377 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-736468f2-openshift-default-578644dff-c98hp"] Jul 07 19:26:21.434344 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:21.434315 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 07 19:26:21.434449 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:21.434384 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 07 19:26:21.434449 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:21.434412 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 07 19:26:21.936718 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:21.936679 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-736468f2-openshift-default-578644dff-c98hp" event={"ID":"1fc689f2-8ddd-4596-a9fb-acfbc45b9fee","Type":"ContainerStarted","Data":"323cf666fa3cc9522529d21d5717ee2e826582ff3ac2fa7936336e8ed0484dc5"} Jul 07 19:26:21.937124 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:21.936731 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-736468f2-openshift-default-578644dff-c98hp" event={"ID":"1fc689f2-8ddd-4596-a9fb-acfbc45b9fee","Type":"ContainerStarted","Data":"6622c09bd292884345e6ab84a12189c08f98c165e47eb1d559bf500935e2ecec"} Jul 07 19:26:21.960389 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:21.960345 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-736468f2-openshift-default-578644dff-c98hp" podStartSLOduration=1.960330265 podStartE2EDuration="1.960330265s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:26:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:26:21.9588927 +0000 UTC m=+1564.974896880" watchObservedRunningTime="2026-07-07 19:26:21.960330265 +0000 UTC m=+1564.976334446" Jul 07 19:26:22.190354 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:22.190271 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:26:22.279492 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:22.279463 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-736468f2-openshift-default-578644dff-c98hp" Jul 07 19:26:22.284505 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:22.284479 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-736468f2-openshift-default-578644dff-c98hp" Jul 07 19:26:22.298842 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:22.298809 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:26:22.466024 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:22.465795 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:26:22.704365 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:22.704328 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:26:22.942131 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:22.942102 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-736468f2-openshift-default-578644dff-c98hp" Jul 07 19:26:22.943185 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:22.943167 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-736468f2-openshift-default-578644dff-c98hp" Jul 07 19:26:23.212798 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:23.212719 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:26:23.316443 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:23.316410 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:26:25.315773 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:25.315744 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:26:27.145534 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:27.145498 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:26:27.295868 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:27.295833 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-736468f2-openshift-default-578644dff-c98hp"] Jul 07 19:26:27.296086 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:27.296059 2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-736468f2-openshift-default-578644dff-c98hp" podUID="1fc689f2-8ddd-4596-a9fb-acfbc45b9fee" containerName="istio-proxy" containerID="cri-o://323cf666fa3cc9522529d21d5717ee2e826582ff3ac2fa7936336e8ed0484dc5" gracePeriod=30 Jul 07 19:26:27.326240 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:27.326189 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:26:27.455270 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:27.455174 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:26:27.632432 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:27.632393 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:26:29.154487 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:29.154457 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:26:33.353482 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:33.353452 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-736468f2-openshift-default-578644dff-c98hp" Jul 07 19:26:33.384009 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:33.383944 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fc689f2-8ddd-4596-a9fb-acfbc45b9fee-credential-socket\") pod \"1fc689f2-8ddd-4596-a9fb-acfbc45b9fee\" (UID: \"1fc689f2-8ddd-4596-a9fb-acfbc45b9fee\") " Jul 07 19:26:33.384168 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:33.384030 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1fc689f2-8ddd-4596-a9fb-acfbc45b9fee-istio-data\") pod \"1fc689f2-8ddd-4596-a9fb-acfbc45b9fee\" (UID: \"1fc689f2-8ddd-4596-a9fb-acfbc45b9fee\") " Jul 07 19:26:33.384168 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:33.384061 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1fc689f2-8ddd-4596-a9fb-acfbc45b9fee-istio-token\") pod \"1fc689f2-8ddd-4596-a9fb-acfbc45b9fee\" (UID: \"1fc689f2-8ddd-4596-a9fb-acfbc45b9fee\") " Jul 07 19:26:33.384168 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:33.384097 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1fc689f2-8ddd-4596-a9fb-acfbc45b9fee-istio-envoy\") pod \"1fc689f2-8ddd-4596-a9fb-acfbc45b9fee\" (UID: \"1fc689f2-8ddd-4596-a9fb-acfbc45b9fee\") " Jul 07 19:26:33.384168 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:33.384127 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1fc689f2-8ddd-4596-a9fb-acfbc45b9fee-istio-podinfo\") pod \"1fc689f2-8ddd-4596-a9fb-acfbc45b9fee\" (UID: \"1fc689f2-8ddd-4596-a9fb-acfbc45b9fee\") " Jul 07 19:26:33.384390 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:33.384245 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fc689f2-8ddd-4596-a9fb-acfbc45b9fee-workload-socket\") pod \"1fc689f2-8ddd-4596-a9fb-acfbc45b9fee\" (UID: \"1fc689f2-8ddd-4596-a9fb-acfbc45b9fee\") " Jul 07 19:26:33.384390 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:33.384246 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fc689f2-8ddd-4596-a9fb-acfbc45b9fee-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "1fc689f2-8ddd-4596-a9fb-acfbc45b9fee" (UID: "1fc689f2-8ddd-4596-a9fb-acfbc45b9fee"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:26:33.384390 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:33.384271 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1fc689f2-8ddd-4596-a9fb-acfbc45b9fee-workload-certs\") pod \"1fc689f2-8ddd-4596-a9fb-acfbc45b9fee\" (UID: \"1fc689f2-8ddd-4596-a9fb-acfbc45b9fee\") " Jul 07 19:26:33.384390 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:33.384358 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f67zw\" (UniqueName: \"kubernetes.io/projected/1fc689f2-8ddd-4596-a9fb-acfbc45b9fee-kube-api-access-f67zw\") pod \"1fc689f2-8ddd-4596-a9fb-acfbc45b9fee\" (UID: \"1fc689f2-8ddd-4596-a9fb-acfbc45b9fee\") " Jul 07 19:26:33.384621 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:33.384430 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1fc689f2-8ddd-4596-a9fb-acfbc45b9fee-istiod-ca-cert\") pod \"1fc689f2-8ddd-4596-a9fb-acfbc45b9fee\" (UID: \"1fc689f2-8ddd-4596-a9fb-acfbc45b9fee\") " Jul 07 19:26:33.384621 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:33.384494 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fc689f2-8ddd-4596-a9fb-acfbc45b9fee-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "1fc689f2-8ddd-4596-a9fb-acfbc45b9fee" (UID: "1fc689f2-8ddd-4596-a9fb-acfbc45b9fee"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:26:33.384724 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:33.384709 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fc689f2-8ddd-4596-a9fb-acfbc45b9fee-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "1fc689f2-8ddd-4596-a9fb-acfbc45b9fee" (UID: "1fc689f2-8ddd-4596-a9fb-acfbc45b9fee"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:26:33.384885 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:33.384841 2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fc689f2-8ddd-4596-a9fb-acfbc45b9fee-workload-socket\") on node \"ip-10-0-142-134.ec2.internal\" DevicePath \"\"" Jul 07 19:26:33.385117 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:33.385090 2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1fc689f2-8ddd-4596-a9fb-acfbc45b9fee-workload-certs\") on node \"ip-10-0-142-134.ec2.internal\" DevicePath \"\"" Jul 07 19:26:33.385117 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:33.385118 2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fc689f2-8ddd-4596-a9fb-acfbc45b9fee-credential-socket\") on node \"ip-10-0-142-134.ec2.internal\" DevicePath \"\"" Jul 07 19:26:33.385298 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:33.385032 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1fc689f2-8ddd-4596-a9fb-acfbc45b9fee-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "1fc689f2-8ddd-4596-a9fb-acfbc45b9fee" (UID: "1fc689f2-8ddd-4596-a9fb-acfbc45b9fee"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 19:26:33.385298 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:33.385107 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fc689f2-8ddd-4596-a9fb-acfbc45b9fee-istio-data" (OuterVolumeSpecName: "istio-data") pod "1fc689f2-8ddd-4596-a9fb-acfbc45b9fee" (UID: "1fc689f2-8ddd-4596-a9fb-acfbc45b9fee"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:26:33.386477 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:33.386454 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1fc689f2-8ddd-4596-a9fb-acfbc45b9fee-istio-token" (OuterVolumeSpecName: "istio-token") pod "1fc689f2-8ddd-4596-a9fb-acfbc45b9fee" (UID: "1fc689f2-8ddd-4596-a9fb-acfbc45b9fee"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 19:26:33.386562 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:33.386468 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1fc689f2-8ddd-4596-a9fb-acfbc45b9fee-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "1fc689f2-8ddd-4596-a9fb-acfbc45b9fee" (UID: "1fc689f2-8ddd-4596-a9fb-acfbc45b9fee"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 07 19:26:33.386740 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:33.386719 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fc689f2-8ddd-4596-a9fb-acfbc45b9fee-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "1fc689f2-8ddd-4596-a9fb-acfbc45b9fee" (UID: "1fc689f2-8ddd-4596-a9fb-acfbc45b9fee"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:26:33.386932 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:33.386879 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1fc689f2-8ddd-4596-a9fb-acfbc45b9fee-kube-api-access-f67zw" (OuterVolumeSpecName: "kube-api-access-f67zw") pod "1fc689f2-8ddd-4596-a9fb-acfbc45b9fee" (UID: "1fc689f2-8ddd-4596-a9fb-acfbc45b9fee"). InnerVolumeSpecName "kube-api-access-f67zw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 19:26:33.485712 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:33.485690 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f67zw\" (UniqueName: \"kubernetes.io/projected/1fc689f2-8ddd-4596-a9fb-acfbc45b9fee-kube-api-access-f67zw\") on node \"ip-10-0-142-134.ec2.internal\" DevicePath \"\"" Jul 07 19:26:33.485712 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:33.485712 2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1fc689f2-8ddd-4596-a9fb-acfbc45b9fee-istiod-ca-cert\") on node \"ip-10-0-142-134.ec2.internal\" DevicePath \"\"" Jul 07 19:26:33.485873 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:33.485721 2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1fc689f2-8ddd-4596-a9fb-acfbc45b9fee-istio-data\") on node \"ip-10-0-142-134.ec2.internal\" DevicePath \"\"" Jul 07 19:26:33.485873 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:33.485729 2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1fc689f2-8ddd-4596-a9fb-acfbc45b9fee-istio-token\") on node \"ip-10-0-142-134.ec2.internal\" DevicePath \"\"" Jul 07 19:26:33.485873 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:33.485737 2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1fc689f2-8ddd-4596-a9fb-acfbc45b9fee-istio-envoy\") on node \"ip-10-0-142-134.ec2.internal\" DevicePath \"\"" Jul 07 19:26:33.485873 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:33.485744 2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1fc689f2-8ddd-4596-a9fb-acfbc45b9fee-istio-podinfo\") on node \"ip-10-0-142-134.ec2.internal\" DevicePath \"\"" Jul 07 19:26:34.015168 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:34.015132 2563 generic.go:356] "Generic (PLEG): container finished" podID="1fc689f2-8ddd-4596-a9fb-acfbc45b9fee" containerID="323cf666fa3cc9522529d21d5717ee2e826582ff3ac2fa7936336e8ed0484dc5" exitCode=0 Jul 07 19:26:34.015168 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:34.015170 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-736468f2-openshift-default-578644dff-c98hp" event={"ID":"1fc689f2-8ddd-4596-a9fb-acfbc45b9fee","Type":"ContainerDied","Data":"323cf666fa3cc9522529d21d5717ee2e826582ff3ac2fa7936336e8ed0484dc5"} Jul 07 19:26:34.015435 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:34.015179 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-736468f2-openshift-default-578644dff-c98hp" Jul 07 19:26:34.015435 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:34.015212 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-736468f2-openshift-default-578644dff-c98hp" event={"ID":"1fc689f2-8ddd-4596-a9fb-acfbc45b9fee","Type":"ContainerDied","Data":"6622c09bd292884345e6ab84a12189c08f98c165e47eb1d559bf500935e2ecec"} Jul 07 19:26:34.015435 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:34.015232 2563 scope.go:117] "RemoveContainer" containerID="323cf666fa3cc9522529d21d5717ee2e826582ff3ac2fa7936336e8ed0484dc5" Jul 07 19:26:34.023907 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:34.023888 2563 scope.go:117] "RemoveContainer" containerID="323cf666fa3cc9522529d21d5717ee2e826582ff3ac2fa7936336e8ed0484dc5" Jul 07 19:26:34.024218 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:26:34.024197 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"323cf666fa3cc9522529d21d5717ee2e826582ff3ac2fa7936336e8ed0484dc5\": container with ID starting with 323cf666fa3cc9522529d21d5717ee2e826582ff3ac2fa7936336e8ed0484dc5 not found: ID does not exist" containerID="323cf666fa3cc9522529d21d5717ee2e826582ff3ac2fa7936336e8ed0484dc5" Jul 07 19:26:34.024308 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:34.024224 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"323cf666fa3cc9522529d21d5717ee2e826582ff3ac2fa7936336e8ed0484dc5"} err="failed to get container status \"323cf666fa3cc9522529d21d5717ee2e826582ff3ac2fa7936336e8ed0484dc5\": rpc error: code = NotFound desc = could not find container \"323cf666fa3cc9522529d21d5717ee2e826582ff3ac2fa7936336e8ed0484dc5\": container with ID starting with 323cf666fa3cc9522529d21d5717ee2e826582ff3ac2fa7936336e8ed0484dc5 not found: ID does not exist" Jul 07 19:26:34.038912 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:34.038884 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-736468f2-openshift-default-578644dff-c98hp"] Jul 07 19:26:34.048936 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:34.048911 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-736468f2-openshift-default-578644dff-c98hp"] Jul 07 19:26:34.303607 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:34.303563 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:26:34.409912 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:34.409861 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:26:34.495764 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:34.495733 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-114401d3-openshift-default-7f497d778c-9qglm"] Jul 07 19:26:34.496341 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:34.496318 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1fc689f2-8ddd-4596-a9fb-acfbc45b9fee" containerName="istio-proxy" Jul 07 19:26:34.496341 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:34.496342 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="1fc689f2-8ddd-4596-a9fb-acfbc45b9fee" containerName="istio-proxy" Jul 07 19:26:34.496515 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:34.496459 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="1fc689f2-8ddd-4596-a9fb-acfbc45b9fee" containerName="istio-proxy" Jul 07 19:26:34.501112 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:34.501092 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-114401d3-openshift-default-7f497d778c-9qglm" Jul 07 19:26:34.503313 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:34.503290 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-114401d3-openshift-default-dockercfg-c5gr8\"" Jul 07 19:26:34.513842 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:34.513821 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-114401d3-openshift-default-7f497d778c-9qglm"] Jul 07 19:26:34.566473 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:34.566391 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:26:34.595794 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:34.595762 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50576a4b-363f-4470-8dce-08694bf4a621-istio-data\") pod \"e2e-ait-114401d3-openshift-default-7f497d778c-9qglm\" (UID: \"50576a4b-363f-4470-8dce-08694bf4a621\") " pod="openshift-ingress/e2e-ait-114401d3-openshift-default-7f497d778c-9qglm" Jul 07 19:26:34.595997 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:34.595820 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50576a4b-363f-4470-8dce-08694bf4a621-istio-podinfo\") pod \"e2e-ait-114401d3-openshift-default-7f497d778c-9qglm\" (UID: \"50576a4b-363f-4470-8dce-08694bf4a621\") " pod="openshift-ingress/e2e-ait-114401d3-openshift-default-7f497d778c-9qglm" Jul 07 19:26:34.595997 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:34.595900 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50576a4b-363f-4470-8dce-08694bf4a621-credential-socket\") pod \"e2e-ait-114401d3-openshift-default-7f497d778c-9qglm\" (UID: \"50576a4b-363f-4470-8dce-08694bf4a621\") " pod="openshift-ingress/e2e-ait-114401d3-openshift-default-7f497d778c-9qglm" Jul 07 19:26:34.595997 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:34.595971 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50576a4b-363f-4470-8dce-08694bf4a621-workload-socket\") pod \"e2e-ait-114401d3-openshift-default-7f497d778c-9qglm\" (UID: \"50576a4b-363f-4470-8dce-08694bf4a621\") " pod="openshift-ingress/e2e-ait-114401d3-openshift-default-7f497d778c-9qglm" Jul 07 19:26:34.596149 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:34.596021 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50576a4b-363f-4470-8dce-08694bf4a621-workload-certs\") pod \"e2e-ait-114401d3-openshift-default-7f497d778c-9qglm\" (UID: \"50576a4b-363f-4470-8dce-08694bf4a621\") " pod="openshift-ingress/e2e-ait-114401d3-openshift-default-7f497d778c-9qglm" Jul 07 19:26:34.596149 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:34.596079 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50576a4b-363f-4470-8dce-08694bf4a621-istio-token\") pod \"e2e-ait-114401d3-openshift-default-7f497d778c-9qglm\" (UID: \"50576a4b-363f-4470-8dce-08694bf4a621\") " pod="openshift-ingress/e2e-ait-114401d3-openshift-default-7f497d778c-9qglm" Jul 07 19:26:34.596149 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:34.596101 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50576a4b-363f-4470-8dce-08694bf4a621-istio-envoy\") pod \"e2e-ait-114401d3-openshift-default-7f497d778c-9qglm\" (UID: \"50576a4b-363f-4470-8dce-08694bf4a621\") " pod="openshift-ingress/e2e-ait-114401d3-openshift-default-7f497d778c-9qglm" Jul 07 19:26:34.596149 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:34.596117 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhpxk\" (UniqueName: \"kubernetes.io/projected/50576a4b-363f-4470-8dce-08694bf4a621-kube-api-access-nhpxk\") pod \"e2e-ait-114401d3-openshift-default-7f497d778c-9qglm\" (UID: \"50576a4b-363f-4470-8dce-08694bf4a621\") " pod="openshift-ingress/e2e-ait-114401d3-openshift-default-7f497d778c-9qglm" Jul 07 19:26:34.596338 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:34.596200 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50576a4b-363f-4470-8dce-08694bf4a621-istiod-ca-cert\") pod \"e2e-ait-114401d3-openshift-default-7f497d778c-9qglm\" (UID: \"50576a4b-363f-4470-8dce-08694bf4a621\") " pod="openshift-ingress/e2e-ait-114401d3-openshift-default-7f497d778c-9qglm" Jul 07 19:26:34.671243 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:34.671206 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:26:34.697725 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:34.697690 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50576a4b-363f-4470-8dce-08694bf4a621-credential-socket\") pod \"e2e-ait-114401d3-openshift-default-7f497d778c-9qglm\" (UID: \"50576a4b-363f-4470-8dce-08694bf4a621\") " pod="openshift-ingress/e2e-ait-114401d3-openshift-default-7f497d778c-9qglm" Jul 07 19:26:34.697725 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:34.697725 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50576a4b-363f-4470-8dce-08694bf4a621-workload-socket\") pod \"e2e-ait-114401d3-openshift-default-7f497d778c-9qglm\" (UID: \"50576a4b-363f-4470-8dce-08694bf4a621\") " pod="openshift-ingress/e2e-ait-114401d3-openshift-default-7f497d778c-9qglm" Jul 07 19:26:34.698061 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:34.697753 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50576a4b-363f-4470-8dce-08694bf4a621-workload-certs\") pod \"e2e-ait-114401d3-openshift-default-7f497d778c-9qglm\" (UID: \"50576a4b-363f-4470-8dce-08694bf4a621\") " pod="openshift-ingress/e2e-ait-114401d3-openshift-default-7f497d778c-9qglm" Jul 07 19:26:34.698061 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:34.697792 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50576a4b-363f-4470-8dce-08694bf4a621-istio-token\") pod \"e2e-ait-114401d3-openshift-default-7f497d778c-9qglm\" (UID: \"50576a4b-363f-4470-8dce-08694bf4a621\") " pod="openshift-ingress/e2e-ait-114401d3-openshift-default-7f497d778c-9qglm" Jul 07 19:26:34.698061 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:34.697812 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50576a4b-363f-4470-8dce-08694bf4a621-istio-envoy\") pod \"e2e-ait-114401d3-openshift-default-7f497d778c-9qglm\" (UID: \"50576a4b-363f-4470-8dce-08694bf4a621\") " pod="openshift-ingress/e2e-ait-114401d3-openshift-default-7f497d778c-9qglm" Jul 07 19:26:34.698061 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:34.697830 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhpxk\" (UniqueName: \"kubernetes.io/projected/50576a4b-363f-4470-8dce-08694bf4a621-kube-api-access-nhpxk\") pod \"e2e-ait-114401d3-openshift-default-7f497d778c-9qglm\" (UID: \"50576a4b-363f-4470-8dce-08694bf4a621\") " pod="openshift-ingress/e2e-ait-114401d3-openshift-default-7f497d778c-9qglm" Jul 07 19:26:34.698061 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:34.697885 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50576a4b-363f-4470-8dce-08694bf4a621-istiod-ca-cert\") pod \"e2e-ait-114401d3-openshift-default-7f497d778c-9qglm\" (UID: \"50576a4b-363f-4470-8dce-08694bf4a621\") " pod="openshift-ingress/e2e-ait-114401d3-openshift-default-7f497d778c-9qglm" Jul 07 19:26:34.698061 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:34.697924 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50576a4b-363f-4470-8dce-08694bf4a621-istio-data\") pod \"e2e-ait-114401d3-openshift-default-7f497d778c-9qglm\" (UID: \"50576a4b-363f-4470-8dce-08694bf4a621\") " pod="openshift-ingress/e2e-ait-114401d3-openshift-default-7f497d778c-9qglm" Jul 07 19:26:34.698061 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:34.698015 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50576a4b-363f-4470-8dce-08694bf4a621-istio-podinfo\") pod \"e2e-ait-114401d3-openshift-default-7f497d778c-9qglm\" (UID: \"50576a4b-363f-4470-8dce-08694bf4a621\") " pod="openshift-ingress/e2e-ait-114401d3-openshift-default-7f497d778c-9qglm" Jul 07 19:26:34.698433 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:34.698212 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50576a4b-363f-4470-8dce-08694bf4a621-workload-socket\") pod \"e2e-ait-114401d3-openshift-default-7f497d778c-9qglm\" (UID: \"50576a4b-363f-4470-8dce-08694bf4a621\") " pod="openshift-ingress/e2e-ait-114401d3-openshift-default-7f497d778c-9qglm" Jul 07 19:26:34.698433 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:34.698248 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50576a4b-363f-4470-8dce-08694bf4a621-credential-socket\") pod \"e2e-ait-114401d3-openshift-default-7f497d778c-9qglm\" (UID: \"50576a4b-363f-4470-8dce-08694bf4a621\") " pod="openshift-ingress/e2e-ait-114401d3-openshift-default-7f497d778c-9qglm" Jul 07 19:26:34.698433 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:34.698279 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50576a4b-363f-4470-8dce-08694bf4a621-workload-certs\") pod \"e2e-ait-114401d3-openshift-default-7f497d778c-9qglm\" (UID: \"50576a4b-363f-4470-8dce-08694bf4a621\") " pod="openshift-ingress/e2e-ait-114401d3-openshift-default-7f497d778c-9qglm" Jul 07 19:26:34.698603 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:34.698527 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50576a4b-363f-4470-8dce-08694bf4a621-istio-data\") pod \"e2e-ait-114401d3-openshift-default-7f497d778c-9qglm\" (UID: \"50576a4b-363f-4470-8dce-08694bf4a621\") " pod="openshift-ingress/e2e-ait-114401d3-openshift-default-7f497d778c-9qglm" Jul 07 19:26:34.698909 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:34.698886 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50576a4b-363f-4470-8dce-08694bf4a621-istiod-ca-cert\") pod \"e2e-ait-114401d3-openshift-default-7f497d778c-9qglm\" (UID: \"50576a4b-363f-4470-8dce-08694bf4a621\") " pod="openshift-ingress/e2e-ait-114401d3-openshift-default-7f497d778c-9qglm" Jul 07 19:26:34.700141 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:34.700122 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50576a4b-363f-4470-8dce-08694bf4a621-istio-envoy\") pod \"e2e-ait-114401d3-openshift-default-7f497d778c-9qglm\" (UID: \"50576a4b-363f-4470-8dce-08694bf4a621\") " pod="openshift-ingress/e2e-ait-114401d3-openshift-default-7f497d778c-9qglm" Jul 07 19:26:34.700401 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:34.700384 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50576a4b-363f-4470-8dce-08694bf4a621-istio-podinfo\") pod \"e2e-ait-114401d3-openshift-default-7f497d778c-9qglm\" (UID: \"50576a4b-363f-4470-8dce-08694bf4a621\") " pod="openshift-ingress/e2e-ait-114401d3-openshift-default-7f497d778c-9qglm" Jul 07 19:26:34.708162 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:34.708138 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50576a4b-363f-4470-8dce-08694bf4a621-istio-token\") pod \"e2e-ait-114401d3-openshift-default-7f497d778c-9qglm\" (UID: \"50576a4b-363f-4470-8dce-08694bf4a621\") " pod="openshift-ingress/e2e-ait-114401d3-openshift-default-7f497d778c-9qglm" Jul 07 19:26:34.708576 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:34.708557 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhpxk\" (UniqueName: \"kubernetes.io/projected/50576a4b-363f-4470-8dce-08694bf4a621-kube-api-access-nhpxk\") pod \"e2e-ait-114401d3-openshift-default-7f497d778c-9qglm\" (UID: \"50576a4b-363f-4470-8dce-08694bf4a621\") " pod="openshift-ingress/e2e-ait-114401d3-openshift-default-7f497d778c-9qglm" Jul 07 19:26:34.812374 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:34.812331 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-114401d3-openshift-default-7f497d778c-9qglm" Jul 07 19:26:34.956565 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:34.956521 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-114401d3-openshift-default-7f497d778c-9qglm"] Jul 07 19:26:34.959118 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:26:34.959089 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50576a4b_363f_4470_8dce_08694bf4a621.slice/crio-8f08a1a49fe90f852a274b1733bb34324ec3bc143b5a38ca496914e81d007833 WatchSource:0}: Error finding container 8f08a1a49fe90f852a274b1733bb34324ec3bc143b5a38ca496914e81d007833: Status 404 returned error can't find the container with id 8f08a1a49fe90f852a274b1733bb34324ec3bc143b5a38ca496914e81d007833 Jul 07 19:26:34.961247 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:34.961215 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 07 19:26:34.961361 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:34.961298 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 07 19:26:34.961425 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:34.961367 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 07 19:26:35.023512 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:35.023485 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-114401d3-openshift-default-7f497d778c-9qglm" event={"ID":"50576a4b-363f-4470-8dce-08694bf4a621","Type":"ContainerStarted","Data":"8f08a1a49fe90f852a274b1733bb34324ec3bc143b5a38ca496914e81d007833"} Jul 07 19:26:35.477381 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:35.477347 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1fc689f2-8ddd-4596-a9fb-acfbc45b9fee" path="/var/lib/kubelet/pods/1fc689f2-8ddd-4596-a9fb-acfbc45b9fee/volumes" Jul 07 19:26:35.730168 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:35.730088 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:26:35.827346 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:35.827314 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:26:35.996080 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:35.996004 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:26:36.029894 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:36.029864 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-114401d3-openshift-default-7f497d778c-9qglm" event={"ID":"50576a4b-363f-4470-8dce-08694bf4a621","Type":"ContainerStarted","Data":"33d1b378e3e7578c4f4cdec84a3b18ce74790613b18836fff2a7933d1c0f6d6c"} Jul 07 19:26:36.054366 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:36.054304 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-114401d3-openshift-default-7f497d778c-9qglm" podStartSLOduration=2.054283804 podStartE2EDuration="2.054283804s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:26:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:26:36.052249293 +0000 UTC m=+1579.068253473" watchObservedRunningTime="2026-07-07 19:26:36.054283804 +0000 UTC m=+1579.070287985" Jul 07 19:26:36.377180 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:36.377149 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:26:36.535496 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:36.535466 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:26:36.632786 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:36.632701 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:26:36.813156 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:36.813117 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-114401d3-openshift-default-7f497d778c-9qglm" Jul 07 19:26:36.818153 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:36.818128 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-114401d3-openshift-default-7f497d778c-9qglm" Jul 07 19:26:37.034937 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:37.034901 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-114401d3-openshift-default-7f497d778c-9qglm" Jul 07 19:26:37.036023 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:37.036004 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-114401d3-openshift-default-7f497d778c-9qglm" Jul 07 19:26:37.737058 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:37.737023 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:26:40.739772 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:40.739743 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:26:41.148129 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:41.148092 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:26:46.776835 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:46.776798 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-114401d3-openshift-default-7f497d778c-9qglm"] Jul 07 19:26:46.777375 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:46.777069 2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-114401d3-openshift-default-7f497d778c-9qglm" podUID="50576a4b-363f-4470-8dce-08694bf4a621" containerName="istio-proxy" containerID="cri-o://33d1b378e3e7578c4f4cdec84a3b18ce74790613b18836fff2a7933d1c0f6d6c" gracePeriod=30 Jul 07 19:26:46.804210 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:46.804177 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:26:46.908176 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:46.908146 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:26:46.927025 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:46.927004 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-114401d3-openshift-default-7f497d778c-9qglm" Jul 07 19:26:47.037176 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:47.037148 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50576a4b-363f-4470-8dce-08694bf4a621-istio-podinfo\") pod \"50576a4b-363f-4470-8dce-08694bf4a621\" (UID: \"50576a4b-363f-4470-8dce-08694bf4a621\") " Jul 07 19:26:47.037333 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:47.037195 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nhpxk\" (UniqueName: \"kubernetes.io/projected/50576a4b-363f-4470-8dce-08694bf4a621-kube-api-access-nhpxk\") pod \"50576a4b-363f-4470-8dce-08694bf4a621\" (UID: \"50576a4b-363f-4470-8dce-08694bf4a621\") " Jul 07 19:26:47.037333 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:47.037248 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50576a4b-363f-4470-8dce-08694bf4a621-workload-socket\") pod \"50576a4b-363f-4470-8dce-08694bf4a621\" (UID: \"50576a4b-363f-4470-8dce-08694bf4a621\") " Jul 07 19:26:47.037449 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:47.037338 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50576a4b-363f-4470-8dce-08694bf4a621-istio-data\") pod \"50576a4b-363f-4470-8dce-08694bf4a621\" (UID: \"50576a4b-363f-4470-8dce-08694bf4a621\") " Jul 07 19:26:47.037449 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:47.037363 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50576a4b-363f-4470-8dce-08694bf4a621-istio-envoy\") pod \"50576a4b-363f-4470-8dce-08694bf4a621\" (UID: \"50576a4b-363f-4470-8dce-08694bf4a621\") " Jul 07 19:26:47.037449 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:47.037388 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50576a4b-363f-4470-8dce-08694bf4a621-istio-token\") pod \"50576a4b-363f-4470-8dce-08694bf4a621\" (UID: \"50576a4b-363f-4470-8dce-08694bf4a621\") " Jul 07 19:26:47.037449 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:47.037443 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50576a4b-363f-4470-8dce-08694bf4a621-istiod-ca-cert\") pod \"50576a4b-363f-4470-8dce-08694bf4a621\" (UID: \"50576a4b-363f-4470-8dce-08694bf4a621\") " Jul 07 19:26:47.037696 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:47.037475 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50576a4b-363f-4470-8dce-08694bf4a621-workload-certs\") pod \"50576a4b-363f-4470-8dce-08694bf4a621\" (UID: \"50576a4b-363f-4470-8dce-08694bf4a621\") " Jul 07 19:26:47.037696 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:47.037520 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50576a4b-363f-4470-8dce-08694bf4a621-credential-socket\") pod \"50576a4b-363f-4470-8dce-08694bf4a621\" (UID: \"50576a4b-363f-4470-8dce-08694bf4a621\") " Jul 07 19:26:47.037696 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:47.037658 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50576a4b-363f-4470-8dce-08694bf4a621-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "50576a4b-363f-4470-8dce-08694bf4a621" (UID: "50576a4b-363f-4470-8dce-08694bf4a621"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:26:47.038159 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:47.037897 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50576a4b-363f-4470-8dce-08694bf4a621-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "50576a4b-363f-4470-8dce-08694bf4a621" (UID: "50576a4b-363f-4470-8dce-08694bf4a621"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:26:47.038159 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:47.037914 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50576a4b-363f-4470-8dce-08694bf4a621-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "50576a4b-363f-4470-8dce-08694bf4a621" (UID: "50576a4b-363f-4470-8dce-08694bf4a621"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 19:26:47.038159 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:47.037926 2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50576a4b-363f-4470-8dce-08694bf4a621-workload-socket\") on node \"ip-10-0-142-134.ec2.internal\" DevicePath \"\"" Jul 07 19:26:47.038159 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:47.038102 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50576a4b-363f-4470-8dce-08694bf4a621-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "50576a4b-363f-4470-8dce-08694bf4a621" (UID: "50576a4b-363f-4470-8dce-08694bf4a621"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:26:47.038602 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:47.038580 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50576a4b-363f-4470-8dce-08694bf4a621-istio-data" (OuterVolumeSpecName: "istio-data") pod "50576a4b-363f-4470-8dce-08694bf4a621" (UID: "50576a4b-363f-4470-8dce-08694bf4a621"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:26:47.039817 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:47.039786 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50576a4b-363f-4470-8dce-08694bf4a621-istio-token" (OuterVolumeSpecName: "istio-token") pod "50576a4b-363f-4470-8dce-08694bf4a621" (UID: "50576a4b-363f-4470-8dce-08694bf4a621"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 19:26:47.039929 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:47.039903 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50576a4b-363f-4470-8dce-08694bf4a621-kube-api-access-nhpxk" (OuterVolumeSpecName: "kube-api-access-nhpxk") pod "50576a4b-363f-4470-8dce-08694bf4a621" (UID: "50576a4b-363f-4470-8dce-08694bf4a621"). InnerVolumeSpecName "kube-api-access-nhpxk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 19:26:47.040024 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:47.040001 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50576a4b-363f-4470-8dce-08694bf4a621-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "50576a4b-363f-4470-8dce-08694bf4a621" (UID: "50576a4b-363f-4470-8dce-08694bf4a621"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:26:47.040062 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:47.040021 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/50576a4b-363f-4470-8dce-08694bf4a621-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "50576a4b-363f-4470-8dce-08694bf4a621" (UID: "50576a4b-363f-4470-8dce-08694bf4a621"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 07 19:26:47.089007 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:47.088973 2563 generic.go:356] "Generic (PLEG): container finished" podID="50576a4b-363f-4470-8dce-08694bf4a621" containerID="33d1b378e3e7578c4f4cdec84a3b18ce74790613b18836fff2a7933d1c0f6d6c" exitCode=0 Jul 07 19:26:47.089175 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:47.089014 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-114401d3-openshift-default-7f497d778c-9qglm" event={"ID":"50576a4b-363f-4470-8dce-08694bf4a621","Type":"ContainerDied","Data":"33d1b378e3e7578c4f4cdec84a3b18ce74790613b18836fff2a7933d1c0f6d6c"} Jul 07 19:26:47.089175 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:47.089028 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-114401d3-openshift-default-7f497d778c-9qglm" Jul 07 19:26:47.089175 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:47.089057 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-114401d3-openshift-default-7f497d778c-9qglm" event={"ID":"50576a4b-363f-4470-8dce-08694bf4a621","Type":"ContainerDied","Data":"8f08a1a49fe90f852a274b1733bb34324ec3bc143b5a38ca496914e81d007833"} Jul 07 19:26:47.089175 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:47.089078 2563 scope.go:117] "RemoveContainer" containerID="33d1b378e3e7578c4f4cdec84a3b18ce74790613b18836fff2a7933d1c0f6d6c" Jul 07 19:26:47.098351 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:47.098334 2563 scope.go:117] "RemoveContainer" containerID="33d1b378e3e7578c4f4cdec84a3b18ce74790613b18836fff2a7933d1c0f6d6c" Jul 07 19:26:47.098594 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:26:47.098574 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33d1b378e3e7578c4f4cdec84a3b18ce74790613b18836fff2a7933d1c0f6d6c\": container with ID starting with 33d1b378e3e7578c4f4cdec84a3b18ce74790613b18836fff2a7933d1c0f6d6c not found: ID does not exist" containerID="33d1b378e3e7578c4f4cdec84a3b18ce74790613b18836fff2a7933d1c0f6d6c" Jul 07 19:26:47.098653 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:47.098603 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33d1b378e3e7578c4f4cdec84a3b18ce74790613b18836fff2a7933d1c0f6d6c"} err="failed to get container status \"33d1b378e3e7578c4f4cdec84a3b18ce74790613b18836fff2a7933d1c0f6d6c\": rpc error: code = NotFound desc = could not find container \"33d1b378e3e7578c4f4cdec84a3b18ce74790613b18836fff2a7933d1c0f6d6c\": container with ID starting with 33d1b378e3e7578c4f4cdec84a3b18ce74790613b18836fff2a7933d1c0f6d6c not found: ID does not exist" Jul 07 19:26:47.125002 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:47.124935 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-114401d3-openshift-default-7f497d778c-9qglm"] Jul 07 19:26:47.132424 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:47.132399 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-114401d3-openshift-default-7f497d778c-9qglm"] Jul 07 19:26:47.139402 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:47.139371 2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50576a4b-363f-4470-8dce-08694bf4a621-credential-socket\") on node \"ip-10-0-142-134.ec2.internal\" DevicePath \"\"" Jul 07 19:26:47.139402 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:47.139407 2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50576a4b-363f-4470-8dce-08694bf4a621-istio-podinfo\") on node \"ip-10-0-142-134.ec2.internal\" DevicePath \"\"" Jul 07 19:26:47.139516 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:47.139419 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nhpxk\" (UniqueName: \"kubernetes.io/projected/50576a4b-363f-4470-8dce-08694bf4a621-kube-api-access-nhpxk\") on node \"ip-10-0-142-134.ec2.internal\" DevicePath \"\"" Jul 07 19:26:47.139516 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:47.139427 2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50576a4b-363f-4470-8dce-08694bf4a621-istio-data\") on node \"ip-10-0-142-134.ec2.internal\" DevicePath \"\"" Jul 07 19:26:47.139516 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:47.139436 2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50576a4b-363f-4470-8dce-08694bf4a621-istio-envoy\") on node \"ip-10-0-142-134.ec2.internal\" DevicePath \"\"" Jul 07 19:26:47.139516 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:47.139443 2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50576a4b-363f-4470-8dce-08694bf4a621-istio-token\") on node \"ip-10-0-142-134.ec2.internal\" DevicePath \"\"" Jul 07 19:26:47.139516 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:47.139450 2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50576a4b-363f-4470-8dce-08694bf4a621-istiod-ca-cert\") on node \"ip-10-0-142-134.ec2.internal\" DevicePath \"\"" Jul 07 19:26:47.139516 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:47.139457 2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50576a4b-363f-4470-8dce-08694bf4a621-workload-certs\") on node \"ip-10-0-142-134.ec2.internal\" DevicePath \"\"" Jul 07 19:26:47.486114 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:47.486073 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="50576a4b-363f-4470-8dce-08694bf4a621" path="/var/lib/kubelet/pods/50576a4b-363f-4470-8dce-08694bf4a621/volumes" Jul 07 19:26:48.715964 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:48.715918 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:26:53.906130 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:53.906097 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:26:53.980683 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:53.980654 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:26:54.094907 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:54.094875 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-9f830d7b-openshift-default-57db7779f-hvpcg"] Jul 07 19:26:54.095466 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:54.095448 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="50576a4b-363f-4470-8dce-08694bf4a621" containerName="istio-proxy" Jul 07 19:26:54.095530 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:54.095469 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="50576a4b-363f-4470-8dce-08694bf4a621" containerName="istio-proxy" Jul 07 19:26:54.095566 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:54.095534 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="50576a4b-363f-4470-8dce-08694bf4a621" containerName="istio-proxy" Jul 07 19:26:54.098791 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:54.098772 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-9f830d7b-openshift-default-57db7779f-hvpcg" Jul 07 19:26:54.101066 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:54.101044 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-9f830d7b-openshift-default-dockercfg-9x8px\"" Jul 07 19:26:54.110807 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:54.110784 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-9f830d7b-openshift-default-57db7779f-hvpcg"] Jul 07 19:26:54.136924 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:54.136874 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:26:54.205069 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:54.204975 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b0b58df-bb5b-4805-b31f-836db5d19dc8-credential-socket\") pod \"e2e-derive-9f830d7b-openshift-default-57db7779f-hvpcg\" (UID: \"2b0b58df-bb5b-4805-b31f-836db5d19dc8\") " pod="openshift-ingress/e2e-derive-9f830d7b-openshift-default-57db7779f-hvpcg" Jul 07 19:26:54.205069 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:54.205027 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b0b58df-bb5b-4805-b31f-836db5d19dc8-istiod-ca-cert\") pod \"e2e-derive-9f830d7b-openshift-default-57db7779f-hvpcg\" (UID: \"2b0b58df-bb5b-4805-b31f-836db5d19dc8\") " pod="openshift-ingress/e2e-derive-9f830d7b-openshift-default-57db7779f-hvpcg" Jul 07 19:26:54.205271 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:54.205094 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b0b58df-bb5b-4805-b31f-836db5d19dc8-istio-data\") pod \"e2e-derive-9f830d7b-openshift-default-57db7779f-hvpcg\" (UID: \"2b0b58df-bb5b-4805-b31f-836db5d19dc8\") " pod="openshift-ingress/e2e-derive-9f830d7b-openshift-default-57db7779f-hvpcg" Jul 07 19:26:54.205271 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:54.205173 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b0b58df-bb5b-4805-b31f-836db5d19dc8-workload-socket\") pod \"e2e-derive-9f830d7b-openshift-default-57db7779f-hvpcg\" (UID: \"2b0b58df-bb5b-4805-b31f-836db5d19dc8\") " pod="openshift-ingress/e2e-derive-9f830d7b-openshift-default-57db7779f-hvpcg" Jul 07 19:26:54.205271 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:54.205222 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b0b58df-bb5b-4805-b31f-836db5d19dc8-istio-token\") pod \"e2e-derive-9f830d7b-openshift-default-57db7779f-hvpcg\" (UID: \"2b0b58df-bb5b-4805-b31f-836db5d19dc8\") " pod="openshift-ingress/e2e-derive-9f830d7b-openshift-default-57db7779f-hvpcg" Jul 07 19:26:54.205271 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:54.205264 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b0b58df-bb5b-4805-b31f-836db5d19dc8-istio-envoy\") pod \"e2e-derive-9f830d7b-openshift-default-57db7779f-hvpcg\" (UID: \"2b0b58df-bb5b-4805-b31f-836db5d19dc8\") " pod="openshift-ingress/e2e-derive-9f830d7b-openshift-default-57db7779f-hvpcg" Jul 07 19:26:54.205474 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:54.205278 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b0b58df-bb5b-4805-b31f-836db5d19dc8-istio-podinfo\") pod \"e2e-derive-9f830d7b-openshift-default-57db7779f-hvpcg\" (UID: \"2b0b58df-bb5b-4805-b31f-836db5d19dc8\") " pod="openshift-ingress/e2e-derive-9f830d7b-openshift-default-57db7779f-hvpcg" Jul 07 19:26:54.205474 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:54.205296 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gzlf9\" (UniqueName: \"kubernetes.io/projected/2b0b58df-bb5b-4805-b31f-836db5d19dc8-kube-api-access-gzlf9\") pod \"e2e-derive-9f830d7b-openshift-default-57db7779f-hvpcg\" (UID: \"2b0b58df-bb5b-4805-b31f-836db5d19dc8\") " pod="openshift-ingress/e2e-derive-9f830d7b-openshift-default-57db7779f-hvpcg" Jul 07 19:26:54.205474 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:54.205354 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b0b58df-bb5b-4805-b31f-836db5d19dc8-workload-certs\") pod \"e2e-derive-9f830d7b-openshift-default-57db7779f-hvpcg\" (UID: \"2b0b58df-bb5b-4805-b31f-836db5d19dc8\") " pod="openshift-ingress/e2e-derive-9f830d7b-openshift-default-57db7779f-hvpcg" Jul 07 19:26:54.237154 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:54.237123 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:26:54.306124 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:54.306088 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b0b58df-bb5b-4805-b31f-836db5d19dc8-workload-socket\") pod \"e2e-derive-9f830d7b-openshift-default-57db7779f-hvpcg\" (UID: \"2b0b58df-bb5b-4805-b31f-836db5d19dc8\") " pod="openshift-ingress/e2e-derive-9f830d7b-openshift-default-57db7779f-hvpcg" Jul 07 19:26:54.306308 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:54.306139 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b0b58df-bb5b-4805-b31f-836db5d19dc8-istio-token\") pod \"e2e-derive-9f830d7b-openshift-default-57db7779f-hvpcg\" (UID: \"2b0b58df-bb5b-4805-b31f-836db5d19dc8\") " pod="openshift-ingress/e2e-derive-9f830d7b-openshift-default-57db7779f-hvpcg" Jul 07 19:26:54.306308 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:54.306162 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b0b58df-bb5b-4805-b31f-836db5d19dc8-istio-envoy\") pod \"e2e-derive-9f830d7b-openshift-default-57db7779f-hvpcg\" (UID: \"2b0b58df-bb5b-4805-b31f-836db5d19dc8\") " pod="openshift-ingress/e2e-derive-9f830d7b-openshift-default-57db7779f-hvpcg" Jul 07 19:26:54.306308 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:54.306178 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b0b58df-bb5b-4805-b31f-836db5d19dc8-istio-podinfo\") pod \"e2e-derive-9f830d7b-openshift-default-57db7779f-hvpcg\" (UID: \"2b0b58df-bb5b-4805-b31f-836db5d19dc8\") " pod="openshift-ingress/e2e-derive-9f830d7b-openshift-default-57db7779f-hvpcg" Jul 07 19:26:54.306308 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:54.306195 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzlf9\" (UniqueName: \"kubernetes.io/projected/2b0b58df-bb5b-4805-b31f-836db5d19dc8-kube-api-access-gzlf9\") pod \"e2e-derive-9f830d7b-openshift-default-57db7779f-hvpcg\" (UID: \"2b0b58df-bb5b-4805-b31f-836db5d19dc8\") " pod="openshift-ingress/e2e-derive-9f830d7b-openshift-default-57db7779f-hvpcg" Jul 07 19:26:54.306308 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:54.306230 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b0b58df-bb5b-4805-b31f-836db5d19dc8-workload-certs\") pod \"e2e-derive-9f830d7b-openshift-default-57db7779f-hvpcg\" (UID: \"2b0b58df-bb5b-4805-b31f-836db5d19dc8\") " pod="openshift-ingress/e2e-derive-9f830d7b-openshift-default-57db7779f-hvpcg" Jul 07 19:26:54.306308 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:54.306264 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b0b58df-bb5b-4805-b31f-836db5d19dc8-credential-socket\") pod \"e2e-derive-9f830d7b-openshift-default-57db7779f-hvpcg\" (UID: \"2b0b58df-bb5b-4805-b31f-836db5d19dc8\") " pod="openshift-ingress/e2e-derive-9f830d7b-openshift-default-57db7779f-hvpcg" Jul 07 19:26:54.306634 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:54.306322 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b0b58df-bb5b-4805-b31f-836db5d19dc8-istiod-ca-cert\") pod \"e2e-derive-9f830d7b-openshift-default-57db7779f-hvpcg\" (UID: \"2b0b58df-bb5b-4805-b31f-836db5d19dc8\") " pod="openshift-ingress/e2e-derive-9f830d7b-openshift-default-57db7779f-hvpcg" Jul 07 19:26:54.306634 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:54.306413 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b0b58df-bb5b-4805-b31f-836db5d19dc8-istio-data\") pod \"e2e-derive-9f830d7b-openshift-default-57db7779f-hvpcg\" (UID: \"2b0b58df-bb5b-4805-b31f-836db5d19dc8\") " pod="openshift-ingress/e2e-derive-9f830d7b-openshift-default-57db7779f-hvpcg" Jul 07 19:26:54.306634 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:54.306565 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b0b58df-bb5b-4805-b31f-836db5d19dc8-workload-socket\") pod \"e2e-derive-9f830d7b-openshift-default-57db7779f-hvpcg\" (UID: \"2b0b58df-bb5b-4805-b31f-836db5d19dc8\") " pod="openshift-ingress/e2e-derive-9f830d7b-openshift-default-57db7779f-hvpcg" Jul 07 19:26:54.306833 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:54.306809 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b0b58df-bb5b-4805-b31f-836db5d19dc8-istio-data\") pod \"e2e-derive-9f830d7b-openshift-default-57db7779f-hvpcg\" (UID: \"2b0b58df-bb5b-4805-b31f-836db5d19dc8\") " pod="openshift-ingress/e2e-derive-9f830d7b-openshift-default-57db7779f-hvpcg" Jul 07 19:26:54.306898 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:54.306847 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b0b58df-bb5b-4805-b31f-836db5d19dc8-workload-certs\") pod \"e2e-derive-9f830d7b-openshift-default-57db7779f-hvpcg\" (UID: \"2b0b58df-bb5b-4805-b31f-836db5d19dc8\") " pod="openshift-ingress/e2e-derive-9f830d7b-openshift-default-57db7779f-hvpcg" Jul 07 19:26:54.307031 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:54.307013 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b0b58df-bb5b-4805-b31f-836db5d19dc8-credential-socket\") pod \"e2e-derive-9f830d7b-openshift-default-57db7779f-hvpcg\" (UID: \"2b0b58df-bb5b-4805-b31f-836db5d19dc8\") " pod="openshift-ingress/e2e-derive-9f830d7b-openshift-default-57db7779f-hvpcg" Jul 07 19:26:54.307101 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:54.307081 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b0b58df-bb5b-4805-b31f-836db5d19dc8-istiod-ca-cert\") pod \"e2e-derive-9f830d7b-openshift-default-57db7779f-hvpcg\" (UID: \"2b0b58df-bb5b-4805-b31f-836db5d19dc8\") " pod="openshift-ingress/e2e-derive-9f830d7b-openshift-default-57db7779f-hvpcg" Jul 07 19:26:54.309201 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:54.309176 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b0b58df-bb5b-4805-b31f-836db5d19dc8-istio-envoy\") pod \"e2e-derive-9f830d7b-openshift-default-57db7779f-hvpcg\" (UID: \"2b0b58df-bb5b-4805-b31f-836db5d19dc8\") " pod="openshift-ingress/e2e-derive-9f830d7b-openshift-default-57db7779f-hvpcg" Jul 07 19:26:54.309295 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:54.309278 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b0b58df-bb5b-4805-b31f-836db5d19dc8-istio-podinfo\") pod \"e2e-derive-9f830d7b-openshift-default-57db7779f-hvpcg\" (UID: \"2b0b58df-bb5b-4805-b31f-836db5d19dc8\") " pod="openshift-ingress/e2e-derive-9f830d7b-openshift-default-57db7779f-hvpcg" Jul 07 19:26:54.315587 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:54.315563 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gzlf9\" (UniqueName: \"kubernetes.io/projected/2b0b58df-bb5b-4805-b31f-836db5d19dc8-kube-api-access-gzlf9\") pod \"e2e-derive-9f830d7b-openshift-default-57db7779f-hvpcg\" (UID: \"2b0b58df-bb5b-4805-b31f-836db5d19dc8\") " pod="openshift-ingress/e2e-derive-9f830d7b-openshift-default-57db7779f-hvpcg" Jul 07 19:26:54.315691 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:54.315564 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b0b58df-bb5b-4805-b31f-836db5d19dc8-istio-token\") pod \"e2e-derive-9f830d7b-openshift-default-57db7779f-hvpcg\" (UID: \"2b0b58df-bb5b-4805-b31f-836db5d19dc8\") " pod="openshift-ingress/e2e-derive-9f830d7b-openshift-default-57db7779f-hvpcg" Jul 07 19:26:54.412111 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:54.412071 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-9f830d7b-openshift-default-57db7779f-hvpcg" Jul 07 19:26:54.554028 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:54.554002 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-9f830d7b-openshift-default-57db7779f-hvpcg"] Jul 07 19:26:54.556647 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:26:54.556620 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b0b58df_bb5b_4805_b31f_836db5d19dc8.slice/crio-3fd3aa65b5db2147255c354a82efa74d27143c0756a41f3e0fb816297f822099 WatchSource:0}: Error finding container 3fd3aa65b5db2147255c354a82efa74d27143c0756a41f3e0fb816297f822099: Status 404 returned error can't find the container with id 3fd3aa65b5db2147255c354a82efa74d27143c0756a41f3e0fb816297f822099 Jul 07 19:26:54.559019 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:54.558983 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 07 19:26:54.559086 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:54.559055 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 07 19:26:54.559086 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:54.559082 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 07 19:26:55.134660 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:55.134618 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-9f830d7b-openshift-default-57db7779f-hvpcg" event={"ID":"2b0b58df-bb5b-4805-b31f-836db5d19dc8","Type":"ContainerStarted","Data":"2655127ffff8e2103c482d083cd98da9f669335b5c38d9c15b18f3dab803c3cf"} Jul 07 19:26:55.135068 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:55.134679 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-9f830d7b-openshift-default-57db7779f-hvpcg" event={"ID":"2b0b58df-bb5b-4805-b31f-836db5d19dc8","Type":"ContainerStarted","Data":"3fd3aa65b5db2147255c354a82efa74d27143c0756a41f3e0fb816297f822099"} Jul 07 19:26:55.158185 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:55.158133 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-9f830d7b-openshift-default-57db7779f-hvpcg" podStartSLOduration=1.158112568 podStartE2EDuration="1.158112568s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:26:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:26:55.156819262 +0000 UTC m=+1598.172823444" watchObservedRunningTime="2026-07-07 19:26:55.158112568 +0000 UTC m=+1598.174116799" Jul 07 19:26:55.328078 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:55.328035 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:26:55.413068 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:55.412980 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-9f830d7b-openshift-default-57db7779f-hvpcg" Jul 07 19:26:55.417842 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:55.417818 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-9f830d7b-openshift-default-57db7779f-hvpcg" Jul 07 19:26:55.447894 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:55.447862 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:26:55.537927 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:55.537897 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:26:56.057658 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:56.057620 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:26:56.139582 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:56.139553 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-9f830d7b-openshift-default-57db7779f-hvpcg" Jul 07 19:26:56.140631 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:56.140611 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-9f830d7b-openshift-default-57db7779f-hvpcg" Jul 07 19:26:56.157792 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:56.157767 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:26:56.260112 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:56.260067 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:26:57.212597 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:57.212567 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:26:59.723589 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:59.723556 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:26:59.882772 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:59.882714 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-9f830d7b-openshift-default-57db7779f-hvpcg"] Jul 07 19:26:59.883297 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:59.883259 2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-9f830d7b-openshift-default-57db7779f-hvpcg" podUID="2b0b58df-bb5b-4805-b31f-836db5d19dc8" containerName="istio-proxy" containerID="cri-o://2655127ffff8e2103c482d083cd98da9f669335b5c38d9c15b18f3dab803c3cf" gracePeriod=30 Jul 07 19:26:59.912785 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:26:59.912757 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:27:00.008854 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:27:00.008768 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:27:00.149413 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:27:00.149382 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:27:03.136834 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:27:03.136802 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:27:06.144497 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:27:06.144434 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-9f830d7b-openshift-default-57db7779f-hvpcg" Jul 07 19:27:06.199463 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:27:06.199423 2563 generic.go:356] "Generic (PLEG): container finished" podID="2b0b58df-bb5b-4805-b31f-836db5d19dc8" containerID="2655127ffff8e2103c482d083cd98da9f669335b5c38d9c15b18f3dab803c3cf" exitCode=0 Jul 07 19:27:06.199668 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:27:06.199482 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-9f830d7b-openshift-default-57db7779f-hvpcg" event={"ID":"2b0b58df-bb5b-4805-b31f-836db5d19dc8","Type":"ContainerDied","Data":"2655127ffff8e2103c482d083cd98da9f669335b5c38d9c15b18f3dab803c3cf"} Jul 07 19:27:06.199668 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:27:06.199539 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-9f830d7b-openshift-default-57db7779f-hvpcg" event={"ID":"2b0b58df-bb5b-4805-b31f-836db5d19dc8","Type":"ContainerDied","Data":"3fd3aa65b5db2147255c354a82efa74d27143c0756a41f3e0fb816297f822099"} Jul 07 19:27:06.199668 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:27:06.199494 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-9f830d7b-openshift-default-57db7779f-hvpcg" Jul 07 19:27:06.199668 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:27:06.199570 2563 scope.go:117] "RemoveContainer" containerID="2655127ffff8e2103c482d083cd98da9f669335b5c38d9c15b18f3dab803c3cf" Jul 07 19:27:06.208386 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:27:06.208366 2563 scope.go:117] "RemoveContainer" containerID="2655127ffff8e2103c482d083cd98da9f669335b5c38d9c15b18f3dab803c3cf" Jul 07 19:27:06.208652 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:27:06.208633 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2655127ffff8e2103c482d083cd98da9f669335b5c38d9c15b18f3dab803c3cf\": container with ID starting with 2655127ffff8e2103c482d083cd98da9f669335b5c38d9c15b18f3dab803c3cf not found: ID does not exist" containerID="2655127ffff8e2103c482d083cd98da9f669335b5c38d9c15b18f3dab803c3cf" Jul 07 19:27:06.208709 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:27:06.208666 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2655127ffff8e2103c482d083cd98da9f669335b5c38d9c15b18f3dab803c3cf"} err="failed to get container status \"2655127ffff8e2103c482d083cd98da9f669335b5c38d9c15b18f3dab803c3cf\": rpc error: code = NotFound desc = could not find container \"2655127ffff8e2103c482d083cd98da9f669335b5c38d9c15b18f3dab803c3cf\": container with ID starting with 2655127ffff8e2103c482d083cd98da9f669335b5c38d9c15b18f3dab803c3cf not found: ID does not exist" Jul 07 19:27:06.238235 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:27:06.238202 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b0b58df-bb5b-4805-b31f-836db5d19dc8-istiod-ca-cert\") pod \"2b0b58df-bb5b-4805-b31f-836db5d19dc8\" (UID: \"2b0b58df-bb5b-4805-b31f-836db5d19dc8\") " Jul 07 19:27:06.238408 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:27:06.238248 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b0b58df-bb5b-4805-b31f-836db5d19dc8-credential-socket\") pod \"2b0b58df-bb5b-4805-b31f-836db5d19dc8\" (UID: \"2b0b58df-bb5b-4805-b31f-836db5d19dc8\") " Jul 07 19:27:06.238408 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:27:06.238284 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b0b58df-bb5b-4805-b31f-836db5d19dc8-istio-data\") pod \"2b0b58df-bb5b-4805-b31f-836db5d19dc8\" (UID: \"2b0b58df-bb5b-4805-b31f-836db5d19dc8\") " Jul 07 19:27:06.238408 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:27:06.238332 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b0b58df-bb5b-4805-b31f-836db5d19dc8-workload-socket\") pod \"2b0b58df-bb5b-4805-b31f-836db5d19dc8\" (UID: \"2b0b58df-bb5b-4805-b31f-836db5d19dc8\") " Jul 07 19:27:06.238408 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:27:06.238363 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b0b58df-bb5b-4805-b31f-836db5d19dc8-istio-envoy\") pod \"2b0b58df-bb5b-4805-b31f-836db5d19dc8\" (UID: \"2b0b58df-bb5b-4805-b31f-836db5d19dc8\") " Jul 07 19:27:06.238625 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:27:06.238413 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b0b58df-bb5b-4805-b31f-836db5d19dc8-workload-certs\") pod \"2b0b58df-bb5b-4805-b31f-836db5d19dc8\" (UID: \"2b0b58df-bb5b-4805-b31f-836db5d19dc8\") " Jul 07 19:27:06.238625 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:27:06.238465 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gzlf9\" (UniqueName: \"kubernetes.io/projected/2b0b58df-bb5b-4805-b31f-836db5d19dc8-kube-api-access-gzlf9\") pod \"2b0b58df-bb5b-4805-b31f-836db5d19dc8\" (UID: \"2b0b58df-bb5b-4805-b31f-836db5d19dc8\") " Jul 07 19:27:06.238625 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:27:06.238581 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b0b58df-bb5b-4805-b31f-836db5d19dc8-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "2b0b58df-bb5b-4805-b31f-836db5d19dc8" (UID: "2b0b58df-bb5b-4805-b31f-836db5d19dc8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:27:06.238779 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:27:06.238690 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b0b58df-bb5b-4805-b31f-836db5d19dc8-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "2b0b58df-bb5b-4805-b31f-836db5d19dc8" (UID: "2b0b58df-bb5b-4805-b31f-836db5d19dc8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:27:06.238779 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:27:06.238721 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b0b58df-bb5b-4805-b31f-836db5d19dc8-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "2b0b58df-bb5b-4805-b31f-836db5d19dc8" (UID: "2b0b58df-bb5b-4805-b31f-836db5d19dc8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 19:27:06.238779 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:27:06.238721 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b0b58df-bb5b-4805-b31f-836db5d19dc8-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "2b0b58df-bb5b-4805-b31f-836db5d19dc8" (UID: "2b0b58df-bb5b-4805-b31f-836db5d19dc8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:27:06.238779 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:27:06.238729 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b0b58df-bb5b-4805-b31f-836db5d19dc8-istio-token\") pod \"2b0b58df-bb5b-4805-b31f-836db5d19dc8\" (UID: \"2b0b58df-bb5b-4805-b31f-836db5d19dc8\") " Jul 07 19:27:06.238989 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:27:06.238835 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b0b58df-bb5b-4805-b31f-836db5d19dc8-istio-podinfo\") pod \"2b0b58df-bb5b-4805-b31f-836db5d19dc8\" (UID: \"2b0b58df-bb5b-4805-b31f-836db5d19dc8\") " Jul 07 19:27:06.239221 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:27:06.239195 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b0b58df-bb5b-4805-b31f-836db5d19dc8-istio-data" (OuterVolumeSpecName: "istio-data") pod "2b0b58df-bb5b-4805-b31f-836db5d19dc8" (UID: "2b0b58df-bb5b-4805-b31f-836db5d19dc8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:27:06.239342 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:27:06.239219 2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b0b58df-bb5b-4805-b31f-836db5d19dc8-istiod-ca-cert\") on node \"ip-10-0-142-134.ec2.internal\" DevicePath \"\"" Jul 07 19:27:06.239342 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:27:06.239240 2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b0b58df-bb5b-4805-b31f-836db5d19dc8-credential-socket\") on node \"ip-10-0-142-134.ec2.internal\" DevicePath \"\"" Jul 07 19:27:06.239342 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:27:06.239256 2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b0b58df-bb5b-4805-b31f-836db5d19dc8-workload-socket\") on node \"ip-10-0-142-134.ec2.internal\" DevicePath \"\"" Jul 07 19:27:06.239342 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:27:06.239270 2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b0b58df-bb5b-4805-b31f-836db5d19dc8-workload-certs\") on node \"ip-10-0-142-134.ec2.internal\" DevicePath \"\"" Jul 07 19:27:06.240719 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:27:06.240689 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b0b58df-bb5b-4805-b31f-836db5d19dc8-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "2b0b58df-bb5b-4805-b31f-836db5d19dc8" (UID: "2b0b58df-bb5b-4805-b31f-836db5d19dc8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:27:06.240812 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:27:06.240743 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b0b58df-bb5b-4805-b31f-836db5d19dc8-istio-token" (OuterVolumeSpecName: "istio-token") pod "2b0b58df-bb5b-4805-b31f-836db5d19dc8" (UID: "2b0b58df-bb5b-4805-b31f-836db5d19dc8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 19:27:06.240812 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:27:06.240777 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b0b58df-bb5b-4805-b31f-836db5d19dc8-kube-api-access-gzlf9" (OuterVolumeSpecName: "kube-api-access-gzlf9") pod "2b0b58df-bb5b-4805-b31f-836db5d19dc8" (UID: "2b0b58df-bb5b-4805-b31f-836db5d19dc8"). InnerVolumeSpecName "kube-api-access-gzlf9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 19:27:06.240942 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:27:06.240929 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2b0b58df-bb5b-4805-b31f-836db5d19dc8-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "2b0b58df-bb5b-4805-b31f-836db5d19dc8" (UID: "2b0b58df-bb5b-4805-b31f-836db5d19dc8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 07 19:27:06.340708 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:27:06.340667 2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b0b58df-bb5b-4805-b31f-836db5d19dc8-istio-data\") on node \"ip-10-0-142-134.ec2.internal\" DevicePath \"\"" Jul 07 19:27:06.340708 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:27:06.340698 2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b0b58df-bb5b-4805-b31f-836db5d19dc8-istio-envoy\") on node \"ip-10-0-142-134.ec2.internal\" DevicePath \"\"" Jul 07 19:27:06.340708 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:27:06.340708 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gzlf9\" (UniqueName: \"kubernetes.io/projected/2b0b58df-bb5b-4805-b31f-836db5d19dc8-kube-api-access-gzlf9\") on node \"ip-10-0-142-134.ec2.internal\" DevicePath \"\"" Jul 07 19:27:06.340708 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:27:06.340717 2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b0b58df-bb5b-4805-b31f-836db5d19dc8-istio-token\") on node \"ip-10-0-142-134.ec2.internal\" DevicePath \"\"" Jul 07 19:27:06.341049 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:27:06.340725 2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b0b58df-bb5b-4805-b31f-836db5d19dc8-istio-podinfo\") on node \"ip-10-0-142-134.ec2.internal\" DevicePath \"\"" Jul 07 19:27:06.523969 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:27:06.523910 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-9f830d7b-openshift-default-57db7779f-hvpcg"] Jul 07 19:27:06.528621 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:27:06.528590 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-9f830d7b-openshift-default-57db7779f-hvpcg"] Jul 07 19:27:06.771552 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:27:06.771468 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:27:06.872373 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:27:06.872341 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:27:07.072652 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:27:07.072624 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:27:07.262290 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:27:07.262260 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:27:07.375429 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:27:07.375349 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:27:07.462113 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:27:07.462079 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:27:07.476497 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:27:07.476464 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2b0b58df-bb5b-4805-b31f-836db5d19dc8" path="/var/lib/kubelet/pods/2b0b58df-bb5b-4805-b31f-836db5d19dc8/volumes" Jul 07 19:27:08.875244 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:27:08.875214 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:27:15.860855 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:27:15.860767 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:27:17.001108 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:27:17.001067 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:27:24.290502 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:27:24.290469 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:27:24.392925 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:27:24.392893 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:27:42.602324 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:27:42.602285 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:27:49.647727 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:27:49.647697 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:27:49.752436 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:27:49.752405 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:27:49.849539 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:27:49.849503 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:28:05.792830 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:28:05.792795 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:28:12.590395 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:28:12.590357 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:28:12.693917 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:28:12.693880 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:28:12.793160 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:28:12.793124 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:28:23.347131 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:28:23.347097 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:28:24.144758 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:28:24.144710 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:28:30.905844 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:28:30.905807 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:28:31.018003 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:28:31.017970 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:28:31.208001 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:28:31.207900 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:28:31.296372 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:28:31.296309 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:28:32.042013 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:28:32.041980 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:28:32.143147 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:28:32.143111 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:28:32.246282 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:28:32.246247 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:28:33.832571 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:28:33.832540 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:28:40.838632 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:28:40.838558 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:28:40.936748 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:28:40.936696 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:28:41.565328 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:28:41.565289 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:28:48.765720 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:28:48.765678 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:28:48.866735 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:28:48.866700 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:28:49.000971 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:28:49.000924 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:28:49.293314 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:28:49.293283 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-85ca1257-openshift-default-68d7d85d75-khqz8"] Jul 07 19:28:49.293699 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:28:49.293687 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2b0b58df-bb5b-4805-b31f-836db5d19dc8" containerName="istio-proxy" Jul 07 19:28:49.293743 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:28:49.293700 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b0b58df-bb5b-4805-b31f-836db5d19dc8" containerName="istio-proxy" Jul 07 19:28:49.293778 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:28:49.293762 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="2b0b58df-bb5b-4805-b31f-836db5d19dc8" containerName="istio-proxy" Jul 07 19:28:49.296823 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:28:49.296806 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-85ca1257-openshift-default-68d7d85d75-khqz8" Jul 07 19:28:49.298878 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:28:49.298854 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-85ca1257-openshift-default-dockercfg-xnfmf\"" Jul 07 19:28:49.318900 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:28:49.318863 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-85ca1257-openshift-default-68d7d85d75-khqz8"] Jul 07 19:28:49.325203 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:28:49.325171 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:28:49.422946 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:28:49.422907 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8356506-39e1-405a-98fd-c49c8c498764-credential-socket\") pod \"e2e-mt-85ca1257-openshift-default-68d7d85d75-khqz8\" (UID: \"c8356506-39e1-405a-98fd-c49c8c498764\") " pod="openshift-ingress/e2e-mt-85ca1257-openshift-default-68d7d85d75-khqz8" Jul 07 19:28:49.422946 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:28:49.422976 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5nlf8\" (UniqueName: \"kubernetes.io/projected/c8356506-39e1-405a-98fd-c49c8c498764-kube-api-access-5nlf8\") pod \"e2e-mt-85ca1257-openshift-default-68d7d85d75-khqz8\" (UID: \"c8356506-39e1-405a-98fd-c49c8c498764\") " pod="openshift-ingress/e2e-mt-85ca1257-openshift-default-68d7d85d75-khqz8" Jul 07 19:28:49.423215 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:28:49.423001 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8356506-39e1-405a-98fd-c49c8c498764-istio-token\") pod \"e2e-mt-85ca1257-openshift-default-68d7d85d75-khqz8\" (UID: \"c8356506-39e1-405a-98fd-c49c8c498764\") " pod="openshift-ingress/e2e-mt-85ca1257-openshift-default-68d7d85d75-khqz8" Jul 07 19:28:49.423215 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:28:49.423026 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8356506-39e1-405a-98fd-c49c8c498764-istio-envoy\") pod \"e2e-mt-85ca1257-openshift-default-68d7d85d75-khqz8\" (UID: \"c8356506-39e1-405a-98fd-c49c8c498764\") " pod="openshift-ingress/e2e-mt-85ca1257-openshift-default-68d7d85d75-khqz8" Jul 07 19:28:49.423215 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:28:49.423080 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8356506-39e1-405a-98fd-c49c8c498764-workload-socket\") pod \"e2e-mt-85ca1257-openshift-default-68d7d85d75-khqz8\" (UID: \"c8356506-39e1-405a-98fd-c49c8c498764\") " pod="openshift-ingress/e2e-mt-85ca1257-openshift-default-68d7d85d75-khqz8" Jul 07 19:28:49.423215 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:28:49.423111 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8356506-39e1-405a-98fd-c49c8c498764-istiod-ca-cert\") pod \"e2e-mt-85ca1257-openshift-default-68d7d85d75-khqz8\" (UID: \"c8356506-39e1-405a-98fd-c49c8c498764\") " pod="openshift-ingress/e2e-mt-85ca1257-openshift-default-68d7d85d75-khqz8" Jul 07 19:28:49.423215 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:28:49.423139 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8356506-39e1-405a-98fd-c49c8c498764-workload-certs\") pod \"e2e-mt-85ca1257-openshift-default-68d7d85d75-khqz8\" (UID: \"c8356506-39e1-405a-98fd-c49c8c498764\") " pod="openshift-ingress/e2e-mt-85ca1257-openshift-default-68d7d85d75-khqz8" Jul 07 19:28:49.423384 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:28:49.423216 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8356506-39e1-405a-98fd-c49c8c498764-istio-data\") pod \"e2e-mt-85ca1257-openshift-default-68d7d85d75-khqz8\" (UID: \"c8356506-39e1-405a-98fd-c49c8c498764\") " pod="openshift-ingress/e2e-mt-85ca1257-openshift-default-68d7d85d75-khqz8" Jul 07 19:28:49.423384 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:28:49.423261 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8356506-39e1-405a-98fd-c49c8c498764-istio-podinfo\") pod \"e2e-mt-85ca1257-openshift-default-68d7d85d75-khqz8\" (UID: \"c8356506-39e1-405a-98fd-c49c8c498764\") " pod="openshift-ingress/e2e-mt-85ca1257-openshift-default-68d7d85d75-khqz8" Jul 07 19:28:49.524336 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:28:49.524291 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8356506-39e1-405a-98fd-c49c8c498764-credential-socket\") pod \"e2e-mt-85ca1257-openshift-default-68d7d85d75-khqz8\" (UID: \"c8356506-39e1-405a-98fd-c49c8c498764\") " pod="openshift-ingress/e2e-mt-85ca1257-openshift-default-68d7d85d75-khqz8" Jul 07 19:28:49.524336 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:28:49.524334 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5nlf8\" (UniqueName: \"kubernetes.io/projected/c8356506-39e1-405a-98fd-c49c8c498764-kube-api-access-5nlf8\") pod \"e2e-mt-85ca1257-openshift-default-68d7d85d75-khqz8\" (UID: \"c8356506-39e1-405a-98fd-c49c8c498764\") " pod="openshift-ingress/e2e-mt-85ca1257-openshift-default-68d7d85d75-khqz8" Jul 07 19:28:49.524569 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:28:49.524355 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8356506-39e1-405a-98fd-c49c8c498764-istio-token\") pod \"e2e-mt-85ca1257-openshift-default-68d7d85d75-khqz8\" (UID: \"c8356506-39e1-405a-98fd-c49c8c498764\") " pod="openshift-ingress/e2e-mt-85ca1257-openshift-default-68d7d85d75-khqz8" Jul 07 19:28:49.524569 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:28:49.524554 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8356506-39e1-405a-98fd-c49c8c498764-istio-envoy\") pod \"e2e-mt-85ca1257-openshift-default-68d7d85d75-khqz8\" (UID: \"c8356506-39e1-405a-98fd-c49c8c498764\") " pod="openshift-ingress/e2e-mt-85ca1257-openshift-default-68d7d85d75-khqz8" Jul 07 19:28:49.524642 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:28:49.524631 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8356506-39e1-405a-98fd-c49c8c498764-workload-socket\") pod \"e2e-mt-85ca1257-openshift-default-68d7d85d75-khqz8\" (UID: \"c8356506-39e1-405a-98fd-c49c8c498764\") " pod="openshift-ingress/e2e-mt-85ca1257-openshift-default-68d7d85d75-khqz8" Jul 07 19:28:49.524693 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:28:49.524679 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8356506-39e1-405a-98fd-c49c8c498764-istiod-ca-cert\") pod \"e2e-mt-85ca1257-openshift-default-68d7d85d75-khqz8\" (UID: \"c8356506-39e1-405a-98fd-c49c8c498764\") " pod="openshift-ingress/e2e-mt-85ca1257-openshift-default-68d7d85d75-khqz8" Jul 07 19:28:49.524734 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:28:49.524716 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8356506-39e1-405a-98fd-c49c8c498764-workload-certs\") pod \"e2e-mt-85ca1257-openshift-default-68d7d85d75-khqz8\" (UID: \"c8356506-39e1-405a-98fd-c49c8c498764\") " pod="openshift-ingress/e2e-mt-85ca1257-openshift-default-68d7d85d75-khqz8" Jul 07 19:28:49.524786 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:28:49.524752 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8356506-39e1-405a-98fd-c49c8c498764-istio-data\") pod \"e2e-mt-85ca1257-openshift-default-68d7d85d75-khqz8\" (UID: \"c8356506-39e1-405a-98fd-c49c8c498764\") " pod="openshift-ingress/e2e-mt-85ca1257-openshift-default-68d7d85d75-khqz8" Jul 07 19:28:49.524786 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:28:49.524778 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8356506-39e1-405a-98fd-c49c8c498764-istio-podinfo\") pod \"e2e-mt-85ca1257-openshift-default-68d7d85d75-khqz8\" (UID: \"c8356506-39e1-405a-98fd-c49c8c498764\") " pod="openshift-ingress/e2e-mt-85ca1257-openshift-default-68d7d85d75-khqz8" Jul 07 19:28:49.524888 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:28:49.524842 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8356506-39e1-405a-98fd-c49c8c498764-credential-socket\") pod \"e2e-mt-85ca1257-openshift-default-68d7d85d75-khqz8\" (UID: \"c8356506-39e1-405a-98fd-c49c8c498764\") " pod="openshift-ingress/e2e-mt-85ca1257-openshift-default-68d7d85d75-khqz8" Jul 07 19:28:49.524888 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:28:49.524862 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8356506-39e1-405a-98fd-c49c8c498764-workload-socket\") pod \"e2e-mt-85ca1257-openshift-default-68d7d85d75-khqz8\" (UID: \"c8356506-39e1-405a-98fd-c49c8c498764\") " pod="openshift-ingress/e2e-mt-85ca1257-openshift-default-68d7d85d75-khqz8" Jul 07 19:28:49.525078 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:28:49.525059 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8356506-39e1-405a-98fd-c49c8c498764-workload-certs\") pod \"e2e-mt-85ca1257-openshift-default-68d7d85d75-khqz8\" (UID: \"c8356506-39e1-405a-98fd-c49c8c498764\") " pod="openshift-ingress/e2e-mt-85ca1257-openshift-default-68d7d85d75-khqz8" Jul 07 19:28:49.525212 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:28:49.525189 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8356506-39e1-405a-98fd-c49c8c498764-istio-data\") pod \"e2e-mt-85ca1257-openshift-default-68d7d85d75-khqz8\" (UID: \"c8356506-39e1-405a-98fd-c49c8c498764\") " pod="openshift-ingress/e2e-mt-85ca1257-openshift-default-68d7d85d75-khqz8" Jul 07 19:28:49.525455 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:28:49.525433 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8356506-39e1-405a-98fd-c49c8c498764-istiod-ca-cert\") pod \"e2e-mt-85ca1257-openshift-default-68d7d85d75-khqz8\" (UID: \"c8356506-39e1-405a-98fd-c49c8c498764\") " pod="openshift-ingress/e2e-mt-85ca1257-openshift-default-68d7d85d75-khqz8" Jul 07 19:28:49.526824 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:28:49.526799 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8356506-39e1-405a-98fd-c49c8c498764-istio-envoy\") pod \"e2e-mt-85ca1257-openshift-default-68d7d85d75-khqz8\" (UID: \"c8356506-39e1-405a-98fd-c49c8c498764\") " pod="openshift-ingress/e2e-mt-85ca1257-openshift-default-68d7d85d75-khqz8" Jul 07 19:28:49.527177 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:28:49.527161 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8356506-39e1-405a-98fd-c49c8c498764-istio-podinfo\") pod \"e2e-mt-85ca1257-openshift-default-68d7d85d75-khqz8\" (UID: \"c8356506-39e1-405a-98fd-c49c8c498764\") " pod="openshift-ingress/e2e-mt-85ca1257-openshift-default-68d7d85d75-khqz8" Jul 07 19:28:49.533240 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:28:49.533211 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8356506-39e1-405a-98fd-c49c8c498764-istio-token\") pod \"e2e-mt-85ca1257-openshift-default-68d7d85d75-khqz8\" (UID: \"c8356506-39e1-405a-98fd-c49c8c498764\") " pod="openshift-ingress/e2e-mt-85ca1257-openshift-default-68d7d85d75-khqz8" Jul 07 19:28:49.533373 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:28:49.533271 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5nlf8\" (UniqueName: \"kubernetes.io/projected/c8356506-39e1-405a-98fd-c49c8c498764-kube-api-access-5nlf8\") pod \"e2e-mt-85ca1257-openshift-default-68d7d85d75-khqz8\" (UID: \"c8356506-39e1-405a-98fd-c49c8c498764\") " pod="openshift-ingress/e2e-mt-85ca1257-openshift-default-68d7d85d75-khqz8" Jul 07 19:28:49.606980 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:28:49.606854 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-85ca1257-openshift-default-68d7d85d75-khqz8" Jul 07 19:28:49.668451 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:28:49.668415 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:28:49.761426 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:28:49.761400 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-85ca1257-openshift-default-68d7d85d75-khqz8"] Jul 07 19:28:49.764124 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:28:49.764098 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8356506_39e1_405a_98fd_c49c8c498764.slice/crio-301e84c46245d3e12386e478c6532f12c5551d74f97facfd1a97f51d07a58e9b WatchSource:0}: Error finding container 301e84c46245d3e12386e478c6532f12c5551d74f97facfd1a97f51d07a58e9b: Status 404 returned error can't find the container with id 301e84c46245d3e12386e478c6532f12c5551d74f97facfd1a97f51d07a58e9b Jul 07 19:28:49.766291 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:28:49.766263 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 07 19:28:49.766569 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:28:49.766345 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 07 19:28:49.766569 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:28:49.766372 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 07 19:28:50.709247 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:28:50.709208 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-85ca1257-openshift-default-68d7d85d75-khqz8" event={"ID":"c8356506-39e1-405a-98fd-c49c8c498764","Type":"ContainerStarted","Data":"fb78a7f8f53c7d9533a26effbbeaebb52da77a7d2e6220e3dec1b6b9da4fd029"} Jul 07 19:28:50.709480 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:28:50.709277 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-85ca1257-openshift-default-68d7d85d75-khqz8" event={"ID":"c8356506-39e1-405a-98fd-c49c8c498764","Type":"ContainerStarted","Data":"301e84c46245d3e12386e478c6532f12c5551d74f97facfd1a97f51d07a58e9b"} Jul 07 19:28:50.736260 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:28:50.736195 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-85ca1257-openshift-default-68d7d85d75-khqz8" podStartSLOduration=1.736175259 podStartE2EDuration="1.736175259s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:28:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:28:50.735887133 +0000 UTC m=+1713.751891317" watchObservedRunningTime="2026-07-07 19:28:50.736175259 +0000 UTC m=+1713.752179442" Jul 07 19:28:51.607363 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:28:51.607324 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-85ca1257-openshift-default-68d7d85d75-khqz8" Jul 07 19:28:51.612011 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:28:51.611984 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-85ca1257-openshift-default-68d7d85d75-khqz8" Jul 07 19:28:51.714598 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:28:51.714568 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-85ca1257-openshift-default-68d7d85d75-khqz8" Jul 07 19:28:51.715523 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:28:51.715499 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-85ca1257-openshift-default-68d7d85d75-khqz8" Jul 07 19:28:52.715853 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:28:52.715817 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:28:52.818315 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:28:52.818283 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:28:52.919866 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:28:52.919832 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:28:53.061186 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:28:53.061144 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:28:53.627464 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:28:53.627434 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:28:53.726903 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:28:53.726874 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:28:57.220061 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:28:57.220020 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:28:57.888729 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:28:57.888699 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:28:57.970195 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:28:57.970164 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:28:58.328225 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:28:58.328193 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:29:05.489857 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:29:05.489816 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:29:05.703686 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:29:05.703642 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:29:05.925700 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:29:05.925665 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:29:06.291629 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:29:06.291598 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:29:06.421070 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:29:06.421040 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:29:07.273235 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:29:07.272314 2563 patch_prober.go:28] interesting pod/e2e-mt-85ca1257-openshift-default-68d7d85d75-khqz8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.41:15021/healthz/ready\": read tcp 10.132.0.2:47828->10.132.0.41:15021: read: connection reset by peer" start-of-body= Jul 07 19:29:07.273235 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:29:07.272383 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-85ca1257-openshift-default-68d7d85d75-khqz8" podUID="c8356506-39e1-405a-98fd-c49c8c498764" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.41:15021/healthz/ready\": read tcp 10.132.0.2:47828->10.132.0.41:15021: read: connection reset by peer" Jul 07 19:29:07.804818 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:29:07.804785 2563 generic.go:356] "Generic (PLEG): container finished" podID="c8356506-39e1-405a-98fd-c49c8c498764" containerID="fb78a7f8f53c7d9533a26effbbeaebb52da77a7d2e6220e3dec1b6b9da4fd029" exitCode=0 Jul 07 19:29:07.805024 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:29:07.804859 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-85ca1257-openshift-default-68d7d85d75-khqz8" event={"ID":"c8356506-39e1-405a-98fd-c49c8c498764","Type":"ContainerDied","Data":"fb78a7f8f53c7d9533a26effbbeaebb52da77a7d2e6220e3dec1b6b9da4fd029"} Jul 07 19:29:07.805267 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:29:07.805253 2563 scope.go:117] "RemoveContainer" containerID="fb78a7f8f53c7d9533a26effbbeaebb52da77a7d2e6220e3dec1b6b9da4fd029" Jul 07 19:29:07.806192 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:29:07.806165 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 07 19:29:07.806295 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:29:07.806236 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 07 19:29:07.806295 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:29:07.806269 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 07 19:29:08.811681 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:29:08.811647 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-85ca1257-openshift-default-68d7d85d75-khqz8" event={"ID":"c8356506-39e1-405a-98fd-c49c8c498764","Type":"ContainerStarted","Data":"16301edc4e62487354bab19d13bd3798bfa6494def8042cc3f02e335c44e6cc0"} Jul 07 19:29:09.607542 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:29:09.607508 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-85ca1257-openshift-default-68d7d85d75-khqz8" Jul 07 19:29:09.607737 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:29:09.607719 2563 patch_prober.go:28] interesting pod/e2e-mt-85ca1257-openshift-default-68d7d85d75-khqz8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" start-of-body= Jul 07 19:29:09.607794 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:29:09.607775 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-85ca1257-openshift-default-68d7d85d75-khqz8" podUID="c8356506-39e1-405a-98fd-c49c8c498764" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" Jul 07 19:29:10.608232 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:29:10.608197 2563 patch_prober.go:28] interesting pod/e2e-mt-85ca1257-openshift-default-68d7d85d75-khqz8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" start-of-body= Jul 07 19:29:10.608743 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:29:10.608272 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-85ca1257-openshift-default-68d7d85d75-khqz8" podUID="c8356506-39e1-405a-98fd-c49c8c498764" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" Jul 07 19:29:11.608129 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:29:11.608096 2563 patch_prober.go:28] interesting pod/e2e-mt-85ca1257-openshift-default-68d7d85d75-khqz8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" start-of-body= Jul 07 19:29:11.608356 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:29:11.608150 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-85ca1257-openshift-default-68d7d85d75-khqz8" podUID="c8356506-39e1-405a-98fd-c49c8c498764" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" Jul 07 19:29:12.608005 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:29:12.607938 2563 patch_prober.go:28] interesting pod/e2e-mt-85ca1257-openshift-default-68d7d85d75-khqz8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" start-of-body= Jul 07 19:29:12.608203 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:29:12.608045 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-85ca1257-openshift-default-68d7d85d75-khqz8" podUID="c8356506-39e1-405a-98fd-c49c8c498764" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" Jul 07 19:29:13.385519 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:29:13.385482 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:29:13.486391 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:29:13.486352 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:29:13.614022 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:29:13.613810 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-85ca1257-openshift-default-68d7d85d75-khqz8" Jul 07 19:29:13.614654 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:29:13.614628 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-85ca1257-openshift-default-68d7d85d75-khqz8" Jul 07 19:29:13.615503 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:29:13.615481 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-85ca1257-openshift-default-68d7d85d75-khqz8" Jul 07 19:29:13.651988 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:29:13.651882 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-85ca1257-openshift-default-68d7d85d75-khqz8"] Jul 07 19:29:13.666867 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:29:13.666838 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:29:13.837109 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:29:13.837079 2563 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/e2e-mt-85ca1257-openshift-default-68d7d85d75-khqz8" secret="" err="secret \"e2e-mt-85ca1257-openshift-default-dockercfg-xnfmf\" not found" Jul 07 19:29:13.886848 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:29:13.886810 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:29:14.026531 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:29:14.026446 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:29:14.122175 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:29:14.122144 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:29:14.222498 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:29:14.222461 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:29:14.843282 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:29:14.843193 2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-85ca1257-openshift-default-68d7d85d75-khqz8" podUID="c8356506-39e1-405a-98fd-c49c8c498764" containerName="istio-proxy" containerID="cri-o://16301edc4e62487354bab19d13bd3798bfa6494def8042cc3f02e335c44e6cc0" gracePeriod=30 Jul 07 19:29:20.102090 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:29:20.102064 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-85ca1257-openshift-default-68d7d85d75-khqz8" Jul 07 19:29:20.225116 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:29:20.225020 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8356506-39e1-405a-98fd-c49c8c498764-istio-token\") pod \"c8356506-39e1-405a-98fd-c49c8c498764\" (UID: \"c8356506-39e1-405a-98fd-c49c8c498764\") " Jul 07 19:29:20.225116 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:29:20.225091 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5nlf8\" (UniqueName: \"kubernetes.io/projected/c8356506-39e1-405a-98fd-c49c8c498764-kube-api-access-5nlf8\") pod \"c8356506-39e1-405a-98fd-c49c8c498764\" (UID: \"c8356506-39e1-405a-98fd-c49c8c498764\") " Jul 07 19:29:20.225311 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:29:20.225126 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8356506-39e1-405a-98fd-c49c8c498764-istio-envoy\") pod \"c8356506-39e1-405a-98fd-c49c8c498764\" (UID: \"c8356506-39e1-405a-98fd-c49c8c498764\") " Jul 07 19:29:20.225311 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:29:20.225153 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8356506-39e1-405a-98fd-c49c8c498764-workload-socket\") pod \"c8356506-39e1-405a-98fd-c49c8c498764\" (UID: \"c8356506-39e1-405a-98fd-c49c8c498764\") " Jul 07 19:29:20.225311 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:29:20.225208 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8356506-39e1-405a-98fd-c49c8c498764-credential-socket\") pod \"c8356506-39e1-405a-98fd-c49c8c498764\" (UID: \"c8356506-39e1-405a-98fd-c49c8c498764\") " Jul 07 19:29:20.225311 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:29:20.225245 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8356506-39e1-405a-98fd-c49c8c498764-istio-data\") pod \"c8356506-39e1-405a-98fd-c49c8c498764\" (UID: \"c8356506-39e1-405a-98fd-c49c8c498764\") " Jul 07 19:29:20.225311 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:29:20.225263 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8356506-39e1-405a-98fd-c49c8c498764-workload-certs\") pod \"c8356506-39e1-405a-98fd-c49c8c498764\" (UID: \"c8356506-39e1-405a-98fd-c49c8c498764\") " Jul 07 19:29:20.225311 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:29:20.225282 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8356506-39e1-405a-98fd-c49c8c498764-istiod-ca-cert\") pod \"c8356506-39e1-405a-98fd-c49c8c498764\" (UID: \"c8356506-39e1-405a-98fd-c49c8c498764\") " Jul 07 19:29:20.225311 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:29:20.225303 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8356506-39e1-405a-98fd-c49c8c498764-istio-podinfo\") pod \"c8356506-39e1-405a-98fd-c49c8c498764\" (UID: \"c8356506-39e1-405a-98fd-c49c8c498764\") " Jul 07 19:29:20.225672 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:29:20.225504 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8356506-39e1-405a-98fd-c49c8c498764-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "c8356506-39e1-405a-98fd-c49c8c498764" (UID: "c8356506-39e1-405a-98fd-c49c8c498764"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:29:20.225672 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:29:20.225531 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8356506-39e1-405a-98fd-c49c8c498764-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "c8356506-39e1-405a-98fd-c49c8c498764" (UID: "c8356506-39e1-405a-98fd-c49c8c498764"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:29:20.225791 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:29:20.225709 2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8356506-39e1-405a-98fd-c49c8c498764-workload-socket\") on node \"ip-10-0-142-134.ec2.internal\" DevicePath \"\"" Jul 07 19:29:20.225791 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:29:20.225729 2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8356506-39e1-405a-98fd-c49c8c498764-credential-socket\") on node \"ip-10-0-142-134.ec2.internal\" DevicePath \"\"" Jul 07 19:29:20.225896 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:29:20.225848 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8356506-39e1-405a-98fd-c49c8c498764-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "c8356506-39e1-405a-98fd-c49c8c498764" (UID: "c8356506-39e1-405a-98fd-c49c8c498764"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 19:29:20.225979 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:29:20.225932 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8356506-39e1-405a-98fd-c49c8c498764-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "c8356506-39e1-405a-98fd-c49c8c498764" (UID: "c8356506-39e1-405a-98fd-c49c8c498764"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:29:20.226275 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:29:20.226244 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8356506-39e1-405a-98fd-c49c8c498764-istio-data" (OuterVolumeSpecName: "istio-data") pod "c8356506-39e1-405a-98fd-c49c8c498764" (UID: "c8356506-39e1-405a-98fd-c49c8c498764"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:29:20.227583 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:29:20.227558 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8356506-39e1-405a-98fd-c49c8c498764-kube-api-access-5nlf8" (OuterVolumeSpecName: "kube-api-access-5nlf8") pod "c8356506-39e1-405a-98fd-c49c8c498764" (UID: "c8356506-39e1-405a-98fd-c49c8c498764"). InnerVolumeSpecName "kube-api-access-5nlf8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 19:29:20.227666 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:29:20.227610 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c8356506-39e1-405a-98fd-c49c8c498764-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "c8356506-39e1-405a-98fd-c49c8c498764" (UID: "c8356506-39e1-405a-98fd-c49c8c498764"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 07 19:29:20.227838 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:29:20.227808 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8356506-39e1-405a-98fd-c49c8c498764-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "c8356506-39e1-405a-98fd-c49c8c498764" (UID: "c8356506-39e1-405a-98fd-c49c8c498764"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:29:20.227924 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:29:20.227814 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8356506-39e1-405a-98fd-c49c8c498764-istio-token" (OuterVolumeSpecName: "istio-token") pod "c8356506-39e1-405a-98fd-c49c8c498764" (UID: "c8356506-39e1-405a-98fd-c49c8c498764"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 19:29:20.327118 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:29:20.327086 2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8356506-39e1-405a-98fd-c49c8c498764-istio-envoy\") on node \"ip-10-0-142-134.ec2.internal\" DevicePath \"\"" Jul 07 19:29:20.327118 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:29:20.327111 2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8356506-39e1-405a-98fd-c49c8c498764-istio-data\") on node \"ip-10-0-142-134.ec2.internal\" DevicePath \"\"" Jul 07 19:29:20.327118 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:29:20.327120 2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8356506-39e1-405a-98fd-c49c8c498764-workload-certs\") on node \"ip-10-0-142-134.ec2.internal\" DevicePath \"\"" Jul 07 19:29:20.327118 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:29:20.327127 2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8356506-39e1-405a-98fd-c49c8c498764-istiod-ca-cert\") on node \"ip-10-0-142-134.ec2.internal\" DevicePath \"\"" Jul 07 19:29:20.327389 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:29:20.327135 2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8356506-39e1-405a-98fd-c49c8c498764-istio-podinfo\") on node \"ip-10-0-142-134.ec2.internal\" DevicePath \"\"" Jul 07 19:29:20.327389 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:29:20.327143 2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8356506-39e1-405a-98fd-c49c8c498764-istio-token\") on node \"ip-10-0-142-134.ec2.internal\" DevicePath \"\"" Jul 07 19:29:20.327389 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:29:20.327151 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5nlf8\" (UniqueName: \"kubernetes.io/projected/c8356506-39e1-405a-98fd-c49c8c498764-kube-api-access-5nlf8\") on node \"ip-10-0-142-134.ec2.internal\" DevicePath \"\"" Jul 07 19:29:20.634838 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:29:20.634803 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:29:20.737223 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:29:20.737192 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:29:20.851119 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:29:20.851086 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:29:20.875324 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:29:20.875295 2563 generic.go:356] "Generic (PLEG): container finished" podID="c8356506-39e1-405a-98fd-c49c8c498764" containerID="16301edc4e62487354bab19d13bd3798bfa6494def8042cc3f02e335c44e6cc0" exitCode=0 Jul 07 19:29:20.875488 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:29:20.875351 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-85ca1257-openshift-default-68d7d85d75-khqz8" Jul 07 19:29:20.875488 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:29:20.875364 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-85ca1257-openshift-default-68d7d85d75-khqz8" event={"ID":"c8356506-39e1-405a-98fd-c49c8c498764","Type":"ContainerDied","Data":"16301edc4e62487354bab19d13bd3798bfa6494def8042cc3f02e335c44e6cc0"} Jul 07 19:29:20.875488 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:29:20.875408 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-85ca1257-openshift-default-68d7d85d75-khqz8" event={"ID":"c8356506-39e1-405a-98fd-c49c8c498764","Type":"ContainerDied","Data":"301e84c46245d3e12386e478c6532f12c5551d74f97facfd1a97f51d07a58e9b"} Jul 07 19:29:20.875488 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:29:20.875431 2563 scope.go:117] "RemoveContainer" containerID="16301edc4e62487354bab19d13bd3798bfa6494def8042cc3f02e335c44e6cc0" Jul 07 19:29:20.885510 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:29:20.885490 2563 scope.go:117] "RemoveContainer" containerID="fb78a7f8f53c7d9533a26effbbeaebb52da77a7d2e6220e3dec1b6b9da4fd029" Jul 07 19:29:20.894114 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:29:20.894093 2563 scope.go:117] "RemoveContainer" containerID="16301edc4e62487354bab19d13bd3798bfa6494def8042cc3f02e335c44e6cc0" Jul 07 19:29:20.894406 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:29:20.894389 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16301edc4e62487354bab19d13bd3798bfa6494def8042cc3f02e335c44e6cc0\": container with ID starting with 16301edc4e62487354bab19d13bd3798bfa6494def8042cc3f02e335c44e6cc0 not found: ID does not exist" containerID="16301edc4e62487354bab19d13bd3798bfa6494def8042cc3f02e335c44e6cc0" Jul 07 19:29:20.894452 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:29:20.894418 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16301edc4e62487354bab19d13bd3798bfa6494def8042cc3f02e335c44e6cc0"} err="failed to get container status \"16301edc4e62487354bab19d13bd3798bfa6494def8042cc3f02e335c44e6cc0\": rpc error: code = NotFound desc = could not find container \"16301edc4e62487354bab19d13bd3798bfa6494def8042cc3f02e335c44e6cc0\": container with ID starting with 16301edc4e62487354bab19d13bd3798bfa6494def8042cc3f02e335c44e6cc0 not found: ID does not exist" Jul 07 19:29:20.894452 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:29:20.894436 2563 scope.go:117] "RemoveContainer" containerID="fb78a7f8f53c7d9533a26effbbeaebb52da77a7d2e6220e3dec1b6b9da4fd029" Jul 07 19:29:20.894691 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:29:20.894671 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb78a7f8f53c7d9533a26effbbeaebb52da77a7d2e6220e3dec1b6b9da4fd029\": container with ID starting with fb78a7f8f53c7d9533a26effbbeaebb52da77a7d2e6220e3dec1b6b9da4fd029 not found: ID does not exist" containerID="fb78a7f8f53c7d9533a26effbbeaebb52da77a7d2e6220e3dec1b6b9da4fd029" Jul 07 19:29:20.894727 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:29:20.894700 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb78a7f8f53c7d9533a26effbbeaebb52da77a7d2e6220e3dec1b6b9da4fd029"} err="failed to get container status \"fb78a7f8f53c7d9533a26effbbeaebb52da77a7d2e6220e3dec1b6b9da4fd029\": rpc error: code = NotFound desc = could not find container \"fb78a7f8f53c7d9533a26effbbeaebb52da77a7d2e6220e3dec1b6b9da4fd029\": container with ID starting with fb78a7f8f53c7d9533a26effbbeaebb52da77a7d2e6220e3dec1b6b9da4fd029 not found: ID does not exist" Jul 07 19:29:20.903024 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:29:20.902992 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-85ca1257-openshift-default-68d7d85d75-khqz8"] Jul 07 19:29:20.908006 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:29:20.907939 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-85ca1257-openshift-default-68d7d85d75-khqz8"] Jul 07 19:29:20.982744 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:29:20.982710 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:29:21.476116 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:29:21.476085 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c8356506-39e1-405a-98fd-c49c8c498764" path="/var/lib/kubelet/pods/c8356506-39e1-405a-98fd-c49c8c498764/volumes" Jul 07 19:29:26.050660 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:29:26.050621 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:29:26.150946 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:29:26.150916 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:29:27.613981 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:29:27.613926 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:29:27.697387 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:29:27.697353 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:29:29.872677 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:29:29.872641 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:29:34.436309 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:29:34.436276 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:29:34.740927 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:29:34.740848 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-d23e2962-openshift-default-66d7df967-nzpxz"] Jul 07 19:29:34.741247 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:29:34.741234 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c8356506-39e1-405a-98fd-c49c8c498764" containerName="istio-proxy" Jul 07 19:29:34.741291 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:29:34.741248 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="c8356506-39e1-405a-98fd-c49c8c498764" containerName="istio-proxy" Jul 07 19:29:34.741291 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:29:34.741267 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c8356506-39e1-405a-98fd-c49c8c498764" containerName="istio-proxy" Jul 07 19:29:34.741291 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:29:34.741273 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="c8356506-39e1-405a-98fd-c49c8c498764" containerName="istio-proxy" Jul 07 19:29:34.741403 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:29:34.741335 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="c8356506-39e1-405a-98fd-c49c8c498764" containerName="istio-proxy" Jul 07 19:29:34.741403 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:29:34.741342 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="c8356506-39e1-405a-98fd-c49c8c498764" containerName="istio-proxy" Jul 07 19:29:34.745911 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:29:34.745889 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d23e2962-openshift-default-66d7df967-nzpxz" Jul 07 19:29:34.748134 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:29:34.748115 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-d23e2962-openshift-default-dockercfg-ltxl7\"" Jul 07 19:29:34.762406 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:29:34.762378 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-d23e2962-openshift-default-66d7df967-nzpxz"] Jul 07 19:29:34.765336 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:29:34.765312 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34360072-32e9-49fc-a2c3-d445848bb987-istio-token\") pod \"e2e-mt-d23e2962-openshift-default-66d7df967-nzpxz\" (UID: \"34360072-32e9-49fc-a2c3-d445848bb987\") " pod="openshift-ingress/e2e-mt-d23e2962-openshift-default-66d7df967-nzpxz" Jul 07 19:29:34.765454 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:29:34.765355 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xjrt\" (UniqueName: \"kubernetes.io/projected/34360072-32e9-49fc-a2c3-d445848bb987-kube-api-access-8xjrt\") pod \"e2e-mt-d23e2962-openshift-default-66d7df967-nzpxz\" (UID: \"34360072-32e9-49fc-a2c3-d445848bb987\") " pod="openshift-ingress/e2e-mt-d23e2962-openshift-default-66d7df967-nzpxz" Jul 07 19:29:34.765454 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:29:34.765442 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34360072-32e9-49fc-a2c3-d445848bb987-credential-socket\") pod \"e2e-mt-d23e2962-openshift-default-66d7df967-nzpxz\" (UID: \"34360072-32e9-49fc-a2c3-d445848bb987\") " pod="openshift-ingress/e2e-mt-d23e2962-openshift-default-66d7df967-nzpxz" Jul 07 19:29:34.765614 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:29:34.765475 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34360072-32e9-49fc-a2c3-d445848bb987-workload-socket\") pod \"e2e-mt-d23e2962-openshift-default-66d7df967-nzpxz\" (UID: \"34360072-32e9-49fc-a2c3-d445848bb987\") " pod="openshift-ingress/e2e-mt-d23e2962-openshift-default-66d7df967-nzpxz" Jul 07 19:29:34.765614 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:29:34.765548 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34360072-32e9-49fc-a2c3-d445848bb987-istiod-ca-cert\") pod \"e2e-mt-d23e2962-openshift-default-66d7df967-nzpxz\" (UID: \"34360072-32e9-49fc-a2c3-d445848bb987\") " pod="openshift-ingress/e2e-mt-d23e2962-openshift-default-66d7df967-nzpxz" Jul 07 19:29:34.765614 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:29:34.765569 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34360072-32e9-49fc-a2c3-d445848bb987-workload-certs\") pod \"e2e-mt-d23e2962-openshift-default-66d7df967-nzpxz\" (UID: \"34360072-32e9-49fc-a2c3-d445848bb987\") " pod="openshift-ingress/e2e-mt-d23e2962-openshift-default-66d7df967-nzpxz" Jul 07 19:29:34.765614 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:29:34.765586 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34360072-32e9-49fc-a2c3-d445848bb987-istio-envoy\") pod \"e2e-mt-d23e2962-openshift-default-66d7df967-nzpxz\" (UID: \"34360072-32e9-49fc-a2c3-d445848bb987\") " pod="openshift-ingress/e2e-mt-d23e2962-openshift-default-66d7df967-nzpxz" Jul 07 19:29:34.765810 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:29:34.765627 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34360072-32e9-49fc-a2c3-d445848bb987-istio-data\") pod \"e2e-mt-d23e2962-openshift-default-66d7df967-nzpxz\" (UID: \"34360072-32e9-49fc-a2c3-d445848bb987\") " pod="openshift-ingress/e2e-mt-d23e2962-openshift-default-66d7df967-nzpxz" Jul 07 19:29:34.765810 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:29:34.765653 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34360072-32e9-49fc-a2c3-d445848bb987-istio-podinfo\") pod \"e2e-mt-d23e2962-openshift-default-66d7df967-nzpxz\" (UID: \"34360072-32e9-49fc-a2c3-d445848bb987\") " pod="openshift-ingress/e2e-mt-d23e2962-openshift-default-66d7df967-nzpxz" Jul 07 19:29:34.772362 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:29:34.772335 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:29:34.866483 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:29:34.866437 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34360072-32e9-49fc-a2c3-d445848bb987-istio-token\") pod \"e2e-mt-d23e2962-openshift-default-66d7df967-nzpxz\" (UID: \"34360072-32e9-49fc-a2c3-d445848bb987\") " pod="openshift-ingress/e2e-mt-d23e2962-openshift-default-66d7df967-nzpxz" Jul 07 19:29:34.866483 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:29:34.866482 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xjrt\" (UniqueName: \"kubernetes.io/projected/34360072-32e9-49fc-a2c3-d445848bb987-kube-api-access-8xjrt\") pod \"e2e-mt-d23e2962-openshift-default-66d7df967-nzpxz\" (UID: \"34360072-32e9-49fc-a2c3-d445848bb987\") " pod="openshift-ingress/e2e-mt-d23e2962-openshift-default-66d7df967-nzpxz" Jul 07 19:29:34.866765 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:29:34.866512 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34360072-32e9-49fc-a2c3-d445848bb987-credential-socket\") pod \"e2e-mt-d23e2962-openshift-default-66d7df967-nzpxz\" (UID: \"34360072-32e9-49fc-a2c3-d445848bb987\") " pod="openshift-ingress/e2e-mt-d23e2962-openshift-default-66d7df967-nzpxz" Jul 07 19:29:34.866765 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:29:34.866529 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34360072-32e9-49fc-a2c3-d445848bb987-workload-socket\") pod \"e2e-mt-d23e2962-openshift-default-66d7df967-nzpxz\" (UID: \"34360072-32e9-49fc-a2c3-d445848bb987\") " pod="openshift-ingress/e2e-mt-d23e2962-openshift-default-66d7df967-nzpxz" Jul 07 19:29:34.866765 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:29:34.866574 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34360072-32e9-49fc-a2c3-d445848bb987-istiod-ca-cert\") pod \"e2e-mt-d23e2962-openshift-default-66d7df967-nzpxz\" (UID: \"34360072-32e9-49fc-a2c3-d445848bb987\") " pod="openshift-ingress/e2e-mt-d23e2962-openshift-default-66d7df967-nzpxz" Jul 07 19:29:34.866765 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:29:34.866594 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34360072-32e9-49fc-a2c3-d445848bb987-workload-certs\") pod \"e2e-mt-d23e2962-openshift-default-66d7df967-nzpxz\" (UID: \"34360072-32e9-49fc-a2c3-d445848bb987\") " pod="openshift-ingress/e2e-mt-d23e2962-openshift-default-66d7df967-nzpxz" Jul 07 19:29:34.866765 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:29:34.866614 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34360072-32e9-49fc-a2c3-d445848bb987-istio-envoy\") pod \"e2e-mt-d23e2962-openshift-default-66d7df967-nzpxz\" (UID: \"34360072-32e9-49fc-a2c3-d445848bb987\") " pod="openshift-ingress/e2e-mt-d23e2962-openshift-default-66d7df967-nzpxz" Jul 07 19:29:34.866765 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:29:34.866636 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34360072-32e9-49fc-a2c3-d445848bb987-istio-data\") pod \"e2e-mt-d23e2962-openshift-default-66d7df967-nzpxz\" (UID: \"34360072-32e9-49fc-a2c3-d445848bb987\") " pod="openshift-ingress/e2e-mt-d23e2962-openshift-default-66d7df967-nzpxz" Jul 07 19:29:34.866765 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:29:34.866663 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34360072-32e9-49fc-a2c3-d445848bb987-istio-podinfo\") pod \"e2e-mt-d23e2962-openshift-default-66d7df967-nzpxz\" (UID: \"34360072-32e9-49fc-a2c3-d445848bb987\") " pod="openshift-ingress/e2e-mt-d23e2962-openshift-default-66d7df967-nzpxz" Jul 07 19:29:34.867182 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:29:34.866987 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34360072-32e9-49fc-a2c3-d445848bb987-credential-socket\") pod \"e2e-mt-d23e2962-openshift-default-66d7df967-nzpxz\" (UID: \"34360072-32e9-49fc-a2c3-d445848bb987\") " pod="openshift-ingress/e2e-mt-d23e2962-openshift-default-66d7df967-nzpxz" Jul 07 19:29:34.867182 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:29:34.867057 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34360072-32e9-49fc-a2c3-d445848bb987-workload-socket\") pod \"e2e-mt-d23e2962-openshift-default-66d7df967-nzpxz\" (UID: \"34360072-32e9-49fc-a2c3-d445848bb987\") " pod="openshift-ingress/e2e-mt-d23e2962-openshift-default-66d7df967-nzpxz" Jul 07 19:29:34.867282 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:29:34.867241 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34360072-32e9-49fc-a2c3-d445848bb987-istio-data\") pod \"e2e-mt-d23e2962-openshift-default-66d7df967-nzpxz\" (UID: \"34360072-32e9-49fc-a2c3-d445848bb987\") " pod="openshift-ingress/e2e-mt-d23e2962-openshift-default-66d7df967-nzpxz" Jul 07 19:29:34.867282 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:29:34.867251 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34360072-32e9-49fc-a2c3-d445848bb987-workload-certs\") pod \"e2e-mt-d23e2962-openshift-default-66d7df967-nzpxz\" (UID: \"34360072-32e9-49fc-a2c3-d445848bb987\") " pod="openshift-ingress/e2e-mt-d23e2962-openshift-default-66d7df967-nzpxz" Jul 07 19:29:34.867449 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:29:34.867390 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34360072-32e9-49fc-a2c3-d445848bb987-istiod-ca-cert\") pod \"e2e-mt-d23e2962-openshift-default-66d7df967-nzpxz\" (UID: \"34360072-32e9-49fc-a2c3-d445848bb987\") " pod="openshift-ingress/e2e-mt-d23e2962-openshift-default-66d7df967-nzpxz" Jul 07 19:29:34.869110 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:29:34.869085 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34360072-32e9-49fc-a2c3-d445848bb987-istio-envoy\") pod \"e2e-mt-d23e2962-openshift-default-66d7df967-nzpxz\" (UID: \"34360072-32e9-49fc-a2c3-d445848bb987\") " pod="openshift-ingress/e2e-mt-d23e2962-openshift-default-66d7df967-nzpxz" Jul 07 19:29:34.869266 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:29:34.869251 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34360072-32e9-49fc-a2c3-d445848bb987-istio-podinfo\") pod \"e2e-mt-d23e2962-openshift-default-66d7df967-nzpxz\" (UID: \"34360072-32e9-49fc-a2c3-d445848bb987\") " pod="openshift-ingress/e2e-mt-d23e2962-openshift-default-66d7df967-nzpxz" Jul 07 19:29:34.878530 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:29:34.878509 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34360072-32e9-49fc-a2c3-d445848bb987-istio-token\") pod \"e2e-mt-d23e2962-openshift-default-66d7df967-nzpxz\" (UID: \"34360072-32e9-49fc-a2c3-d445848bb987\") " pod="openshift-ingress/e2e-mt-d23e2962-openshift-default-66d7df967-nzpxz" Jul 07 19:29:34.878613 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:29:34.878569 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xjrt\" (UniqueName: \"kubernetes.io/projected/34360072-32e9-49fc-a2c3-d445848bb987-kube-api-access-8xjrt\") pod \"e2e-mt-d23e2962-openshift-default-66d7df967-nzpxz\" (UID: \"34360072-32e9-49fc-a2c3-d445848bb987\") " pod="openshift-ingress/e2e-mt-d23e2962-openshift-default-66d7df967-nzpxz" Jul 07 19:29:35.056321 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:29:35.056291 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d23e2962-openshift-default-66d7df967-nzpxz" Jul 07 19:29:35.134574 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:29:35.134545 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:29:35.200198 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:29:35.200175 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-d23e2962-openshift-default-66d7df967-nzpxz"] Jul 07 19:29:35.202629 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:29:35.202605 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34360072_32e9_49fc_a2c3_d445848bb987.slice/crio-1cf1fd4c4441b716f56c97a8e9777a82afa953b300a20634efdbdac3db0dc180 WatchSource:0}: Error finding container 1cf1fd4c4441b716f56c97a8e9777a82afa953b300a20634efdbdac3db0dc180: Status 404 returned error can't find the container with id 1cf1fd4c4441b716f56c97a8e9777a82afa953b300a20634efdbdac3db0dc180 Jul 07 19:29:35.204633 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:29:35.204600 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 07 19:29:35.204708 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:29:35.204671 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 07 19:29:35.204708 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:29:35.204698 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 07 19:29:35.951982 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:29:35.951931 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d23e2962-openshift-default-66d7df967-nzpxz" event={"ID":"34360072-32e9-49fc-a2c3-d445848bb987","Type":"ContainerStarted","Data":"c5b26bcf802eac945d69509817372c2e89e8d6fca44218ce7bb0f8b7983e9310"} Jul 07 19:29:35.952410 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:29:35.952001 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d23e2962-openshift-default-66d7df967-nzpxz" event={"ID":"34360072-32e9-49fc-a2c3-d445848bb987","Type":"ContainerStarted","Data":"1cf1fd4c4441b716f56c97a8e9777a82afa953b300a20634efdbdac3db0dc180"} Jul 07 19:29:36.057246 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:29:36.057218 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-d23e2962-openshift-default-66d7df967-nzpxz" Jul 07 19:29:36.062059 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:29:36.062038 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-d23e2962-openshift-default-66d7df967-nzpxz" Jul 07 19:29:36.084195 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:29:36.084144 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-d23e2962-openshift-default-66d7df967-nzpxz" podStartSLOduration=2.08412743 podStartE2EDuration="2.08412743s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:29:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:29:35.97545704 +0000 UTC m=+1758.991461221" watchObservedRunningTime="2026-07-07 19:29:36.08412743 +0000 UTC m=+1759.100131610" Jul 07 19:29:36.957223 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:29:36.957190 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-d23e2962-openshift-default-66d7df967-nzpxz" Jul 07 19:29:36.958054 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:29:36.958036 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-d23e2962-openshift-default-66d7df967-nzpxz" Jul 07 19:29:37.966377 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:29:37.966346 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:29:38.065445 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:29:38.065418 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:29:38.173007 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:29:38.172975 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:29:38.514529 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:29:38.514489 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:29:38.740893 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:29:38.740860 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:29:38.841515 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:29:38.841482 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:29:42.476912 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:29:42.476872 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:29:50.023593 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:29:50.023556 2563 generic.go:356] "Generic (PLEG): container finished" podID="34360072-32e9-49fc-a2c3-d445848bb987" containerID="c5b26bcf802eac945d69509817372c2e89e8d6fca44218ce7bb0f8b7983e9310" exitCode=0 Jul 07 19:29:50.024044 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:29:50.023630 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d23e2962-openshift-default-66d7df967-nzpxz" event={"ID":"34360072-32e9-49fc-a2c3-d445848bb987","Type":"ContainerDied","Data":"c5b26bcf802eac945d69509817372c2e89e8d6fca44218ce7bb0f8b7983e9310"} Jul 07 19:29:50.024044 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:29:50.024041 2563 scope.go:117] "RemoveContainer" containerID="c5b26bcf802eac945d69509817372c2e89e8d6fca44218ce7bb0f8b7983e9310" Jul 07 19:29:50.025012 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:29:50.024986 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 07 19:29:50.025085 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:29:50.025051 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 07 19:29:50.025123 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:29:50.025079 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 07 19:29:51.030718 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:29:51.030683 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d23e2962-openshift-default-66d7df967-nzpxz" event={"ID":"34360072-32e9-49fc-a2c3-d445848bb987","Type":"ContainerStarted","Data":"96643a40295df8cafdf50455bb2eb0943ca8dfe1b197c20a22f681923f565d95"} Jul 07 19:29:51.057038 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:29:51.057008 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-d23e2962-openshift-default-66d7df967-nzpxz" Jul 07 19:29:51.058118 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:29:51.058096 2563 patch_prober.go:28] interesting pod/e2e-mt-d23e2962-openshift-default-66d7df967-nzpxz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body= Jul 07 19:29:51.058189 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:29:51.058144 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d23e2962-openshift-default-66d7df967-nzpxz" podUID="34360072-32e9-49fc-a2c3-d445848bb987" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" Jul 07 19:29:51.957588 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:29:51.957549 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-d23e2962-openshift-default-66d7df967-nzpxz" Jul 07 19:29:52.056787 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:29:52.056756 2563 patch_prober.go:28] interesting pod/e2e-mt-d23e2962-openshift-default-66d7df967-nzpxz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body= Jul 07 19:29:52.057240 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:29:52.056810 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d23e2962-openshift-default-66d7df967-nzpxz" podUID="34360072-32e9-49fc-a2c3-d445848bb987" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" Jul 07 19:29:53.056916 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:29:53.056884 2563 patch_prober.go:28] interesting pod/e2e-mt-d23e2962-openshift-default-66d7df967-nzpxz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body= Jul 07 19:29:53.057335 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:29:53.056972 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d23e2962-openshift-default-66d7df967-nzpxz" podUID="34360072-32e9-49fc-a2c3-d445848bb987" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" Jul 07 19:29:54.057318 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:29:54.057283 2563 patch_prober.go:28] interesting pod/e2e-mt-d23e2962-openshift-default-66d7df967-nzpxz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body= Jul 07 19:29:54.057727 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:29:54.057337 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d23e2962-openshift-default-66d7df967-nzpxz" podUID="34360072-32e9-49fc-a2c3-d445848bb987" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" Jul 07 19:29:55.060882 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:29:55.060853 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-d23e2962-openshift-default-66d7df967-nzpxz" Jul 07 19:29:55.061701 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:29:55.061684 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-d23e2962-openshift-default-66d7df967-nzpxz" Jul 07 19:29:58.783371 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:29:58.783328 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:30:17.578700 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:30:17.578670 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-frt7c_e25ab82f-72fb-4cde-89c4-0a0dd80f5feb/console-operator/2.log" Jul 07 19:30:17.583671 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:30:17.583648 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-frt7c_e25ab82f-72fb-4cde-89c4-0a0dd80f5feb/console-operator/2.log" Jul 07 19:30:17.585001 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:30:17.584981 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zzwbq_4f0c5d5c-147f-426a-965d-8886e5e79984/ovn-acl-logging/0.log" Jul 07 19:30:17.589888 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:30:17.589872 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zzwbq_4f0c5d5c-147f-426a-965d-8886e5e79984/ovn-acl-logging/0.log" Jul 07 19:30:26.040798 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:30:26.040754 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:30:26.446671 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:30:26.446597 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:30:32.116633 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:30:32.116595 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:30:37.183624 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:30:37.183588 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:31:13.640714 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:31:13.640677 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:31:14.995198 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:31:14.995166 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:31:21.336645 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:31:21.336607 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:31:28.012254 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:31:28.012219 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:31:28.089996 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:31:28.089965 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:31:28.280827 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:31:28.280798 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-d23e2962-openshift-default-66d7df967-nzpxz"] Jul 07 19:31:28.281048 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:31:28.281012 2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-d23e2962-openshift-default-66d7df967-nzpxz" podUID="34360072-32e9-49fc-a2c3-d445848bb987" containerName="istio-proxy" containerID="cri-o://96643a40295df8cafdf50455bb2eb0943ca8dfe1b197c20a22f681923f565d95" gracePeriod=30 Jul 07 19:31:28.304144 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:31:28.304116 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:31:28.532601 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:31:28.532505 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:31:29.605182 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:31:29.605141 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:31:29.704468 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:31:29.704440 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:31:29.805203 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:31:29.805162 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:31:30.575338 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:31:30.575302 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:31:30.680428 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:31:30.680388 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:31:30.898671 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:31:30.898580 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:31:32.226431 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:31:32.226398 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:31:33.536474 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:31:33.536445 2563 generic.go:356] "Generic (PLEG): container finished" podID="34360072-32e9-49fc-a2c3-d445848bb987" containerID="96643a40295df8cafdf50455bb2eb0943ca8dfe1b197c20a22f681923f565d95" exitCode=0 Jul 07 19:31:33.536805 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:31:33.536519 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d23e2962-openshift-default-66d7df967-nzpxz" event={"ID":"34360072-32e9-49fc-a2c3-d445848bb987","Type":"ContainerDied","Data":"96643a40295df8cafdf50455bb2eb0943ca8dfe1b197c20a22f681923f565d95"} Jul 07 19:31:33.536805 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:31:33.536571 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d23e2962-openshift-default-66d7df967-nzpxz" event={"ID":"34360072-32e9-49fc-a2c3-d445848bb987","Type":"ContainerDied","Data":"1cf1fd4c4441b716f56c97a8e9777a82afa953b300a20634efdbdac3db0dc180"} Jul 07 19:31:33.536805 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:31:33.536590 2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1cf1fd4c4441b716f56c97a8e9777a82afa953b300a20634efdbdac3db0dc180" Jul 07 19:31:33.536805 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:31:33.536603 2563 scope.go:117] "RemoveContainer" containerID="c5b26bcf802eac945d69509817372c2e89e8d6fca44218ce7bb0f8b7983e9310" Jul 07 19:31:33.538531 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:31:33.538510 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d23e2962-openshift-default-66d7df967-nzpxz" Jul 07 19:31:33.659281 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:31:33.659254 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34360072-32e9-49fc-a2c3-d445848bb987-workload-certs\") pod \"34360072-32e9-49fc-a2c3-d445848bb987\" (UID: \"34360072-32e9-49fc-a2c3-d445848bb987\") " Jul 07 19:31:33.659452 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:31:33.659291 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34360072-32e9-49fc-a2c3-d445848bb987-istio-podinfo\") pod \"34360072-32e9-49fc-a2c3-d445848bb987\" (UID: \"34360072-32e9-49fc-a2c3-d445848bb987\") " Jul 07 19:31:33.659452 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:31:33.659345 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34360072-32e9-49fc-a2c3-d445848bb987-workload-socket\") pod \"34360072-32e9-49fc-a2c3-d445848bb987\" (UID: \"34360072-32e9-49fc-a2c3-d445848bb987\") " Jul 07 19:31:33.659452 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:31:33.659380 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34360072-32e9-49fc-a2c3-d445848bb987-istiod-ca-cert\") pod \"34360072-32e9-49fc-a2c3-d445848bb987\" (UID: \"34360072-32e9-49fc-a2c3-d445848bb987\") " Jul 07 19:31:33.659452 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:31:33.659415 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8xjrt\" (UniqueName: \"kubernetes.io/projected/34360072-32e9-49fc-a2c3-d445848bb987-kube-api-access-8xjrt\") pod \"34360072-32e9-49fc-a2c3-d445848bb987\" (UID: \"34360072-32e9-49fc-a2c3-d445848bb987\") " Jul 07 19:31:33.659667 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:31:33.659463 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34360072-32e9-49fc-a2c3-d445848bb987-istio-token\") pod \"34360072-32e9-49fc-a2c3-d445848bb987\" (UID: \"34360072-32e9-49fc-a2c3-d445848bb987\") " Jul 07 19:31:33.659667 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:31:33.659528 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34360072-32e9-49fc-a2c3-d445848bb987-istio-envoy\") pod \"34360072-32e9-49fc-a2c3-d445848bb987\" (UID: \"34360072-32e9-49fc-a2c3-d445848bb987\") " Jul 07 19:31:33.659667 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:31:33.659555 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34360072-32e9-49fc-a2c3-d445848bb987-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "34360072-32e9-49fc-a2c3-d445848bb987" (UID: "34360072-32e9-49fc-a2c3-d445848bb987"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:31:33.659667 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:31:33.659597 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34360072-32e9-49fc-a2c3-d445848bb987-credential-socket\") pod \"34360072-32e9-49fc-a2c3-d445848bb987\" (UID: \"34360072-32e9-49fc-a2c3-d445848bb987\") " Jul 07 19:31:33.659667 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:31:33.659641 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34360072-32e9-49fc-a2c3-d445848bb987-istio-data\") pod \"34360072-32e9-49fc-a2c3-d445848bb987\" (UID: \"34360072-32e9-49fc-a2c3-d445848bb987\") " Jul 07 19:31:33.659667 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:31:33.659645 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34360072-32e9-49fc-a2c3-d445848bb987-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "34360072-32e9-49fc-a2c3-d445848bb987" (UID: "34360072-32e9-49fc-a2c3-d445848bb987"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:31:33.660033 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:31:33.659871 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34360072-32e9-49fc-a2c3-d445848bb987-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "34360072-32e9-49fc-a2c3-d445848bb987" (UID: "34360072-32e9-49fc-a2c3-d445848bb987"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:31:33.660170 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:31:33.660142 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34360072-32e9-49fc-a2c3-d445848bb987-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "34360072-32e9-49fc-a2c3-d445848bb987" (UID: "34360072-32e9-49fc-a2c3-d445848bb987"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 19:31:33.660683 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:31:33.660663 2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34360072-32e9-49fc-a2c3-d445848bb987-credential-socket\") on node \"ip-10-0-142-134.ec2.internal\" DevicePath \"\"" Jul 07 19:31:33.660747 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:31:33.660689 2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34360072-32e9-49fc-a2c3-d445848bb987-workload-certs\") on node \"ip-10-0-142-134.ec2.internal\" DevicePath \"\"" Jul 07 19:31:33.660747 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:31:33.660703 2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34360072-32e9-49fc-a2c3-d445848bb987-workload-socket\") on node \"ip-10-0-142-134.ec2.internal\" DevicePath \"\"" Jul 07 19:31:33.660747 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:31:33.660716 2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34360072-32e9-49fc-a2c3-d445848bb987-istiod-ca-cert\") on node \"ip-10-0-142-134.ec2.internal\" DevicePath \"\"" Jul 07 19:31:33.660747 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:31:33.660726 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34360072-32e9-49fc-a2c3-d445848bb987-istio-data" (OuterVolumeSpecName: "istio-data") pod "34360072-32e9-49fc-a2c3-d445848bb987" (UID: "34360072-32e9-49fc-a2c3-d445848bb987"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:31:33.661574 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:31:33.661548 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/34360072-32e9-49fc-a2c3-d445848bb987-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "34360072-32e9-49fc-a2c3-d445848bb987" (UID: "34360072-32e9-49fc-a2c3-d445848bb987"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 07 19:31:33.661707 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:31:33.661689 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34360072-32e9-49fc-a2c3-d445848bb987-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "34360072-32e9-49fc-a2c3-d445848bb987" (UID: "34360072-32e9-49fc-a2c3-d445848bb987"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:31:33.662041 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:31:33.662015 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34360072-32e9-49fc-a2c3-d445848bb987-istio-token" (OuterVolumeSpecName: "istio-token") pod "34360072-32e9-49fc-a2c3-d445848bb987" (UID: "34360072-32e9-49fc-a2c3-d445848bb987"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 19:31:33.662041 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:31:33.662013 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34360072-32e9-49fc-a2c3-d445848bb987-kube-api-access-8xjrt" (OuterVolumeSpecName: "kube-api-access-8xjrt") pod "34360072-32e9-49fc-a2c3-d445848bb987" (UID: "34360072-32e9-49fc-a2c3-d445848bb987"). InnerVolumeSpecName "kube-api-access-8xjrt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 19:31:33.761397 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:31:33.761369 2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34360072-32e9-49fc-a2c3-d445848bb987-istio-token\") on node \"ip-10-0-142-134.ec2.internal\" DevicePath \"\"" Jul 07 19:31:33.761397 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:31:33.761394 2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34360072-32e9-49fc-a2c3-d445848bb987-istio-envoy\") on node \"ip-10-0-142-134.ec2.internal\" DevicePath \"\"" Jul 07 19:31:33.761397 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:31:33.761402 2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34360072-32e9-49fc-a2c3-d445848bb987-istio-data\") on node \"ip-10-0-142-134.ec2.internal\" DevicePath \"\"" Jul 07 19:31:33.761610 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:31:33.761410 2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34360072-32e9-49fc-a2c3-d445848bb987-istio-podinfo\") on node \"ip-10-0-142-134.ec2.internal\" DevicePath \"\"" Jul 07 19:31:33.761610 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:31:33.761418 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8xjrt\" (UniqueName: \"kubernetes.io/projected/34360072-32e9-49fc-a2c3-d445848bb987-kube-api-access-8xjrt\") on node \"ip-10-0-142-134.ec2.internal\" DevicePath \"\"" Jul 07 19:31:34.543698 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:31:34.543669 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d23e2962-openshift-default-66d7df967-nzpxz" Jul 07 19:31:34.582927 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:31:34.582887 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-d23e2962-openshift-default-66d7df967-nzpxz"] Jul 07 19:31:34.588896 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:31:34.588856 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-d23e2962-openshift-default-66d7df967-nzpxz"] Jul 07 19:31:35.476362 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:31:35.476327 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="34360072-32e9-49fc-a2c3-d445848bb987" path="/var/lib/kubelet/pods/34360072-32e9-49fc-a2c3-d445848bb987/volumes" Jul 07 19:31:39.191658 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:31:39.191620 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:31:39.294714 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:31:39.294680 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:31:39.880099 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:31:39.880060 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:31:47.805198 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:31:47.805163 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:31:47.948668 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:31:47.948621 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:31:48.102019 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:31:48.101931 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:31:58.574236 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:31:58.574197 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:31:58.938321 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:31:58.938242 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:32:19.896370 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:32:19.896341 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:32:27.262043 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:32:27.262010 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:32:27.351153 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:32:27.351117 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:32:27.627933 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:32:27.627903 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:32:27.948363 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:32:27.948279 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-a17037-openshift-default-5df47d4cc5-nwn4s"] Jul 07 19:32:27.948765 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:32:27.948746 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="34360072-32e9-49fc-a2c3-d445848bb987" containerName="istio-proxy" Jul 07 19:32:27.948864 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:32:27.948767 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="34360072-32e9-49fc-a2c3-d445848bb987" containerName="istio-proxy" Jul 07 19:32:27.948864 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:32:27.948783 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="34360072-32e9-49fc-a2c3-d445848bb987" containerName="istio-proxy" Jul 07 19:32:27.948864 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:32:27.948791 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="34360072-32e9-49fc-a2c3-d445848bb987" containerName="istio-proxy" Jul 07 19:32:27.949041 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:32:27.948884 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="34360072-32e9-49fc-a2c3-d445848bb987" containerName="istio-proxy" Jul 07 19:32:27.949041 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:32:27.948902 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="34360072-32e9-49fc-a2c3-d445848bb987" containerName="istio-proxy" Jul 07 19:32:27.952000 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:32:27.951978 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a17037-openshift-default-5df47d4cc5-nwn4s" Jul 07 19:32:27.953982 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:32:27.953935 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-a17037-openshift-default-dockercfg-lf89q\"" Jul 07 19:32:27.963919 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:32:27.963896 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-a17037-openshift-default-5df47d4cc5-nwn4s"] Jul 07 19:32:27.993867 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:32:27.993835 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:32:28.093667 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:32:28.093630 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9x6wb\" (UniqueName: \"kubernetes.io/projected/9007ca76-b883-45e8-a276-22feea095a05-kube-api-access-9x6wb\") pod \"e2e-inf-a-a17037-openshift-default-5df47d4cc5-nwn4s\" (UID: \"9007ca76-b883-45e8-a276-22feea095a05\") " pod="openshift-ingress/e2e-inf-a-a17037-openshift-default-5df47d4cc5-nwn4s" Jul 07 19:32:28.093849 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:32:28.093684 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9007ca76-b883-45e8-a276-22feea095a05-istio-envoy\") pod \"e2e-inf-a-a17037-openshift-default-5df47d4cc5-nwn4s\" (UID: \"9007ca76-b883-45e8-a276-22feea095a05\") " pod="openshift-ingress/e2e-inf-a-a17037-openshift-default-5df47d4cc5-nwn4s" Jul 07 19:32:28.093849 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:32:28.093781 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9007ca76-b883-45e8-a276-22feea095a05-istiod-ca-cert\") pod \"e2e-inf-a-a17037-openshift-default-5df47d4cc5-nwn4s\" (UID: \"9007ca76-b883-45e8-a276-22feea095a05\") " pod="openshift-ingress/e2e-inf-a-a17037-openshift-default-5df47d4cc5-nwn4s" Jul 07 19:32:28.093849 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:32:28.093845 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9007ca76-b883-45e8-a276-22feea095a05-credential-socket\") pod \"e2e-inf-a-a17037-openshift-default-5df47d4cc5-nwn4s\" (UID: \"9007ca76-b883-45e8-a276-22feea095a05\") " pod="openshift-ingress/e2e-inf-a-a17037-openshift-default-5df47d4cc5-nwn4s" Jul 07 19:32:28.093963 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:32:28.093879 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9007ca76-b883-45e8-a276-22feea095a05-istio-podinfo\") pod \"e2e-inf-a-a17037-openshift-default-5df47d4cc5-nwn4s\" (UID: \"9007ca76-b883-45e8-a276-22feea095a05\") " pod="openshift-ingress/e2e-inf-a-a17037-openshift-default-5df47d4cc5-nwn4s" Jul 07 19:32:28.093963 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:32:28.093907 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9007ca76-b883-45e8-a276-22feea095a05-istio-token\") pod \"e2e-inf-a-a17037-openshift-default-5df47d4cc5-nwn4s\" (UID: \"9007ca76-b883-45e8-a276-22feea095a05\") " pod="openshift-ingress/e2e-inf-a-a17037-openshift-default-5df47d4cc5-nwn4s" Jul 07 19:32:28.093963 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:32:28.093944 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9007ca76-b883-45e8-a276-22feea095a05-workload-socket\") pod \"e2e-inf-a-a17037-openshift-default-5df47d4cc5-nwn4s\" (UID: \"9007ca76-b883-45e8-a276-22feea095a05\") " pod="openshift-ingress/e2e-inf-a-a17037-openshift-default-5df47d4cc5-nwn4s" Jul 07 19:32:28.094064 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:32:28.094001 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9007ca76-b883-45e8-a276-22feea095a05-istio-data\") pod \"e2e-inf-a-a17037-openshift-default-5df47d4cc5-nwn4s\" (UID: \"9007ca76-b883-45e8-a276-22feea095a05\") " pod="openshift-ingress/e2e-inf-a-a17037-openshift-default-5df47d4cc5-nwn4s" Jul 07 19:32:28.094064 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:32:28.094016 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9007ca76-b883-45e8-a276-22feea095a05-workload-certs\") pod \"e2e-inf-a-a17037-openshift-default-5df47d4cc5-nwn4s\" (UID: \"9007ca76-b883-45e8-a276-22feea095a05\") " pod="openshift-ingress/e2e-inf-a-a17037-openshift-default-5df47d4cc5-nwn4s" Jul 07 19:32:28.195206 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:32:28.195165 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9007ca76-b883-45e8-a276-22feea095a05-workload-socket\") pod \"e2e-inf-a-a17037-openshift-default-5df47d4cc5-nwn4s\" (UID: \"9007ca76-b883-45e8-a276-22feea095a05\") " pod="openshift-ingress/e2e-inf-a-a17037-openshift-default-5df47d4cc5-nwn4s" Jul 07 19:32:28.195206 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:32:28.195214 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9007ca76-b883-45e8-a276-22feea095a05-istio-data\") pod \"e2e-inf-a-a17037-openshift-default-5df47d4cc5-nwn4s\" (UID: \"9007ca76-b883-45e8-a276-22feea095a05\") " pod="openshift-ingress/e2e-inf-a-a17037-openshift-default-5df47d4cc5-nwn4s" Jul 07 19:32:28.195479 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:32:28.195230 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9007ca76-b883-45e8-a276-22feea095a05-workload-certs\") pod \"e2e-inf-a-a17037-openshift-default-5df47d4cc5-nwn4s\" (UID: \"9007ca76-b883-45e8-a276-22feea095a05\") " pod="openshift-ingress/e2e-inf-a-a17037-openshift-default-5df47d4cc5-nwn4s" Jul 07 19:32:28.195479 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:32:28.195262 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9x6wb\" (UniqueName: \"kubernetes.io/projected/9007ca76-b883-45e8-a276-22feea095a05-kube-api-access-9x6wb\") pod \"e2e-inf-a-a17037-openshift-default-5df47d4cc5-nwn4s\" (UID: \"9007ca76-b883-45e8-a276-22feea095a05\") " pod="openshift-ingress/e2e-inf-a-a17037-openshift-default-5df47d4cc5-nwn4s" Jul 07 19:32:28.195479 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:32:28.195289 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9007ca76-b883-45e8-a276-22feea095a05-istio-envoy\") pod \"e2e-inf-a-a17037-openshift-default-5df47d4cc5-nwn4s\" (UID: \"9007ca76-b883-45e8-a276-22feea095a05\") " pod="openshift-ingress/e2e-inf-a-a17037-openshift-default-5df47d4cc5-nwn4s" Jul 07 19:32:28.195479 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:32:28.195316 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9007ca76-b883-45e8-a276-22feea095a05-istiod-ca-cert\") pod \"e2e-inf-a-a17037-openshift-default-5df47d4cc5-nwn4s\" (UID: \"9007ca76-b883-45e8-a276-22feea095a05\") " pod="openshift-ingress/e2e-inf-a-a17037-openshift-default-5df47d4cc5-nwn4s" Jul 07 19:32:28.195479 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:32:28.195454 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9007ca76-b883-45e8-a276-22feea095a05-credential-socket\") pod \"e2e-inf-a-a17037-openshift-default-5df47d4cc5-nwn4s\" (UID: \"9007ca76-b883-45e8-a276-22feea095a05\") " pod="openshift-ingress/e2e-inf-a-a17037-openshift-default-5df47d4cc5-nwn4s" Jul 07 19:32:28.195729 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:32:28.195518 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9007ca76-b883-45e8-a276-22feea095a05-istio-podinfo\") pod \"e2e-inf-a-a17037-openshift-default-5df47d4cc5-nwn4s\" (UID: \"9007ca76-b883-45e8-a276-22feea095a05\") " pod="openshift-ingress/e2e-inf-a-a17037-openshift-default-5df47d4cc5-nwn4s" Jul 07 19:32:28.195729 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:32:28.195552 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9007ca76-b883-45e8-a276-22feea095a05-istio-token\") pod \"e2e-inf-a-a17037-openshift-default-5df47d4cc5-nwn4s\" (UID: \"9007ca76-b883-45e8-a276-22feea095a05\") " pod="openshift-ingress/e2e-inf-a-a17037-openshift-default-5df47d4cc5-nwn4s" Jul 07 19:32:28.195729 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:32:28.195697 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9007ca76-b883-45e8-a276-22feea095a05-workload-socket\") pod \"e2e-inf-a-a17037-openshift-default-5df47d4cc5-nwn4s\" (UID: \"9007ca76-b883-45e8-a276-22feea095a05\") " pod="openshift-ingress/e2e-inf-a-a17037-openshift-default-5df47d4cc5-nwn4s" Jul 07 19:32:28.195873 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:32:28.195775 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9007ca76-b883-45e8-a276-22feea095a05-workload-certs\") pod \"e2e-inf-a-a17037-openshift-default-5df47d4cc5-nwn4s\" (UID: \"9007ca76-b883-45e8-a276-22feea095a05\") " pod="openshift-ingress/e2e-inf-a-a17037-openshift-default-5df47d4cc5-nwn4s" Jul 07 19:32:28.195873 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:32:28.195819 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9007ca76-b883-45e8-a276-22feea095a05-credential-socket\") pod \"e2e-inf-a-a17037-openshift-default-5df47d4cc5-nwn4s\" (UID: \"9007ca76-b883-45e8-a276-22feea095a05\") " pod="openshift-ingress/e2e-inf-a-a17037-openshift-default-5df47d4cc5-nwn4s" Jul 07 19:32:28.195942 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:32:28.195931 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9007ca76-b883-45e8-a276-22feea095a05-istio-data\") pod \"e2e-inf-a-a17037-openshift-default-5df47d4cc5-nwn4s\" (UID: \"9007ca76-b883-45e8-a276-22feea095a05\") " pod="openshift-ingress/e2e-inf-a-a17037-openshift-default-5df47d4cc5-nwn4s" Jul 07 19:32:28.196337 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:32:28.196314 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9007ca76-b883-45e8-a276-22feea095a05-istiod-ca-cert\") pod \"e2e-inf-a-a17037-openshift-default-5df47d4cc5-nwn4s\" (UID: \"9007ca76-b883-45e8-a276-22feea095a05\") " pod="openshift-ingress/e2e-inf-a-a17037-openshift-default-5df47d4cc5-nwn4s" Jul 07 19:32:28.197709 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:32:28.197686 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9007ca76-b883-45e8-a276-22feea095a05-istio-envoy\") pod \"e2e-inf-a-a17037-openshift-default-5df47d4cc5-nwn4s\" (UID: \"9007ca76-b883-45e8-a276-22feea095a05\") " pod="openshift-ingress/e2e-inf-a-a17037-openshift-default-5df47d4cc5-nwn4s" Jul 07 19:32:28.197899 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:32:28.197884 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9007ca76-b883-45e8-a276-22feea095a05-istio-podinfo\") pod \"e2e-inf-a-a17037-openshift-default-5df47d4cc5-nwn4s\" (UID: \"9007ca76-b883-45e8-a276-22feea095a05\") " pod="openshift-ingress/e2e-inf-a-a17037-openshift-default-5df47d4cc5-nwn4s" Jul 07 19:32:28.208661 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:32:28.208592 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9007ca76-b883-45e8-a276-22feea095a05-istio-token\") pod \"e2e-inf-a-a17037-openshift-default-5df47d4cc5-nwn4s\" (UID: \"9007ca76-b883-45e8-a276-22feea095a05\") " pod="openshift-ingress/e2e-inf-a-a17037-openshift-default-5df47d4cc5-nwn4s" Jul 07 19:32:28.208891 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:32:28.208869 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9x6wb\" (UniqueName: \"kubernetes.io/projected/9007ca76-b883-45e8-a276-22feea095a05-kube-api-access-9x6wb\") pod \"e2e-inf-a-a17037-openshift-default-5df47d4cc5-nwn4s\" (UID: \"9007ca76-b883-45e8-a276-22feea095a05\") " pod="openshift-ingress/e2e-inf-a-a17037-openshift-default-5df47d4cc5-nwn4s" Jul 07 19:32:28.262705 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:32:28.262669 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a17037-openshift-default-5df47d4cc5-nwn4s" Jul 07 19:32:28.409471 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:32:28.409444 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-a17037-openshift-default-5df47d4cc5-nwn4s"] Jul 07 19:32:28.410644 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:32:28.410619 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9007ca76_b883_45e8_a276_22feea095a05.slice/crio-0755a4ee4a3158afca9fa8e7af353b7f72b4e684ed0d281d52e19668ba947f1c WatchSource:0}: Error finding container 0755a4ee4a3158afca9fa8e7af353b7f72b4e684ed0d281d52e19668ba947f1c: Status 404 returned error can't find the container with id 0755a4ee4a3158afca9fa8e7af353b7f72b4e684ed0d281d52e19668ba947f1c Jul 07 19:32:28.412770 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:32:28.412743 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 07 19:32:28.412832 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:32:28.412802 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 07 19:32:28.412869 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:32:28.412831 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 07 19:32:28.684358 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:32:28.684312 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:32:28.825160 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:32:28.825123 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a17037-openshift-default-5df47d4cc5-nwn4s" event={"ID":"9007ca76-b883-45e8-a276-22feea095a05","Type":"ContainerStarted","Data":"0677eb4b0cedcdd8413a1ac68780655e0808eb2b496dab3586c84a9998c6028f"} Jul 07 19:32:28.825352 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:32:28.825215 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a17037-openshift-default-5df47d4cc5-nwn4s" event={"ID":"9007ca76-b883-45e8-a276-22feea095a05","Type":"ContainerStarted","Data":"0755a4ee4a3158afca9fa8e7af353b7f72b4e684ed0d281d52e19668ba947f1c"} Jul 07 19:32:28.847855 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:32:28.847797 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-a17037-openshift-default-5df47d4cc5-nwn4s" podStartSLOduration=1.847780562 podStartE2EDuration="1.847780562s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:32:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:32:28.845564196 +0000 UTC m=+1931.861568406" watchObservedRunningTime="2026-07-07 19:32:28.847780562 +0000 UTC m=+1931.863784742" Jul 07 19:32:29.263095 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:32:29.262993 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-a17037-openshift-default-5df47d4cc5-nwn4s" Jul 07 19:32:29.268279 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:32:29.268253 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-a17037-openshift-default-5df47d4cc5-nwn4s" Jul 07 19:32:29.830381 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:32:29.830350 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-a17037-openshift-default-5df47d4cc5-nwn4s" Jul 07 19:32:29.831380 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:32:29.831360 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-a17037-openshift-default-5df47d4cc5-nwn4s" Jul 07 19:32:30.822426 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:32:30.822391 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:32:30.922235 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:32:30.922205 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:32:30.996423 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:32:30.996382 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:32:31.425416 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:32:31.425382 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:32:31.528411 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:32:31.528380 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:32:35.335221 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:32:35.335184 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:32:35.671310 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:32:35.671224 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:32:35.991534 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:32:35.991452 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:32:36.165653 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:32:36.165603 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:32:42.188782 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:32:42.188751 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:32:42.288367 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:32:42.288328 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:32:42.385821 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:32:42.385780 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:32:42.773276 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:32:42.773166 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:32:42.900196 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:32:42.900165 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:32:43.006166 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:32:43.006127 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:32:45.872168 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:32:45.872119 2563 patch_prober.go:28] interesting pod/e2e-inf-a-a17037-openshift-default-5df47d4cc5-nwn4s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 07 19:32:45.872652 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:32:45.872206 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-a17037-openshift-default-5df47d4cc5-nwn4s" podUID="9007ca76-b883-45e8-a276-22feea095a05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 07 19:32:46.748487 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:32:46.748403 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:32:46.918217 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:32:46.918184 2563 generic.go:356] "Generic (PLEG): container finished" podID="9007ca76-b883-45e8-a276-22feea095a05" containerID="0677eb4b0cedcdd8413a1ac68780655e0808eb2b496dab3586c84a9998c6028f" exitCode=0 Jul 07 19:32:46.918626 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:32:46.918251 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a17037-openshift-default-5df47d4cc5-nwn4s" event={"ID":"9007ca76-b883-45e8-a276-22feea095a05","Type":"ContainerDied","Data":"0677eb4b0cedcdd8413a1ac68780655e0808eb2b496dab3586c84a9998c6028f"} Jul 07 19:32:46.918682 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:32:46.918668 2563 scope.go:117] "RemoveContainer" containerID="0677eb4b0cedcdd8413a1ac68780655e0808eb2b496dab3586c84a9998c6028f" Jul 07 19:32:46.919737 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:32:46.919711 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 07 19:32:46.919804 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:32:46.919789 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 07 19:32:46.919844 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:32:46.919824 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 07 19:32:47.924478 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:32:47.924445 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a17037-openshift-default-5df47d4cc5-nwn4s" event={"ID":"9007ca76-b883-45e8-a276-22feea095a05","Type":"ContainerStarted","Data":"24558864156ba0272aa853ea8a044198e6ccbcc940a78665c337c9130e870c22"} Jul 07 19:32:48.263207 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:32:48.263113 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-a17037-openshift-default-5df47d4cc5-nwn4s" Jul 07 19:32:48.263385 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:32:48.263272 2563 patch_prober.go:28] interesting pod/e2e-inf-a-a17037-openshift-default-5df47d4cc5-nwn4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body= Jul 07 19:32:48.263385 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:32:48.263332 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a17037-openshift-default-5df47d4cc5-nwn4s" podUID="9007ca76-b883-45e8-a276-22feea095a05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" Jul 07 19:32:49.263648 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:32:49.263601 2563 patch_prober.go:28] interesting pod/e2e-inf-a-a17037-openshift-default-5df47d4cc5-nwn4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body= Jul 07 19:32:49.264205 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:32:49.263691 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a17037-openshift-default-5df47d4cc5-nwn4s" podUID="9007ca76-b883-45e8-a276-22feea095a05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" Jul 07 19:32:50.264172 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:32:50.264132 2563 patch_prober.go:28] interesting pod/e2e-inf-a-a17037-openshift-default-5df47d4cc5-nwn4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body= Jul 07 19:32:50.264804 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:32:50.264772 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a17037-openshift-default-5df47d4cc5-nwn4s" podUID="9007ca76-b883-45e8-a276-22feea095a05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" Jul 07 19:32:51.264173 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:32:51.264135 2563 patch_prober.go:28] interesting pod/e2e-inf-a-a17037-openshift-default-5df47d4cc5-nwn4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body= Jul 07 19:32:51.264886 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:32:51.264853 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a17037-openshift-default-5df47d4cc5-nwn4s" podUID="9007ca76-b883-45e8-a276-22feea095a05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" Jul 07 19:32:52.267116 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:32:52.267081 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-a17037-openshift-default-5df47d4cc5-nwn4s" Jul 07 19:32:52.267664 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:32:52.267476 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-a17037-openshift-default-5df47d4cc5-nwn4s" Jul 07 19:32:52.268220 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:32:52.268201 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-a17037-openshift-default-5df47d4cc5-nwn4s" Jul 07 19:32:52.479370 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:32:52.479329 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:32:52.575465 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:32:52.575432 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:33:03.695695 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:33:03.695663 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:33:03.795410 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:33:03.795376 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:33:06.788989 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:33:06.788931 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-a17037/test-model-a17037-kserve-867b8d7d99-nrglc"] Jul 07 19:33:06.792780 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:33:06.792759 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-a17037/test-model-a17037-kserve-867b8d7d99-nrglc" Jul 07 19:33:06.794612 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:33:06.794586 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-a17037\"/\"kube-root-ca.crt\"" Jul 07 19:33:06.794717 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:33:06.794658 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-a17037\"/\"default-dockercfg-bx4j5\"" Jul 07 19:33:06.795080 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:33:06.795061 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-a17037\"/\"openshift-service-ca.crt\"" Jul 07 19:33:06.795204 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:33:06.795101 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-a17037\"/\"test-model-a17037-kserve-self-signed-certs\"" Jul 07 19:33:06.808237 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:33:06.808205 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-a17037/test-model-a17037-kserve-867b8d7d99-nrglc"] Jul 07 19:33:06.840737 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:33:06.840706 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:33:06.975741 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:33:06.975702 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8fcfb0d5-5602-4090-95cd-203852cda57c-model-cache\") pod \"test-model-a17037-kserve-867b8d7d99-nrglc\" (UID: \"8fcfb0d5-5602-4090-95cd-203852cda57c\") " pod="ai-tenant-e2e-inf-a-a17037/test-model-a17037-kserve-867b8d7d99-nrglc" Jul 07 19:33:06.975741 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:33:06.975740 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9dmt\" (UniqueName: \"kubernetes.io/projected/8fcfb0d5-5602-4090-95cd-203852cda57c-kube-api-access-x9dmt\") pod \"test-model-a17037-kserve-867b8d7d99-nrglc\" (UID: \"8fcfb0d5-5602-4090-95cd-203852cda57c\") " pod="ai-tenant-e2e-inf-a-a17037/test-model-a17037-kserve-867b8d7d99-nrglc" Jul 07 19:33:06.975935 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:33:06.975787 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8fcfb0d5-5602-4090-95cd-203852cda57c-home\") pod \"test-model-a17037-kserve-867b8d7d99-nrglc\" (UID: \"8fcfb0d5-5602-4090-95cd-203852cda57c\") " pod="ai-tenant-e2e-inf-a-a17037/test-model-a17037-kserve-867b8d7d99-nrglc" Jul 07 19:33:06.975935 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:33:06.975871 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8fcfb0d5-5602-4090-95cd-203852cda57c-tls-certs\") pod \"test-model-a17037-kserve-867b8d7d99-nrglc\" (UID: \"8fcfb0d5-5602-4090-95cd-203852cda57c\") " pod="ai-tenant-e2e-inf-a-a17037/test-model-a17037-kserve-867b8d7d99-nrglc" Jul 07 19:33:06.975935 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:33:06.975892 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8fcfb0d5-5602-4090-95cd-203852cda57c-tmp-dir\") pod \"test-model-a17037-kserve-867b8d7d99-nrglc\" (UID: \"8fcfb0d5-5602-4090-95cd-203852cda57c\") " pod="ai-tenant-e2e-inf-a-a17037/test-model-a17037-kserve-867b8d7d99-nrglc" Jul 07 19:33:06.975935 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:33:06.975914 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8fcfb0d5-5602-4090-95cd-203852cda57c-dshm\") pod \"test-model-a17037-kserve-867b8d7d99-nrglc\" (UID: \"8fcfb0d5-5602-4090-95cd-203852cda57c\") " pod="ai-tenant-e2e-inf-a-a17037/test-model-a17037-kserve-867b8d7d99-nrglc" Jul 07 19:33:07.077139 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:33:07.077054 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8fcfb0d5-5602-4090-95cd-203852cda57c-tls-certs\") pod \"test-model-a17037-kserve-867b8d7d99-nrglc\" (UID: \"8fcfb0d5-5602-4090-95cd-203852cda57c\") " pod="ai-tenant-e2e-inf-a-a17037/test-model-a17037-kserve-867b8d7d99-nrglc" Jul 07 19:33:07.077139 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:33:07.077098 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8fcfb0d5-5602-4090-95cd-203852cda57c-tmp-dir\") pod \"test-model-a17037-kserve-867b8d7d99-nrglc\" (UID: \"8fcfb0d5-5602-4090-95cd-203852cda57c\") " pod="ai-tenant-e2e-inf-a-a17037/test-model-a17037-kserve-867b8d7d99-nrglc" Jul 07 19:33:07.077139 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:33:07.077134 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8fcfb0d5-5602-4090-95cd-203852cda57c-dshm\") pod \"test-model-a17037-kserve-867b8d7d99-nrglc\" (UID: \"8fcfb0d5-5602-4090-95cd-203852cda57c\") " pod="ai-tenant-e2e-inf-a-a17037/test-model-a17037-kserve-867b8d7d99-nrglc" Jul 07 19:33:07.077411 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:33:07.077164 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8fcfb0d5-5602-4090-95cd-203852cda57c-model-cache\") pod \"test-model-a17037-kserve-867b8d7d99-nrglc\" (UID: \"8fcfb0d5-5602-4090-95cd-203852cda57c\") " pod="ai-tenant-e2e-inf-a-a17037/test-model-a17037-kserve-867b8d7d99-nrglc" Jul 07 19:33:07.077411 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:33:07.077286 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9dmt\" (UniqueName: \"kubernetes.io/projected/8fcfb0d5-5602-4090-95cd-203852cda57c-kube-api-access-x9dmt\") pod \"test-model-a17037-kserve-867b8d7d99-nrglc\" (UID: \"8fcfb0d5-5602-4090-95cd-203852cda57c\") " pod="ai-tenant-e2e-inf-a-a17037/test-model-a17037-kserve-867b8d7d99-nrglc" Jul 07 19:33:07.077411 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:33:07.077372 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8fcfb0d5-5602-4090-95cd-203852cda57c-home\") pod \"test-model-a17037-kserve-867b8d7d99-nrglc\" (UID: \"8fcfb0d5-5602-4090-95cd-203852cda57c\") " pod="ai-tenant-e2e-inf-a-a17037/test-model-a17037-kserve-867b8d7d99-nrglc" Jul 07 19:33:07.077578 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:33:07.077550 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8fcfb0d5-5602-4090-95cd-203852cda57c-tmp-dir\") pod \"test-model-a17037-kserve-867b8d7d99-nrglc\" (UID: \"8fcfb0d5-5602-4090-95cd-203852cda57c\") " pod="ai-tenant-e2e-inf-a-a17037/test-model-a17037-kserve-867b8d7d99-nrglc" Jul 07 19:33:07.077785 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:33:07.077764 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8fcfb0d5-5602-4090-95cd-203852cda57c-home\") pod \"test-model-a17037-kserve-867b8d7d99-nrglc\" (UID: \"8fcfb0d5-5602-4090-95cd-203852cda57c\") " pod="ai-tenant-e2e-inf-a-a17037/test-model-a17037-kserve-867b8d7d99-nrglc" Jul 07 19:33:07.078148 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:33:07.078128 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8fcfb0d5-5602-4090-95cd-203852cda57c-model-cache\") pod \"test-model-a17037-kserve-867b8d7d99-nrglc\" (UID: \"8fcfb0d5-5602-4090-95cd-203852cda57c\") " pod="ai-tenant-e2e-inf-a-a17037/test-model-a17037-kserve-867b8d7d99-nrglc" Jul 07 19:33:07.079378 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:33:07.079354 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8fcfb0d5-5602-4090-95cd-203852cda57c-dshm\") pod \"test-model-a17037-kserve-867b8d7d99-nrglc\" (UID: \"8fcfb0d5-5602-4090-95cd-203852cda57c\") " pod="ai-tenant-e2e-inf-a-a17037/test-model-a17037-kserve-867b8d7d99-nrglc" Jul 07 19:33:07.079680 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:33:07.079664 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8fcfb0d5-5602-4090-95cd-203852cda57c-tls-certs\") pod \"test-model-a17037-kserve-867b8d7d99-nrglc\" (UID: \"8fcfb0d5-5602-4090-95cd-203852cda57c\") " pod="ai-tenant-e2e-inf-a-a17037/test-model-a17037-kserve-867b8d7d99-nrglc" Jul 07 19:33:07.089414 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:33:07.089388 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9dmt\" (UniqueName: \"kubernetes.io/projected/8fcfb0d5-5602-4090-95cd-203852cda57c-kube-api-access-x9dmt\") pod \"test-model-a17037-kserve-867b8d7d99-nrglc\" (UID: \"8fcfb0d5-5602-4090-95cd-203852cda57c\") " pod="ai-tenant-e2e-inf-a-a17037/test-model-a17037-kserve-867b8d7d99-nrglc" Jul 07 19:33:07.101928 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:33:07.101904 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-a17037/test-model-a17037-kserve-867b8d7d99-nrglc" Jul 07 19:33:07.247238 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:33:07.247202 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-a17037/test-model-a17037-kserve-867b8d7d99-nrglc"] Jul 07 19:33:07.250556 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:33:07.250526 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8fcfb0d5_5602_4090_95cd_203852cda57c.slice/crio-e663f98698ec733741bdeda5a60aa839a43bd2e2b67a8cb339a2ce46a7a9ef33 WatchSource:0}: Error finding container e663f98698ec733741bdeda5a60aa839a43bd2e2b67a8cb339a2ce46a7a9ef33: Status 404 returned error can't find the container with id e663f98698ec733741bdeda5a60aa839a43bd2e2b67a8cb339a2ce46a7a9ef33 Jul 07 19:33:08.025336 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:33:08.025299 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-a17037/test-model-a17037-kserve-867b8d7d99-nrglc" event={"ID":"8fcfb0d5-5602-4090-95cd-203852cda57c","Type":"ContainerStarted","Data":"745fc6401f6561652f14828f476842b93d32bbe0a4d8a2801df1cf212c544323"} Jul 07 19:33:08.025724 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:33:08.025343 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-a17037/test-model-a17037-kserve-867b8d7d99-nrglc" event={"ID":"8fcfb0d5-5602-4090-95cd-203852cda57c","Type":"ContainerStarted","Data":"e663f98698ec733741bdeda5a60aa839a43bd2e2b67a8cb339a2ce46a7a9ef33"} Jul 07 19:33:08.046091 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:33:08.046039 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-a17037/test-model-a17037-kserve-867b8d7d99-nrglc" podStartSLOduration=2.046024503 podStartE2EDuration="2.046024503s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:33:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:33:08.044425607 +0000 UTC m=+1971.060429790" watchObservedRunningTime="2026-07-07 19:33:08.046024503 +0000 UTC m=+1971.062028683" Jul 07 19:33:17.102802 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:33:17.102727 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-a17037/test-model-a17037-kserve-867b8d7d99-nrglc" Jul 07 19:33:17.102802 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:33:17.102772 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-a17037/test-model-a17037-kserve-867b8d7d99-nrglc" Jul 07 19:33:17.115372 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:33:17.115345 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-a17037/test-model-a17037-kserve-867b8d7d99-nrglc" Jul 07 19:33:18.089415 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:33:18.089381 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-a17037/test-model-a17037-kserve-867b8d7d99-nrglc" Jul 07 19:33:29.287489 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:33:29.287458 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:33:44.137635 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:33:44.137601 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:33:50.432235 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:33:50.432195 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-a17037/test-model-a17037-kserve-867b8d7d99-nrglc"] Jul 07 19:33:50.459963 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:33:50.459891 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:33:51.371528 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:33:51.371491 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:33:51.700779 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:33:51.700683 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:33:51.948731 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:33:51.948696 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:33:52.077449 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:33:52.077407 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:33:52.177423 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:33:52.177388 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:33:52.358163 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:33:52.358064 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:33:57.017424 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:33:57.017397 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:34:05.453108 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:34:05.453044 2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-a17037/test-model-a17037-kserve-867b8d7d99-nrglc" podUID="8fcfb0d5-5602-4090-95cd-203852cda57c" containerName="main" containerID="cri-o://745fc6401f6561652f14828f476842b93d32bbe0a4d8a2801df1cf212c544323" gracePeriod=45 Jul 07 19:34:05.706270 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:34:05.706203 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-a17037/test-model-a17037-kserve-867b8d7d99-nrglc" Jul 07 19:34:05.752804 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:34:05.752762 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8fcfb0d5-5602-4090-95cd-203852cda57c-tls-certs\") pod \"8fcfb0d5-5602-4090-95cd-203852cda57c\" (UID: \"8fcfb0d5-5602-4090-95cd-203852cda57c\") " Jul 07 19:34:05.752994 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:34:05.752827 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8fcfb0d5-5602-4090-95cd-203852cda57c-model-cache\") pod \"8fcfb0d5-5602-4090-95cd-203852cda57c\" (UID: \"8fcfb0d5-5602-4090-95cd-203852cda57c\") " Jul 07 19:34:05.752994 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:34:05.752977 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8fcfb0d5-5602-4090-95cd-203852cda57c-home\") pod \"8fcfb0d5-5602-4090-95cd-203852cda57c\" (UID: \"8fcfb0d5-5602-4090-95cd-203852cda57c\") " Jul 07 19:34:05.753125 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:34:05.753044 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8fcfb0d5-5602-4090-95cd-203852cda57c-dshm\") pod \"8fcfb0d5-5602-4090-95cd-203852cda57c\" (UID: \"8fcfb0d5-5602-4090-95cd-203852cda57c\") " Jul 07 19:34:05.753125 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:34:05.753093 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fcfb0d5-5602-4090-95cd-203852cda57c-model-cache" (OuterVolumeSpecName: "model-cache") pod "8fcfb0d5-5602-4090-95cd-203852cda57c" (UID: "8fcfb0d5-5602-4090-95cd-203852cda57c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:34:05.753234 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:34:05.753098 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8fcfb0d5-5602-4090-95cd-203852cda57c-tmp-dir\") pod \"8fcfb0d5-5602-4090-95cd-203852cda57c\" (UID: \"8fcfb0d5-5602-4090-95cd-203852cda57c\") " Jul 07 19:34:05.753234 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:34:05.753174 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x9dmt\" (UniqueName: \"kubernetes.io/projected/8fcfb0d5-5602-4090-95cd-203852cda57c-kube-api-access-x9dmt\") pod \"8fcfb0d5-5602-4090-95cd-203852cda57c\" (UID: \"8fcfb0d5-5602-4090-95cd-203852cda57c\") " Jul 07 19:34:05.753343 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:34:05.753265 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fcfb0d5-5602-4090-95cd-203852cda57c-home" (OuterVolumeSpecName: "home") pod "8fcfb0d5-5602-4090-95cd-203852cda57c" (UID: "8fcfb0d5-5602-4090-95cd-203852cda57c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:34:05.753343 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:34:05.753329 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fcfb0d5-5602-4090-95cd-203852cda57c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8fcfb0d5-5602-4090-95cd-203852cda57c" (UID: "8fcfb0d5-5602-4090-95cd-203852cda57c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:34:05.753601 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:34:05.753580 2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8fcfb0d5-5602-4090-95cd-203852cda57c-model-cache\") on node \"ip-10-0-142-134.ec2.internal\" DevicePath \"\"" Jul 07 19:34:05.753601 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:34:05.753603 2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8fcfb0d5-5602-4090-95cd-203852cda57c-home\") on node \"ip-10-0-142-134.ec2.internal\" DevicePath \"\"" Jul 07 19:34:05.753740 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:34:05.753612 2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8fcfb0d5-5602-4090-95cd-203852cda57c-tmp-dir\") on node \"ip-10-0-142-134.ec2.internal\" DevicePath \"\"" Jul 07 19:34:05.755199 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:34:05.755174 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fcfb0d5-5602-4090-95cd-203852cda57c-dshm" (OuterVolumeSpecName: "dshm") pod "8fcfb0d5-5602-4090-95cd-203852cda57c" (UID: "8fcfb0d5-5602-4090-95cd-203852cda57c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:34:05.755199 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:34:05.755184 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8fcfb0d5-5602-4090-95cd-203852cda57c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8fcfb0d5-5602-4090-95cd-203852cda57c" (UID: "8fcfb0d5-5602-4090-95cd-203852cda57c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 19:34:05.755515 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:34:05.755498 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fcfb0d5-5602-4090-95cd-203852cda57c-kube-api-access-x9dmt" (OuterVolumeSpecName: "kube-api-access-x9dmt") pod "8fcfb0d5-5602-4090-95cd-203852cda57c" (UID: "8fcfb0d5-5602-4090-95cd-203852cda57c"). InnerVolumeSpecName "kube-api-access-x9dmt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 19:34:05.854243 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:34:05.854204 2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8fcfb0d5-5602-4090-95cd-203852cda57c-dshm\") on node \"ip-10-0-142-134.ec2.internal\" DevicePath \"\"" Jul 07 19:34:05.854243 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:34:05.854246 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x9dmt\" (UniqueName: \"kubernetes.io/projected/8fcfb0d5-5602-4090-95cd-203852cda57c-kube-api-access-x9dmt\") on node \"ip-10-0-142-134.ec2.internal\" DevicePath \"\"" Jul 07 19:34:05.854455 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:34:05.854259 2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8fcfb0d5-5602-4090-95cd-203852cda57c-tls-certs\") on node \"ip-10-0-142-134.ec2.internal\" DevicePath \"\"" Jul 07 19:34:06.327602 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:34:06.327565 2563 generic.go:356] "Generic (PLEG): container finished" podID="8fcfb0d5-5602-4090-95cd-203852cda57c" containerID="745fc6401f6561652f14828f476842b93d32bbe0a4d8a2801df1cf212c544323" exitCode=0 Jul 07 19:34:06.327795 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:34:06.327615 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-a17037/test-model-a17037-kserve-867b8d7d99-nrglc" Jul 07 19:34:06.327795 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:34:06.327653 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-a17037/test-model-a17037-kserve-867b8d7d99-nrglc" event={"ID":"8fcfb0d5-5602-4090-95cd-203852cda57c","Type":"ContainerDied","Data":"745fc6401f6561652f14828f476842b93d32bbe0a4d8a2801df1cf212c544323"} Jul 07 19:34:06.327795 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:34:06.327697 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-a17037/test-model-a17037-kserve-867b8d7d99-nrglc" event={"ID":"8fcfb0d5-5602-4090-95cd-203852cda57c","Type":"ContainerDied","Data":"e663f98698ec733741bdeda5a60aa839a43bd2e2b67a8cb339a2ce46a7a9ef33"} Jul 07 19:34:06.327795 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:34:06.327719 2563 scope.go:117] "RemoveContainer" containerID="745fc6401f6561652f14828f476842b93d32bbe0a4d8a2801df1cf212c544323" Jul 07 19:34:06.336833 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:34:06.336812 2563 scope.go:117] "RemoveContainer" containerID="745fc6401f6561652f14828f476842b93d32bbe0a4d8a2801df1cf212c544323" Jul 07 19:34:06.337131 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:34:06.337113 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"745fc6401f6561652f14828f476842b93d32bbe0a4d8a2801df1cf212c544323\": container with ID starting with 745fc6401f6561652f14828f476842b93d32bbe0a4d8a2801df1cf212c544323 not found: ID does not exist" containerID="745fc6401f6561652f14828f476842b93d32bbe0a4d8a2801df1cf212c544323" Jul 07 19:34:06.337203 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:34:06.337139 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"745fc6401f6561652f14828f476842b93d32bbe0a4d8a2801df1cf212c544323"} err="failed to get container status \"745fc6401f6561652f14828f476842b93d32bbe0a4d8a2801df1cf212c544323\": rpc error: code = NotFound desc = could not find container \"745fc6401f6561652f14828f476842b93d32bbe0a4d8a2801df1cf212c544323\": container with ID starting with 745fc6401f6561652f14828f476842b93d32bbe0a4d8a2801df1cf212c544323 not found: ID does not exist" Jul 07 19:34:06.355742 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:34:06.355708 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-a17037/test-model-a17037-kserve-867b8d7d99-nrglc"] Jul 07 19:34:06.363686 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:34:06.363660 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-a17037/test-model-a17037-kserve-867b8d7d99-nrglc"] Jul 07 19:34:07.476858 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:34:07.476821 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8fcfb0d5-5602-4090-95cd-203852cda57c" path="/var/lib/kubelet/pods/8fcfb0d5-5602-4090-95cd-203852cda57c/volumes" Jul 07 19:34:26.770805 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:34:26.770762 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:34:26.869469 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:34:26.869427 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:34:34.557608 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:34:34.557570 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:34:34.785327 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:34:34.785293 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-a17037-openshift-default-5df47d4cc5-nwn4s"] Jul 07 19:34:34.785561 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:34:34.785513 2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-a17037-openshift-default-5df47d4cc5-nwn4s" podUID="9007ca76-b883-45e8-a276-22feea095a05" containerName="istio-proxy" containerID="cri-o://24558864156ba0272aa853ea8a044198e6ccbcc940a78665c337c9130e870c22" gracePeriod=30 Jul 07 19:34:34.812736 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:34:34.812656 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:34:35.040307 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:34:35.040270 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:34:35.220841 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:34:35.220752 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:34:35.636216 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:34:35.636180 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:34:35.732543 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:34:35.732497 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:34:35.833875 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:34:35.833842 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:34:35.995745 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:34:35.995659 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:34:37.270194 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:34:37.270152 2563 patch_prober.go:28] interesting pod/e2e-inf-a-a17037-openshift-default-5df47d4cc5-nwn4s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 07 19:34:37.270660 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:34:37.270232 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-a17037-openshift-default-5df47d4cc5-nwn4s" podUID="9007ca76-b883-45e8-a276-22feea095a05" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 07 19:34:40.054963 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:34:40.054881 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a17037-openshift-default-5df47d4cc5-nwn4s" Jul 07 19:34:40.085320 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:34:40.085282 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9007ca76-b883-45e8-a276-22feea095a05-istio-podinfo\") pod \"9007ca76-b883-45e8-a276-22feea095a05\" (UID: \"9007ca76-b883-45e8-a276-22feea095a05\") " Jul 07 19:34:40.085489 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:34:40.085330 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9007ca76-b883-45e8-a276-22feea095a05-credential-socket\") pod \"9007ca76-b883-45e8-a276-22feea095a05\" (UID: \"9007ca76-b883-45e8-a276-22feea095a05\") " Jul 07 19:34:40.085489 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:34:40.085356 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9007ca76-b883-45e8-a276-22feea095a05-istio-token\") pod \"9007ca76-b883-45e8-a276-22feea095a05\" (UID: \"9007ca76-b883-45e8-a276-22feea095a05\") " Jul 07 19:34:40.085489 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:34:40.085418 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9007ca76-b883-45e8-a276-22feea095a05-workload-socket\") pod \"9007ca76-b883-45e8-a276-22feea095a05\" (UID: \"9007ca76-b883-45e8-a276-22feea095a05\") " Jul 07 19:34:40.085665 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:34:40.085501 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9x6wb\" (UniqueName: \"kubernetes.io/projected/9007ca76-b883-45e8-a276-22feea095a05-kube-api-access-9x6wb\") pod \"9007ca76-b883-45e8-a276-22feea095a05\" (UID: \"9007ca76-b883-45e8-a276-22feea095a05\") " Jul 07 19:34:40.085665 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:34:40.085532 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9007ca76-b883-45e8-a276-22feea095a05-istio-data\") pod \"9007ca76-b883-45e8-a276-22feea095a05\" (UID: \"9007ca76-b883-45e8-a276-22feea095a05\") " Jul 07 19:34:40.085665 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:34:40.085560 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9007ca76-b883-45e8-a276-22feea095a05-workload-certs\") pod \"9007ca76-b883-45e8-a276-22feea095a05\" (UID: \"9007ca76-b883-45e8-a276-22feea095a05\") " Jul 07 19:34:40.085665 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:34:40.085610 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9007ca76-b883-45e8-a276-22feea095a05-istio-envoy\") pod \"9007ca76-b883-45e8-a276-22feea095a05\" (UID: \"9007ca76-b883-45e8-a276-22feea095a05\") " Jul 07 19:34:40.085665 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:34:40.085635 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9007ca76-b883-45e8-a276-22feea095a05-istiod-ca-cert\") pod \"9007ca76-b883-45e8-a276-22feea095a05\" (UID: \"9007ca76-b883-45e8-a276-22feea095a05\") " Jul 07 19:34:40.085926 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:34:40.085698 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9007ca76-b883-45e8-a276-22feea095a05-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "9007ca76-b883-45e8-a276-22feea095a05" (UID: "9007ca76-b883-45e8-a276-22feea095a05"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:34:40.085926 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:34:40.085763 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9007ca76-b883-45e8-a276-22feea095a05-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "9007ca76-b883-45e8-a276-22feea095a05" (UID: "9007ca76-b883-45e8-a276-22feea095a05"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:34:40.086062 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:34:40.086031 2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9007ca76-b883-45e8-a276-22feea095a05-workload-socket\") on node \"ip-10-0-142-134.ec2.internal\" DevicePath \"\"" Jul 07 19:34:40.086062 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:34:40.086050 2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9007ca76-b883-45e8-a276-22feea095a05-credential-socket\") on node \"ip-10-0-142-134.ec2.internal\" DevicePath \"\"" Jul 07 19:34:40.086171 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:34:40.086103 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9007ca76-b883-45e8-a276-22feea095a05-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "9007ca76-b883-45e8-a276-22feea095a05" (UID: "9007ca76-b883-45e8-a276-22feea095a05"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 19:34:40.086593 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:34:40.086542 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9007ca76-b883-45e8-a276-22feea095a05-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "9007ca76-b883-45e8-a276-22feea095a05" (UID: "9007ca76-b883-45e8-a276-22feea095a05"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:34:40.086735 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:34:40.086664 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9007ca76-b883-45e8-a276-22feea095a05-istio-data" (OuterVolumeSpecName: "istio-data") pod "9007ca76-b883-45e8-a276-22feea095a05" (UID: "9007ca76-b883-45e8-a276-22feea095a05"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:34:40.087729 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:34:40.087681 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9007ca76-b883-45e8-a276-22feea095a05-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "9007ca76-b883-45e8-a276-22feea095a05" (UID: "9007ca76-b883-45e8-a276-22feea095a05"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 07 19:34:40.087729 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:34:40.087692 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9007ca76-b883-45e8-a276-22feea095a05-istio-token" (OuterVolumeSpecName: "istio-token") pod "9007ca76-b883-45e8-a276-22feea095a05" (UID: "9007ca76-b883-45e8-a276-22feea095a05"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 19:34:40.088204 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:34:40.088178 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9007ca76-b883-45e8-a276-22feea095a05-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "9007ca76-b883-45e8-a276-22feea095a05" (UID: "9007ca76-b883-45e8-a276-22feea095a05"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:34:40.088204 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:34:40.088190 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9007ca76-b883-45e8-a276-22feea095a05-kube-api-access-9x6wb" (OuterVolumeSpecName: "kube-api-access-9x6wb") pod "9007ca76-b883-45e8-a276-22feea095a05" (UID: "9007ca76-b883-45e8-a276-22feea095a05"). InnerVolumeSpecName "kube-api-access-9x6wb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 19:34:40.186524 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:34:40.186480 2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9007ca76-b883-45e8-a276-22feea095a05-istio-podinfo\") on node \"ip-10-0-142-134.ec2.internal\" DevicePath \"\"" Jul 07 19:34:40.186524 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:34:40.186515 2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9007ca76-b883-45e8-a276-22feea095a05-istio-token\") on node \"ip-10-0-142-134.ec2.internal\" DevicePath \"\"" Jul 07 19:34:40.186524 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:34:40.186524 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9x6wb\" (UniqueName: \"kubernetes.io/projected/9007ca76-b883-45e8-a276-22feea095a05-kube-api-access-9x6wb\") on node \"ip-10-0-142-134.ec2.internal\" DevicePath \"\"" Jul 07 19:34:40.186524 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:34:40.186532 2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9007ca76-b883-45e8-a276-22feea095a05-istio-data\") on node \"ip-10-0-142-134.ec2.internal\" DevicePath \"\"" Jul 07 19:34:40.186802 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:34:40.186543 2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9007ca76-b883-45e8-a276-22feea095a05-workload-certs\") on node \"ip-10-0-142-134.ec2.internal\" DevicePath \"\"" Jul 07 19:34:40.186802 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:34:40.186551 2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9007ca76-b883-45e8-a276-22feea095a05-istio-envoy\") on node \"ip-10-0-142-134.ec2.internal\" DevicePath \"\"" Jul 07 19:34:40.186802 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:34:40.186558 2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9007ca76-b883-45e8-a276-22feea095a05-istiod-ca-cert\") on node \"ip-10-0-142-134.ec2.internal\" DevicePath \"\"" Jul 07 19:34:40.501903 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:34:40.501816 2563 generic.go:356] "Generic (PLEG): container finished" podID="9007ca76-b883-45e8-a276-22feea095a05" containerID="24558864156ba0272aa853ea8a044198e6ccbcc940a78665c337c9130e870c22" exitCode=0 Jul 07 19:34:40.501903 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:34:40.501859 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a17037-openshift-default-5df47d4cc5-nwn4s" Jul 07 19:34:40.502138 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:34:40.501902 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a17037-openshift-default-5df47d4cc5-nwn4s" event={"ID":"9007ca76-b883-45e8-a276-22feea095a05","Type":"ContainerDied","Data":"24558864156ba0272aa853ea8a044198e6ccbcc940a78665c337c9130e870c22"} Jul 07 19:34:40.502138 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:34:40.501981 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a17037-openshift-default-5df47d4cc5-nwn4s" event={"ID":"9007ca76-b883-45e8-a276-22feea095a05","Type":"ContainerDied","Data":"0755a4ee4a3158afca9fa8e7af353b7f72b4e684ed0d281d52e19668ba947f1c"} Jul 07 19:34:40.502138 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:34:40.502007 2563 scope.go:117] "RemoveContainer" containerID="24558864156ba0272aa853ea8a044198e6ccbcc940a78665c337c9130e870c22" Jul 07 19:34:40.511862 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:34:40.511844 2563 scope.go:117] "RemoveContainer" containerID="0677eb4b0cedcdd8413a1ac68780655e0808eb2b496dab3586c84a9998c6028f" Jul 07 19:34:40.522194 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:34:40.521296 2563 scope.go:117] "RemoveContainer" containerID="24558864156ba0272aa853ea8a044198e6ccbcc940a78665c337c9130e870c22" Jul 07 19:34:40.522194 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:34:40.521983 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"24558864156ba0272aa853ea8a044198e6ccbcc940a78665c337c9130e870c22\": container with ID starting with 24558864156ba0272aa853ea8a044198e6ccbcc940a78665c337c9130e870c22 not found: ID does not exist" containerID="24558864156ba0272aa853ea8a044198e6ccbcc940a78665c337c9130e870c22" Jul 07 19:34:40.522194 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:34:40.522017 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24558864156ba0272aa853ea8a044198e6ccbcc940a78665c337c9130e870c22"} err="failed to get container status \"24558864156ba0272aa853ea8a044198e6ccbcc940a78665c337c9130e870c22\": rpc error: code = NotFound desc = could not find container \"24558864156ba0272aa853ea8a044198e6ccbcc940a78665c337c9130e870c22\": container with ID starting with 24558864156ba0272aa853ea8a044198e6ccbcc940a78665c337c9130e870c22 not found: ID does not exist" Jul 07 19:34:40.522194 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:34:40.522039 2563 scope.go:117] "RemoveContainer" containerID="0677eb4b0cedcdd8413a1ac68780655e0808eb2b496dab3586c84a9998c6028f" Jul 07 19:34:40.522626 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:34:40.522579 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0677eb4b0cedcdd8413a1ac68780655e0808eb2b496dab3586c84a9998c6028f\": container with ID starting with 0677eb4b0cedcdd8413a1ac68780655e0808eb2b496dab3586c84a9998c6028f not found: ID does not exist" containerID="0677eb4b0cedcdd8413a1ac68780655e0808eb2b496dab3586c84a9998c6028f" Jul 07 19:34:40.522626 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:34:40.522615 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0677eb4b0cedcdd8413a1ac68780655e0808eb2b496dab3586c84a9998c6028f"} err="failed to get container status \"0677eb4b0cedcdd8413a1ac68780655e0808eb2b496dab3586c84a9998c6028f\": rpc error: code = NotFound desc = could not find container \"0677eb4b0cedcdd8413a1ac68780655e0808eb2b496dab3586c84a9998c6028f\": container with ID starting with 0677eb4b0cedcdd8413a1ac68780655e0808eb2b496dab3586c84a9998c6028f not found: ID does not exist" Jul 07 19:34:40.524393 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:34:40.524370 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-a17037-openshift-default-5df47d4cc5-nwn4s"] Jul 07 19:34:40.531025 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:34:40.530997 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-a17037-openshift-default-5df47d4cc5-nwn4s"] Jul 07 19:34:41.477591 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:34:41.477557 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9007ca76-b883-45e8-a276-22feea095a05" path="/var/lib/kubelet/pods/9007ca76-b883-45e8-a276-22feea095a05/volumes" Jul 07 19:34:42.027372 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:34:42.027338 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:34:42.101295 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:34:42.101255 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:34:42.295512 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:34:42.295488 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:34:42.518415 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:34:42.518371 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:34:42.856036 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:34:42.856003 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:34:43.149082 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:34:43.149007 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-2e6bb8-openshift-default-5c59fbb68f-n4n8h"] Jul 07 19:34:43.149389 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:34:43.149378 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9007ca76-b883-45e8-a276-22feea095a05" containerName="istio-proxy" Jul 07 19:34:43.149439 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:34:43.149391 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="9007ca76-b883-45e8-a276-22feea095a05" containerName="istio-proxy" Jul 07 19:34:43.149439 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:34:43.149420 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8fcfb0d5-5602-4090-95cd-203852cda57c" containerName="main" Jul 07 19:34:43.149439 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:34:43.149425 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="8fcfb0d5-5602-4090-95cd-203852cda57c" containerName="main" Jul 07 19:34:43.149544 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:34:43.149485 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="8fcfb0d5-5602-4090-95cd-203852cda57c" containerName="main" Jul 07 19:34:43.149544 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:34:43.149494 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="9007ca76-b883-45e8-a276-22feea095a05" containerName="istio-proxy" Jul 07 19:34:43.154216 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:34:43.154197 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-2e6bb8-openshift-default-5c59fbb68f-n4n8h" Jul 07 19:34:43.156224 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:34:43.156202 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-2e6bb8-openshift-default-dockercfg-2h92g\"" Jul 07 19:34:43.166093 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:34:43.166074 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-2e6bb8-openshift-default-5c59fbb68f-n4n8h"] Jul 07 19:34:43.185143 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:34:43.185117 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:34:43.216021 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:34:43.215989 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3132cd7e-44ee-4423-af07-b4bffe6ccdd6-istio-token\") pod \"e2e-api-a-2e6bb8-openshift-default-5c59fbb68f-n4n8h\" (UID: \"3132cd7e-44ee-4423-af07-b4bffe6ccdd6\") " pod="openshift-ingress/e2e-api-a-2e6bb8-openshift-default-5c59fbb68f-n4n8h" Jul 07 19:34:43.216160 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:34:43.216033 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3132cd7e-44ee-4423-af07-b4bffe6ccdd6-istiod-ca-cert\") pod \"e2e-api-a-2e6bb8-openshift-default-5c59fbb68f-n4n8h\" (UID: \"3132cd7e-44ee-4423-af07-b4bffe6ccdd6\") " pod="openshift-ingress/e2e-api-a-2e6bb8-openshift-default-5c59fbb68f-n4n8h" Jul 07 19:34:43.216160 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:34:43.216055 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3132cd7e-44ee-4423-af07-b4bffe6ccdd6-istio-podinfo\") pod \"e2e-api-a-2e6bb8-openshift-default-5c59fbb68f-n4n8h\" (UID: \"3132cd7e-44ee-4423-af07-b4bffe6ccdd6\") " pod="openshift-ingress/e2e-api-a-2e6bb8-openshift-default-5c59fbb68f-n4n8h" Jul 07 19:34:43.216160 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:34:43.216134 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3132cd7e-44ee-4423-af07-b4bffe6ccdd6-credential-socket\") pod \"e2e-api-a-2e6bb8-openshift-default-5c59fbb68f-n4n8h\" (UID: \"3132cd7e-44ee-4423-af07-b4bffe6ccdd6\") " pod="openshift-ingress/e2e-api-a-2e6bb8-openshift-default-5c59fbb68f-n4n8h" Jul 07 19:34:43.216280 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:34:43.216161 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3132cd7e-44ee-4423-af07-b4bffe6ccdd6-istio-envoy\") pod \"e2e-api-a-2e6bb8-openshift-default-5c59fbb68f-n4n8h\" (UID: \"3132cd7e-44ee-4423-af07-b4bffe6ccdd6\") " pod="openshift-ingress/e2e-api-a-2e6bb8-openshift-default-5c59fbb68f-n4n8h" Jul 07 19:34:43.216280 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:34:43.216208 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3132cd7e-44ee-4423-af07-b4bffe6ccdd6-workload-socket\") pod \"e2e-api-a-2e6bb8-openshift-default-5c59fbb68f-n4n8h\" (UID: \"3132cd7e-44ee-4423-af07-b4bffe6ccdd6\") " pod="openshift-ingress/e2e-api-a-2e6bb8-openshift-default-5c59fbb68f-n4n8h" Jul 07 19:34:43.216280 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:34:43.216239 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3132cd7e-44ee-4423-af07-b4bffe6ccdd6-istio-data\") pod \"e2e-api-a-2e6bb8-openshift-default-5c59fbb68f-n4n8h\" (UID: \"3132cd7e-44ee-4423-af07-b4bffe6ccdd6\") " pod="openshift-ingress/e2e-api-a-2e6bb8-openshift-default-5c59fbb68f-n4n8h" Jul 07 19:34:43.216280 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:34:43.216261 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbf8z\" (UniqueName: \"kubernetes.io/projected/3132cd7e-44ee-4423-af07-b4bffe6ccdd6-kube-api-access-sbf8z\") pod \"e2e-api-a-2e6bb8-openshift-default-5c59fbb68f-n4n8h\" (UID: \"3132cd7e-44ee-4423-af07-b4bffe6ccdd6\") " pod="openshift-ingress/e2e-api-a-2e6bb8-openshift-default-5c59fbb68f-n4n8h" Jul 07 19:34:43.216280 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:34:43.216277 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3132cd7e-44ee-4423-af07-b4bffe6ccdd6-workload-certs\") pod \"e2e-api-a-2e6bb8-openshift-default-5c59fbb68f-n4n8h\" (UID: \"3132cd7e-44ee-4423-af07-b4bffe6ccdd6\") " pod="openshift-ingress/e2e-api-a-2e6bb8-openshift-default-5c59fbb68f-n4n8h" Jul 07 19:34:43.298772 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:34:43.298740 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:34:43.317093 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:34:43.317062 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3132cd7e-44ee-4423-af07-b4bffe6ccdd6-istio-envoy\") pod \"e2e-api-a-2e6bb8-openshift-default-5c59fbb68f-n4n8h\" (UID: \"3132cd7e-44ee-4423-af07-b4bffe6ccdd6\") " pod="openshift-ingress/e2e-api-a-2e6bb8-openshift-default-5c59fbb68f-n4n8h" Jul 07 19:34:43.317238 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:34:43.317121 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3132cd7e-44ee-4423-af07-b4bffe6ccdd6-workload-socket\") pod \"e2e-api-a-2e6bb8-openshift-default-5c59fbb68f-n4n8h\" (UID: \"3132cd7e-44ee-4423-af07-b4bffe6ccdd6\") " pod="openshift-ingress/e2e-api-a-2e6bb8-openshift-default-5c59fbb68f-n4n8h" Jul 07 19:34:43.317238 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:34:43.317151 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3132cd7e-44ee-4423-af07-b4bffe6ccdd6-istio-data\") pod \"e2e-api-a-2e6bb8-openshift-default-5c59fbb68f-n4n8h\" (UID: \"3132cd7e-44ee-4423-af07-b4bffe6ccdd6\") " pod="openshift-ingress/e2e-api-a-2e6bb8-openshift-default-5c59fbb68f-n4n8h" Jul 07 19:34:43.317238 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:34:43.317176 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbf8z\" (UniqueName: \"kubernetes.io/projected/3132cd7e-44ee-4423-af07-b4bffe6ccdd6-kube-api-access-sbf8z\") pod \"e2e-api-a-2e6bb8-openshift-default-5c59fbb68f-n4n8h\" (UID: \"3132cd7e-44ee-4423-af07-b4bffe6ccdd6\") " pod="openshift-ingress/e2e-api-a-2e6bb8-openshift-default-5c59fbb68f-n4n8h" Jul 07 19:34:43.317238 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:34:43.317193 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3132cd7e-44ee-4423-af07-b4bffe6ccdd6-workload-certs\") pod \"e2e-api-a-2e6bb8-openshift-default-5c59fbb68f-n4n8h\" (UID: \"3132cd7e-44ee-4423-af07-b4bffe6ccdd6\") " pod="openshift-ingress/e2e-api-a-2e6bb8-openshift-default-5c59fbb68f-n4n8h" Jul 07 19:34:43.317238 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:34:43.317223 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3132cd7e-44ee-4423-af07-b4bffe6ccdd6-istio-token\") pod \"e2e-api-a-2e6bb8-openshift-default-5c59fbb68f-n4n8h\" (UID: \"3132cd7e-44ee-4423-af07-b4bffe6ccdd6\") " pod="openshift-ingress/e2e-api-a-2e6bb8-openshift-default-5c59fbb68f-n4n8h" Jul 07 19:34:43.317507 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:34:43.317254 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3132cd7e-44ee-4423-af07-b4bffe6ccdd6-istiod-ca-cert\") pod \"e2e-api-a-2e6bb8-openshift-default-5c59fbb68f-n4n8h\" (UID: \"3132cd7e-44ee-4423-af07-b4bffe6ccdd6\") " pod="openshift-ingress/e2e-api-a-2e6bb8-openshift-default-5c59fbb68f-n4n8h" Jul 07 19:34:43.317507 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:34:43.317283 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3132cd7e-44ee-4423-af07-b4bffe6ccdd6-istio-podinfo\") pod \"e2e-api-a-2e6bb8-openshift-default-5c59fbb68f-n4n8h\" (UID: \"3132cd7e-44ee-4423-af07-b4bffe6ccdd6\") " pod="openshift-ingress/e2e-api-a-2e6bb8-openshift-default-5c59fbb68f-n4n8h" Jul 07 19:34:43.317507 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:34:43.317359 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3132cd7e-44ee-4423-af07-b4bffe6ccdd6-credential-socket\") pod \"e2e-api-a-2e6bb8-openshift-default-5c59fbb68f-n4n8h\" (UID: \"3132cd7e-44ee-4423-af07-b4bffe6ccdd6\") " pod="openshift-ingress/e2e-api-a-2e6bb8-openshift-default-5c59fbb68f-n4n8h" Jul 07 19:34:43.317724 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:34:43.317693 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3132cd7e-44ee-4423-af07-b4bffe6ccdd6-istio-data\") pod \"e2e-api-a-2e6bb8-openshift-default-5c59fbb68f-n4n8h\" (UID: \"3132cd7e-44ee-4423-af07-b4bffe6ccdd6\") " pod="openshift-ingress/e2e-api-a-2e6bb8-openshift-default-5c59fbb68f-n4n8h" Jul 07 19:34:43.317869 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:34:43.317748 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3132cd7e-44ee-4423-af07-b4bffe6ccdd6-workload-certs\") pod \"e2e-api-a-2e6bb8-openshift-default-5c59fbb68f-n4n8h\" (UID: \"3132cd7e-44ee-4423-af07-b4bffe6ccdd6\") " pod="openshift-ingress/e2e-api-a-2e6bb8-openshift-default-5c59fbb68f-n4n8h" Jul 07 19:34:43.317869 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:34:43.317821 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3132cd7e-44ee-4423-af07-b4bffe6ccdd6-workload-socket\") pod \"e2e-api-a-2e6bb8-openshift-default-5c59fbb68f-n4n8h\" (UID: \"3132cd7e-44ee-4423-af07-b4bffe6ccdd6\") " pod="openshift-ingress/e2e-api-a-2e6bb8-openshift-default-5c59fbb68f-n4n8h" Jul 07 19:34:43.317869 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:34:43.317840 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3132cd7e-44ee-4423-af07-b4bffe6ccdd6-credential-socket\") pod \"e2e-api-a-2e6bb8-openshift-default-5c59fbb68f-n4n8h\" (UID: \"3132cd7e-44ee-4423-af07-b4bffe6ccdd6\") " pod="openshift-ingress/e2e-api-a-2e6bb8-openshift-default-5c59fbb68f-n4n8h" Jul 07 19:34:43.318059 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:34:43.318040 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3132cd7e-44ee-4423-af07-b4bffe6ccdd6-istiod-ca-cert\") pod \"e2e-api-a-2e6bb8-openshift-default-5c59fbb68f-n4n8h\" (UID: \"3132cd7e-44ee-4423-af07-b4bffe6ccdd6\") " pod="openshift-ingress/e2e-api-a-2e6bb8-openshift-default-5c59fbb68f-n4n8h" Jul 07 19:34:43.320087 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:34:43.320069 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3132cd7e-44ee-4423-af07-b4bffe6ccdd6-istio-envoy\") pod \"e2e-api-a-2e6bb8-openshift-default-5c59fbb68f-n4n8h\" (UID: \"3132cd7e-44ee-4423-af07-b4bffe6ccdd6\") " pod="openshift-ingress/e2e-api-a-2e6bb8-openshift-default-5c59fbb68f-n4n8h" Jul 07 19:34:43.320332 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:34:43.320314 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3132cd7e-44ee-4423-af07-b4bffe6ccdd6-istio-podinfo\") pod \"e2e-api-a-2e6bb8-openshift-default-5c59fbb68f-n4n8h\" (UID: \"3132cd7e-44ee-4423-af07-b4bffe6ccdd6\") " pod="openshift-ingress/e2e-api-a-2e6bb8-openshift-default-5c59fbb68f-n4n8h" Jul 07 19:34:43.326199 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:34:43.326176 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbf8z\" (UniqueName: \"kubernetes.io/projected/3132cd7e-44ee-4423-af07-b4bffe6ccdd6-kube-api-access-sbf8z\") pod \"e2e-api-a-2e6bb8-openshift-default-5c59fbb68f-n4n8h\" (UID: \"3132cd7e-44ee-4423-af07-b4bffe6ccdd6\") " pod="openshift-ingress/e2e-api-a-2e6bb8-openshift-default-5c59fbb68f-n4n8h" Jul 07 19:34:43.326556 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:34:43.326536 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3132cd7e-44ee-4423-af07-b4bffe6ccdd6-istio-token\") pod \"e2e-api-a-2e6bb8-openshift-default-5c59fbb68f-n4n8h\" (UID: \"3132cd7e-44ee-4423-af07-b4bffe6ccdd6\") " pod="openshift-ingress/e2e-api-a-2e6bb8-openshift-default-5c59fbb68f-n4n8h" Jul 07 19:34:43.464935 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:34:43.464840 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-2e6bb8-openshift-default-5c59fbb68f-n4n8h" Jul 07 19:34:43.610271 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:34:43.610248 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-2e6bb8-openshift-default-5c59fbb68f-n4n8h"] Jul 07 19:34:43.612900 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:34:43.612865 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3132cd7e_44ee_4423_af07_b4bffe6ccdd6.slice/crio-3199563223c862a3c2ed44af879e40a82940ed825169998414f2b69be52b691c WatchSource:0}: Error finding container 3199563223c862a3c2ed44af879e40a82940ed825169998414f2b69be52b691c: Status 404 returned error can't find the container with id 3199563223c862a3c2ed44af879e40a82940ed825169998414f2b69be52b691c Jul 07 19:34:43.615225 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:34:43.615191 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 07 19:34:43.615331 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:34:43.615258 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 07 19:34:43.615331 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:34:43.615285 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 07 19:34:44.526325 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:34:44.526291 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-2e6bb8-openshift-default-5c59fbb68f-n4n8h" event={"ID":"3132cd7e-44ee-4423-af07-b4bffe6ccdd6","Type":"ContainerStarted","Data":"362d52c84266b5a0312c3b3ae45a12445d8eaea763b272cc7d9d156145ba60e6"} Jul 07 19:34:44.526536 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:34:44.526346 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-2e6bb8-openshift-default-5c59fbb68f-n4n8h" event={"ID":"3132cd7e-44ee-4423-af07-b4bffe6ccdd6","Type":"ContainerStarted","Data":"3199563223c862a3c2ed44af879e40a82940ed825169998414f2b69be52b691c"} Jul 07 19:34:44.551807 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:34:44.551756 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-2e6bb8-openshift-default-5c59fbb68f-n4n8h" podStartSLOduration=1.551742044 podStartE2EDuration="1.551742044s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:34:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:34:44.549457237 +0000 UTC m=+2067.565461410" watchObservedRunningTime="2026-07-07 19:34:44.551742044 +0000 UTC m=+2067.567746225" Jul 07 19:34:45.466032 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:34:45.465990 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-2e6bb8-openshift-default-5c59fbb68f-n4n8h" Jul 07 19:34:45.478099 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:34:45.478072 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-2e6bb8-openshift-default-5c59fbb68f-n4n8h" Jul 07 19:34:45.531823 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:34:45.531793 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-2e6bb8-openshift-default-5c59fbb68f-n4n8h" Jul 07 19:34:45.532845 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:34:45.532824 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-2e6bb8-openshift-default-5c59fbb68f-n4n8h" Jul 07 19:34:46.488289 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:34:46.488256 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:34:46.588072 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:34:46.588041 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:34:46.693005 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:34:46.692972 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:34:46.898034 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:34:46.898004 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:34:47.297868 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:34:47.297840 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:34:47.431252 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:34:47.431219 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:34:50.990695 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:34:50.990660 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:34:51.461671 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:34:51.461638 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:34:51.752366 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:34:51.752270 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-89748e-openshift-default-794dd46c5f-qp8hk"] Jul 07 19:34:51.752687 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:34:51.752673 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9007ca76-b883-45e8-a276-22feea095a05" containerName="istio-proxy" Jul 07 19:34:51.752687 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:34:51.752687 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="9007ca76-b883-45e8-a276-22feea095a05" containerName="istio-proxy" Jul 07 19:34:51.752815 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:34:51.752748 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="9007ca76-b883-45e8-a276-22feea095a05" containerName="istio-proxy" Jul 07 19:34:51.755848 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:34:51.755827 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-89748e-openshift-default-794dd46c5f-qp8hk" Jul 07 19:34:51.758009 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:34:51.757988 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-89748e-openshift-default-dockercfg-mxlz6\"" Jul 07 19:34:51.779997 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:34:51.768649 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-89748e-openshift-default-794dd46c5f-qp8hk"] Jul 07 19:34:51.799405 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:34:51.799357 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:34:51.805569 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:34:51.805543 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d9ba960-8f2e-4920-b787-3da2276be176-credential-socket\") pod \"e2e-api-b-89748e-openshift-default-794dd46c5f-qp8hk\" (UID: \"8d9ba960-8f2e-4920-b787-3da2276be176\") " pod="openshift-ingress/e2e-api-b-89748e-openshift-default-794dd46c5f-qp8hk" Jul 07 19:34:51.805641 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:34:51.805577 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d9ba960-8f2e-4920-b787-3da2276be176-istio-podinfo\") pod \"e2e-api-b-89748e-openshift-default-794dd46c5f-qp8hk\" (UID: \"8d9ba960-8f2e-4920-b787-3da2276be176\") " pod="openshift-ingress/e2e-api-b-89748e-openshift-default-794dd46c5f-qp8hk" Jul 07 19:34:51.805641 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:34:51.805615 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tlbr6\" (UniqueName: \"kubernetes.io/projected/8d9ba960-8f2e-4920-b787-3da2276be176-kube-api-access-tlbr6\") pod \"e2e-api-b-89748e-openshift-default-794dd46c5f-qp8hk\" (UID: \"8d9ba960-8f2e-4920-b787-3da2276be176\") " pod="openshift-ingress/e2e-api-b-89748e-openshift-default-794dd46c5f-qp8hk" Jul 07 19:34:51.805715 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:34:51.805665 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d9ba960-8f2e-4920-b787-3da2276be176-workload-certs\") pod \"e2e-api-b-89748e-openshift-default-794dd46c5f-qp8hk\" (UID: \"8d9ba960-8f2e-4920-b787-3da2276be176\") " pod="openshift-ingress/e2e-api-b-89748e-openshift-default-794dd46c5f-qp8hk" Jul 07 19:34:51.805715 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:34:51.805686 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d9ba960-8f2e-4920-b787-3da2276be176-istio-envoy\") pod \"e2e-api-b-89748e-openshift-default-794dd46c5f-qp8hk\" (UID: \"8d9ba960-8f2e-4920-b787-3da2276be176\") " pod="openshift-ingress/e2e-api-b-89748e-openshift-default-794dd46c5f-qp8hk" Jul 07 19:34:51.805715 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:34:51.805713 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d9ba960-8f2e-4920-b787-3da2276be176-workload-socket\") pod \"e2e-api-b-89748e-openshift-default-794dd46c5f-qp8hk\" (UID: \"8d9ba960-8f2e-4920-b787-3da2276be176\") " pod="openshift-ingress/e2e-api-b-89748e-openshift-default-794dd46c5f-qp8hk" Jul 07 19:34:51.805850 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:34:51.805736 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d9ba960-8f2e-4920-b787-3da2276be176-istio-token\") pod \"e2e-api-b-89748e-openshift-default-794dd46c5f-qp8hk\" (UID: \"8d9ba960-8f2e-4920-b787-3da2276be176\") " pod="openshift-ingress/e2e-api-b-89748e-openshift-default-794dd46c5f-qp8hk" Jul 07 19:34:51.805850 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:34:51.805761 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d9ba960-8f2e-4920-b787-3da2276be176-istio-data\") pod \"e2e-api-b-89748e-openshift-default-794dd46c5f-qp8hk\" (UID: \"8d9ba960-8f2e-4920-b787-3da2276be176\") " pod="openshift-ingress/e2e-api-b-89748e-openshift-default-794dd46c5f-qp8hk" Jul 07 19:34:51.805850 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:34:51.805782 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d9ba960-8f2e-4920-b787-3da2276be176-istiod-ca-cert\") pod \"e2e-api-b-89748e-openshift-default-794dd46c5f-qp8hk\" (UID: \"8d9ba960-8f2e-4920-b787-3da2276be176\") " pod="openshift-ingress/e2e-api-b-89748e-openshift-default-794dd46c5f-qp8hk" Jul 07 19:34:51.906411 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:34:51.906374 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tlbr6\" (UniqueName: \"kubernetes.io/projected/8d9ba960-8f2e-4920-b787-3da2276be176-kube-api-access-tlbr6\") pod \"e2e-api-b-89748e-openshift-default-794dd46c5f-qp8hk\" (UID: \"8d9ba960-8f2e-4920-b787-3da2276be176\") " pod="openshift-ingress/e2e-api-b-89748e-openshift-default-794dd46c5f-qp8hk" Jul 07 19:34:51.906590 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:34:51.906437 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d9ba960-8f2e-4920-b787-3da2276be176-workload-certs\") pod \"e2e-api-b-89748e-openshift-default-794dd46c5f-qp8hk\" (UID: \"8d9ba960-8f2e-4920-b787-3da2276be176\") " pod="openshift-ingress/e2e-api-b-89748e-openshift-default-794dd46c5f-qp8hk" Jul 07 19:34:51.906590 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:34:51.906456 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d9ba960-8f2e-4920-b787-3da2276be176-istio-envoy\") pod \"e2e-api-b-89748e-openshift-default-794dd46c5f-qp8hk\" (UID: \"8d9ba960-8f2e-4920-b787-3da2276be176\") " pod="openshift-ingress/e2e-api-b-89748e-openshift-default-794dd46c5f-qp8hk" Jul 07 19:34:51.906590 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:34:51.906481 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d9ba960-8f2e-4920-b787-3da2276be176-workload-socket\") pod \"e2e-api-b-89748e-openshift-default-794dd46c5f-qp8hk\" (UID: \"8d9ba960-8f2e-4920-b787-3da2276be176\") " pod="openshift-ingress/e2e-api-b-89748e-openshift-default-794dd46c5f-qp8hk" Jul 07 19:34:51.906590 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:34:51.906515 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d9ba960-8f2e-4920-b787-3da2276be176-istio-token\") pod \"e2e-api-b-89748e-openshift-default-794dd46c5f-qp8hk\" (UID: \"8d9ba960-8f2e-4920-b787-3da2276be176\") " pod="openshift-ingress/e2e-api-b-89748e-openshift-default-794dd46c5f-qp8hk" Jul 07 19:34:51.906590 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:34:51.906546 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d9ba960-8f2e-4920-b787-3da2276be176-istio-data\") pod \"e2e-api-b-89748e-openshift-default-794dd46c5f-qp8hk\" (UID: \"8d9ba960-8f2e-4920-b787-3da2276be176\") " pod="openshift-ingress/e2e-api-b-89748e-openshift-default-794dd46c5f-qp8hk" Jul 07 19:34:51.906894 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:34:51.906667 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d9ba960-8f2e-4920-b787-3da2276be176-istiod-ca-cert\") pod \"e2e-api-b-89748e-openshift-default-794dd46c5f-qp8hk\" (UID: \"8d9ba960-8f2e-4920-b787-3da2276be176\") " pod="openshift-ingress/e2e-api-b-89748e-openshift-default-794dd46c5f-qp8hk" Jul 07 19:34:51.906894 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:34:51.906791 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d9ba960-8f2e-4920-b787-3da2276be176-credential-socket\") pod \"e2e-api-b-89748e-openshift-default-794dd46c5f-qp8hk\" (UID: \"8d9ba960-8f2e-4920-b787-3da2276be176\") " pod="openshift-ingress/e2e-api-b-89748e-openshift-default-794dd46c5f-qp8hk" Jul 07 19:34:51.906894 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:34:51.906818 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d9ba960-8f2e-4920-b787-3da2276be176-istio-podinfo\") pod \"e2e-api-b-89748e-openshift-default-794dd46c5f-qp8hk\" (UID: \"8d9ba960-8f2e-4920-b787-3da2276be176\") " pod="openshift-ingress/e2e-api-b-89748e-openshift-default-794dd46c5f-qp8hk" Jul 07 19:34:51.907100 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:34:51.906980 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d9ba960-8f2e-4920-b787-3da2276be176-istio-data\") pod \"e2e-api-b-89748e-openshift-default-794dd46c5f-qp8hk\" (UID: \"8d9ba960-8f2e-4920-b787-3da2276be176\") " pod="openshift-ingress/e2e-api-b-89748e-openshift-default-794dd46c5f-qp8hk" Jul 07 19:34:51.907160 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:34:51.907136 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d9ba960-8f2e-4920-b787-3da2276be176-credential-socket\") pod \"e2e-api-b-89748e-openshift-default-794dd46c5f-qp8hk\" (UID: \"8d9ba960-8f2e-4920-b787-3da2276be176\") " pod="openshift-ingress/e2e-api-b-89748e-openshift-default-794dd46c5f-qp8hk" Jul 07 19:34:51.907274 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:34:51.907202 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d9ba960-8f2e-4920-b787-3da2276be176-workload-socket\") pod \"e2e-api-b-89748e-openshift-default-794dd46c5f-qp8hk\" (UID: \"8d9ba960-8f2e-4920-b787-3da2276be176\") " pod="openshift-ingress/e2e-api-b-89748e-openshift-default-794dd46c5f-qp8hk" Jul 07 19:34:51.907539 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:34:51.907385 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d9ba960-8f2e-4920-b787-3da2276be176-istiod-ca-cert\") pod \"e2e-api-b-89748e-openshift-default-794dd46c5f-qp8hk\" (UID: \"8d9ba960-8f2e-4920-b787-3da2276be176\") " pod="openshift-ingress/e2e-api-b-89748e-openshift-default-794dd46c5f-qp8hk" Jul 07 19:34:51.907539 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:34:51.907427 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d9ba960-8f2e-4920-b787-3da2276be176-workload-certs\") pod \"e2e-api-b-89748e-openshift-default-794dd46c5f-qp8hk\" (UID: \"8d9ba960-8f2e-4920-b787-3da2276be176\") " pod="openshift-ingress/e2e-api-b-89748e-openshift-default-794dd46c5f-qp8hk" Jul 07 19:34:51.909429 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:34:51.909408 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d9ba960-8f2e-4920-b787-3da2276be176-istio-envoy\") pod \"e2e-api-b-89748e-openshift-default-794dd46c5f-qp8hk\" (UID: \"8d9ba960-8f2e-4920-b787-3da2276be176\") " pod="openshift-ingress/e2e-api-b-89748e-openshift-default-794dd46c5f-qp8hk" Jul 07 19:34:51.909668 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:34:51.909649 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d9ba960-8f2e-4920-b787-3da2276be176-istio-podinfo\") pod \"e2e-api-b-89748e-openshift-default-794dd46c5f-qp8hk\" (UID: \"8d9ba960-8f2e-4920-b787-3da2276be176\") " pod="openshift-ingress/e2e-api-b-89748e-openshift-default-794dd46c5f-qp8hk" Jul 07 19:34:51.912095 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:34:51.912062 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:34:51.914924 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:34:51.914890 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d9ba960-8f2e-4920-b787-3da2276be176-istio-token\") pod \"e2e-api-b-89748e-openshift-default-794dd46c5f-qp8hk\" (UID: \"8d9ba960-8f2e-4920-b787-3da2276be176\") " pod="openshift-ingress/e2e-api-b-89748e-openshift-default-794dd46c5f-qp8hk" Jul 07 19:34:51.915026 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:34:51.914989 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tlbr6\" (UniqueName: \"kubernetes.io/projected/8d9ba960-8f2e-4920-b787-3da2276be176-kube-api-access-tlbr6\") pod \"e2e-api-b-89748e-openshift-default-794dd46c5f-qp8hk\" (UID: \"8d9ba960-8f2e-4920-b787-3da2276be176\") " pod="openshift-ingress/e2e-api-b-89748e-openshift-default-794dd46c5f-qp8hk" Jul 07 19:34:52.097478 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:34:52.097428 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-89748e-openshift-default-794dd46c5f-qp8hk" Jul 07 19:34:52.285175 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:34:52.285130 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-89748e-openshift-default-794dd46c5f-qp8hk"] Jul 07 19:34:52.289220 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:34:52.289179 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d9ba960_8f2e_4920_b787_3da2276be176.slice/crio-9de5651b7a8b65b6097c9b8fb28b0e54e552c9cf3e86d7b02c5085968ffba698 WatchSource:0}: Error finding container 9de5651b7a8b65b6097c9b8fb28b0e54e552c9cf3e86d7b02c5085968ffba698: Status 404 returned error can't find the container with id 9de5651b7a8b65b6097c9b8fb28b0e54e552c9cf3e86d7b02c5085968ffba698 Jul 07 19:34:52.291686 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:34:52.291651 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 07 19:34:52.291795 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:34:52.291735 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 07 19:34:52.291795 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:34:52.291778 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 07 19:34:52.575164 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:34:52.575120 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-89748e-openshift-default-794dd46c5f-qp8hk" event={"ID":"8d9ba960-8f2e-4920-b787-3da2276be176","Type":"ContainerStarted","Data":"35c1b7e56a932f9b20f921ee6c4acd01cd967824151c965342a27c5cd35aab27"} Jul 07 19:34:52.575367 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:34:52.575194 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-89748e-openshift-default-794dd46c5f-qp8hk" event={"ID":"8d9ba960-8f2e-4920-b787-3da2276be176","Type":"ContainerStarted","Data":"9de5651b7a8b65b6097c9b8fb28b0e54e552c9cf3e86d7b02c5085968ffba698"} Jul 07 19:34:52.600894 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:34:52.600796 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-89748e-openshift-default-794dd46c5f-qp8hk" podStartSLOduration=1.600779362 podStartE2EDuration="1.600779362s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:34:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:34:52.598333384 +0000 UTC m=+2075.614337566" watchObservedRunningTime="2026-07-07 19:34:52.600779362 +0000 UTC m=+2075.616783543" Jul 07 19:34:53.097838 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:34:53.097801 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-89748e-openshift-default-794dd46c5f-qp8hk" Jul 07 19:34:53.102898 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:34:53.102871 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-89748e-openshift-default-794dd46c5f-qp8hk" Jul 07 19:34:53.584303 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:34:53.584226 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-89748e-openshift-default-794dd46c5f-qp8hk" Jul 07 19:34:53.585974 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:34:53.585916 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-89748e-openshift-default-794dd46c5f-qp8hk" Jul 07 19:34:54.522241 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:34:54.522215 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:34:54.619860 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:34:54.619829 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:34:54.698569 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:34:54.698538 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:34:54.847727 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:34:54.847690 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:34:55.129702 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:34:55.129623 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:34:55.230046 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:34:55.230015 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:34:58.612425 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:34:58.612346 2563 generic.go:356] "Generic (PLEG): container finished" podID="3132cd7e-44ee-4423-af07-b4bffe6ccdd6" containerID="362d52c84266b5a0312c3b3ae45a12445d8eaea763b272cc7d9d156145ba60e6" exitCode=0 Jul 07 19:34:58.612425 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:34:58.612390 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-2e6bb8-openshift-default-5c59fbb68f-n4n8h" event={"ID":"3132cd7e-44ee-4423-af07-b4bffe6ccdd6","Type":"ContainerDied","Data":"362d52c84266b5a0312c3b3ae45a12445d8eaea763b272cc7d9d156145ba60e6"} Jul 07 19:34:58.612864 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:34:58.612768 2563 scope.go:117] "RemoveContainer" containerID="362d52c84266b5a0312c3b3ae45a12445d8eaea763b272cc7d9d156145ba60e6" Jul 07 19:34:58.613841 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:34:58.613812 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 07 19:34:58.613977 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:34:58.613880 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 07 19:34:58.613977 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:34:58.613917 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 07 19:34:59.039331 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:34:59.039302 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:34:59.619289 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:34:59.619255 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-2e6bb8-openshift-default-5c59fbb68f-n4n8h" event={"ID":"3132cd7e-44ee-4423-af07-b4bffe6ccdd6","Type":"ContainerStarted","Data":"5c4a51cb9cbd7f17d0fb2eb0f0c1727217bdb20fff7e51bfcc65922888b3e97d"} Jul 07 19:35:00.465635 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:35:00.465589 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-2e6bb8-openshift-default-5c59fbb68f-n4n8h" Jul 07 19:35:00.466815 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:35:00.466778 2563 patch_prober.go:28] interesting pod/e2e-api-a-2e6bb8-openshift-default-5c59fbb68f-n4n8h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body= Jul 07 19:35:00.467007 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:35:00.466859 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-2e6bb8-openshift-default-5c59fbb68f-n4n8h" podUID="3132cd7e-44ee-4423-af07-b4bffe6ccdd6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" Jul 07 19:35:00.532249 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:35:00.532179 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-2e6bb8-openshift-default-5c59fbb68f-n4n8h" Jul 07 19:35:01.465630 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:35:01.465581 2563 patch_prober.go:28] interesting pod/e2e-api-a-2e6bb8-openshift-default-5c59fbb68f-n4n8h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body= Jul 07 19:35:01.466175 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:35:01.465675 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-2e6bb8-openshift-default-5c59fbb68f-n4n8h" podUID="3132cd7e-44ee-4423-af07-b4bffe6ccdd6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" Jul 07 19:35:02.468477 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:35:02.468426 2563 patch_prober.go:28] interesting pod/e2e-api-a-2e6bb8-openshift-default-5c59fbb68f-n4n8h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body= Jul 07 19:35:02.469046 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:35:02.468500 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-2e6bb8-openshift-default-5c59fbb68f-n4n8h" podUID="3132cd7e-44ee-4423-af07-b4bffe6ccdd6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" Jul 07 19:35:03.466841 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:35:03.466805 2563 patch_prober.go:28] interesting pod/e2e-api-a-2e6bb8-openshift-default-5c59fbb68f-n4n8h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body= Jul 07 19:35:03.467049 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:35:03.466868 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-2e6bb8-openshift-default-5c59fbb68f-n4n8h" podUID="3132cd7e-44ee-4423-af07-b4bffe6ccdd6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" Jul 07 19:35:04.469536 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:35:04.469504 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-2e6bb8-openshift-default-5c59fbb68f-n4n8h" Jul 07 19:35:04.470624 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:35:04.470600 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-2e6bb8-openshift-default-5c59fbb68f-n4n8h" Jul 07 19:35:06.659157 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:35:06.659123 2563 generic.go:356] "Generic (PLEG): container finished" podID="8d9ba960-8f2e-4920-b787-3da2276be176" containerID="35c1b7e56a932f9b20f921ee6c4acd01cd967824151c965342a27c5cd35aab27" exitCode=0 Jul 07 19:35:06.659597 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:35:06.659208 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-89748e-openshift-default-794dd46c5f-qp8hk" event={"ID":"8d9ba960-8f2e-4920-b787-3da2276be176","Type":"ContainerDied","Data":"35c1b7e56a932f9b20f921ee6c4acd01cd967824151c965342a27c5cd35aab27"} Jul 07 19:35:06.659661 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:35:06.659647 2563 scope.go:117] "RemoveContainer" containerID="35c1b7e56a932f9b20f921ee6c4acd01cd967824151c965342a27c5cd35aab27" Jul 07 19:35:06.660755 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:35:06.660726 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 07 19:35:06.660844 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:35:06.660824 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 07 19:35:06.660889 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:35:06.660878 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 07 19:35:07.153922 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:35:07.153888 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:35:07.553562 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:35:07.553531 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:35:07.666774 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:35:07.666744 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-89748e-openshift-default-794dd46c5f-qp8hk" event={"ID":"8d9ba960-8f2e-4920-b787-3da2276be176","Type":"ContainerStarted","Data":"6a4628c57545963d71d30837f4fe0c9ca69eea25b0375b4a15ddc594ba10d57b"} Jul 07 19:35:07.741057 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:35:07.741026 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:35:07.838670 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:35:07.838581 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:35:08.036277 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:35:08.036247 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:35:08.098563 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:35:08.098462 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-89748e-openshift-default-794dd46c5f-qp8hk" Jul 07 19:35:08.099774 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:35:08.099746 2563 patch_prober.go:28] interesting pod/e2e-api-b-89748e-openshift-default-794dd46c5f-qp8hk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body= Jul 07 19:35:08.099890 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:35:08.099801 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-89748e-openshift-default-794dd46c5f-qp8hk" podUID="8d9ba960-8f2e-4920-b787-3da2276be176" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" Jul 07 19:35:08.584600 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:35:08.584561 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-89748e-openshift-default-794dd46c5f-qp8hk" Jul 07 19:35:09.098346 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:35:09.098312 2563 patch_prober.go:28] interesting pod/e2e-api-b-89748e-openshift-default-794dd46c5f-qp8hk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body= Jul 07 19:35:09.098747 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:35:09.098370 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-89748e-openshift-default-794dd46c5f-qp8hk" podUID="8d9ba960-8f2e-4920-b787-3da2276be176" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" Jul 07 19:35:10.098405 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:35:10.098369 2563 patch_prober.go:28] interesting pod/e2e-api-b-89748e-openshift-default-794dd46c5f-qp8hk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body= Jul 07 19:35:10.098827 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:35:10.098426 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-89748e-openshift-default-794dd46c5f-qp8hk" podUID="8d9ba960-8f2e-4920-b787-3da2276be176" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" Jul 07 19:35:11.098860 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:35:11.098827 2563 patch_prober.go:28] interesting pod/e2e-api-b-89748e-openshift-default-794dd46c5f-qp8hk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body= Jul 07 19:35:11.099316 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:35:11.098884 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-89748e-openshift-default-794dd46c5f-qp8hk" podUID="8d9ba960-8f2e-4920-b787-3da2276be176" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" Jul 07 19:35:12.102155 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:35:12.102124 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-89748e-openshift-default-794dd46c5f-qp8hk" Jul 07 19:35:12.103295 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:35:12.103274 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-89748e-openshift-default-794dd46c5f-qp8hk" Jul 07 19:35:14.490924 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:35:14.490879 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:35:14.595599 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:35:14.595555 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:35:14.762415 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:35:14.762328 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-2e6bb8-openshift-default-5c59fbb68f-n4n8h"] Jul 07 19:35:14.762594 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:35:14.762537 2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-2e6bb8-openshift-default-5c59fbb68f-n4n8h" podUID="3132cd7e-44ee-4423-af07-b4bffe6ccdd6" containerName="istio-proxy" containerID="cri-o://5c4a51cb9cbd7f17d0fb2eb0f0c1727217bdb20fff7e51bfcc65922888b3e97d" gracePeriod=30 Jul 07 19:35:14.796239 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:35:14.796210 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:35:15.024466 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:35:15.024380 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:35:15.265660 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:35:15.265627 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:35:15.612600 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:35:15.612567 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:35:15.947263 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:35:15.947161 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:35:16.047725 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:35:16.047695 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:35:16.336156 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:35:16.336129 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:35:17.613379 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:35:17.613352 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-frt7c_e25ab82f-72fb-4cde-89c4-0a0dd80f5feb/console-operator/2.log" Jul 07 19:35:17.619936 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:35:17.619913 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zzwbq_4f0c5d5c-147f-426a-965d-8886e5e79984/ovn-acl-logging/0.log" Jul 07 19:35:17.619936 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:35:17.619927 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-frt7c_e25ab82f-72fb-4cde-89c4-0a0dd80f5feb/console-operator/2.log" Jul 07 19:35:17.626320 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:35:17.626298 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zzwbq_4f0c5d5c-147f-426a-965d-8886e5e79984/ovn-acl-logging/0.log" Jul 07 19:35:19.472493 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:35:19.472454 2563 patch_prober.go:28] interesting pod/e2e-api-a-2e6bb8-openshift-default-5c59fbb68f-n4n8h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 07 19:35:19.472867 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:35:19.472534 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-2e6bb8-openshift-default-5c59fbb68f-n4n8h" podUID="3132cd7e-44ee-4423-af07-b4bffe6ccdd6" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 07 19:35:20.022737 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:35:20.022674 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-2e6bb8-openshift-default-5c59fbb68f-n4n8h" Jul 07 19:35:20.127069 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:35:20.127041 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3132cd7e-44ee-4423-af07-b4bffe6ccdd6-credential-socket\") pod \"3132cd7e-44ee-4423-af07-b4bffe6ccdd6\" (UID: \"3132cd7e-44ee-4423-af07-b4bffe6ccdd6\") " Jul 07 19:35:20.127069 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:35:20.127073 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3132cd7e-44ee-4423-af07-b4bffe6ccdd6-istio-token\") pod \"3132cd7e-44ee-4423-af07-b4bffe6ccdd6\" (UID: \"3132cd7e-44ee-4423-af07-b4bffe6ccdd6\") " Jul 07 19:35:20.127324 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:35:20.127105 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3132cd7e-44ee-4423-af07-b4bffe6ccdd6-istio-envoy\") pod \"3132cd7e-44ee-4423-af07-b4bffe6ccdd6\" (UID: \"3132cd7e-44ee-4423-af07-b4bffe6ccdd6\") " Jul 07 19:35:20.127324 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:35:20.127126 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3132cd7e-44ee-4423-af07-b4bffe6ccdd6-istio-data\") pod \"3132cd7e-44ee-4423-af07-b4bffe6ccdd6\" (UID: \"3132cd7e-44ee-4423-af07-b4bffe6ccdd6\") " Jul 07 19:35:20.127324 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:35:20.127178 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3132cd7e-44ee-4423-af07-b4bffe6ccdd6-istiod-ca-cert\") pod \"3132cd7e-44ee-4423-af07-b4bffe6ccdd6\" (UID: \"3132cd7e-44ee-4423-af07-b4bffe6ccdd6\") " Jul 07 19:35:20.127324 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:35:20.127218 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3132cd7e-44ee-4423-af07-b4bffe6ccdd6-workload-certs\") pod \"3132cd7e-44ee-4423-af07-b4bffe6ccdd6\" (UID: \"3132cd7e-44ee-4423-af07-b4bffe6ccdd6\") " Jul 07 19:35:20.127324 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:35:20.127289 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3132cd7e-44ee-4423-af07-b4bffe6ccdd6-istio-podinfo\") pod \"3132cd7e-44ee-4423-af07-b4bffe6ccdd6\" (UID: \"3132cd7e-44ee-4423-af07-b4bffe6ccdd6\") " Jul 07 19:35:20.127324 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:35:20.127311 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sbf8z\" (UniqueName: \"kubernetes.io/projected/3132cd7e-44ee-4423-af07-b4bffe6ccdd6-kube-api-access-sbf8z\") pod \"3132cd7e-44ee-4423-af07-b4bffe6ccdd6\" (UID: \"3132cd7e-44ee-4423-af07-b4bffe6ccdd6\") " Jul 07 19:35:20.127629 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:35:20.127373 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3132cd7e-44ee-4423-af07-b4bffe6ccdd6-workload-socket\") pod \"3132cd7e-44ee-4423-af07-b4bffe6ccdd6\" (UID: \"3132cd7e-44ee-4423-af07-b4bffe6ccdd6\") " Jul 07 19:35:20.127629 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:35:20.127371 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3132cd7e-44ee-4423-af07-b4bffe6ccdd6-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "3132cd7e-44ee-4423-af07-b4bffe6ccdd6" (UID: "3132cd7e-44ee-4423-af07-b4bffe6ccdd6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:35:20.127733 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:35:20.127647 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3132cd7e-44ee-4423-af07-b4bffe6ccdd6-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "3132cd7e-44ee-4423-af07-b4bffe6ccdd6" (UID: "3132cd7e-44ee-4423-af07-b4bffe6ccdd6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:35:20.127790 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:35:20.127755 2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3132cd7e-44ee-4423-af07-b4bffe6ccdd6-workload-socket\") on node \"ip-10-0-142-134.ec2.internal\" DevicePath \"\"" Jul 07 19:35:20.127790 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:35:20.127772 2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3132cd7e-44ee-4423-af07-b4bffe6ccdd6-credential-socket\") on node \"ip-10-0-142-134.ec2.internal\" DevicePath \"\"" Jul 07 19:35:20.127891 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:35:20.127822 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3132cd7e-44ee-4423-af07-b4bffe6ccdd6-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "3132cd7e-44ee-4423-af07-b4bffe6ccdd6" (UID: "3132cd7e-44ee-4423-af07-b4bffe6ccdd6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:35:20.127891 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:35:20.127865 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3132cd7e-44ee-4423-af07-b4bffe6ccdd6-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "3132cd7e-44ee-4423-af07-b4bffe6ccdd6" (UID: "3132cd7e-44ee-4423-af07-b4bffe6ccdd6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 19:35:20.128258 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:35:20.128228 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3132cd7e-44ee-4423-af07-b4bffe6ccdd6-istio-data" (OuterVolumeSpecName: "istio-data") pod "3132cd7e-44ee-4423-af07-b4bffe6ccdd6" (UID: "3132cd7e-44ee-4423-af07-b4bffe6ccdd6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:35:20.129432 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:35:20.129409 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3132cd7e-44ee-4423-af07-b4bffe6ccdd6-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "3132cd7e-44ee-4423-af07-b4bffe6ccdd6" (UID: "3132cd7e-44ee-4423-af07-b4bffe6ccdd6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 07 19:35:20.129529 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:35:20.129506 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3132cd7e-44ee-4423-af07-b4bffe6ccdd6-istio-token" (OuterVolumeSpecName: "istio-token") pod "3132cd7e-44ee-4423-af07-b4bffe6ccdd6" (UID: "3132cd7e-44ee-4423-af07-b4bffe6ccdd6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 19:35:20.129728 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:35:20.129710 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3132cd7e-44ee-4423-af07-b4bffe6ccdd6-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "3132cd7e-44ee-4423-af07-b4bffe6ccdd6" (UID: "3132cd7e-44ee-4423-af07-b4bffe6ccdd6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:35:20.129787 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:35:20.129770 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3132cd7e-44ee-4423-af07-b4bffe6ccdd6-kube-api-access-sbf8z" (OuterVolumeSpecName: "kube-api-access-sbf8z") pod "3132cd7e-44ee-4423-af07-b4bffe6ccdd6" (UID: "3132cd7e-44ee-4423-af07-b4bffe6ccdd6"). InnerVolumeSpecName "kube-api-access-sbf8z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 19:35:20.228600 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:35:20.228569 2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3132cd7e-44ee-4423-af07-b4bffe6ccdd6-istio-token\") on node \"ip-10-0-142-134.ec2.internal\" DevicePath \"\"" Jul 07 19:35:20.228600 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:35:20.228592 2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3132cd7e-44ee-4423-af07-b4bffe6ccdd6-istio-envoy\") on node \"ip-10-0-142-134.ec2.internal\" DevicePath \"\"" Jul 07 19:35:20.228759 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:35:20.228605 2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3132cd7e-44ee-4423-af07-b4bffe6ccdd6-istio-data\") on node \"ip-10-0-142-134.ec2.internal\" DevicePath \"\"" Jul 07 19:35:20.228759 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:35:20.228618 2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3132cd7e-44ee-4423-af07-b4bffe6ccdd6-istiod-ca-cert\") on node \"ip-10-0-142-134.ec2.internal\" DevicePath \"\"" Jul 07 19:35:20.228759 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:35:20.228629 2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3132cd7e-44ee-4423-af07-b4bffe6ccdd6-workload-certs\") on node \"ip-10-0-142-134.ec2.internal\" DevicePath \"\"" Jul 07 19:35:20.228759 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:35:20.228640 2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3132cd7e-44ee-4423-af07-b4bffe6ccdd6-istio-podinfo\") on node \"ip-10-0-142-134.ec2.internal\" DevicePath \"\"" Jul 07 19:35:20.228759 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:35:20.228652 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sbf8z\" (UniqueName: \"kubernetes.io/projected/3132cd7e-44ee-4423-af07-b4bffe6ccdd6-kube-api-access-sbf8z\") on node \"ip-10-0-142-134.ec2.internal\" DevicePath \"\"" Jul 07 19:35:20.744551 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:35:20.744518 2563 generic.go:356] "Generic (PLEG): container finished" podID="3132cd7e-44ee-4423-af07-b4bffe6ccdd6" containerID="5c4a51cb9cbd7f17d0fb2eb0f0c1727217bdb20fff7e51bfcc65922888b3e97d" exitCode=0 Jul 07 19:35:20.745002 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:35:20.744574 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-2e6bb8-openshift-default-5c59fbb68f-n4n8h" event={"ID":"3132cd7e-44ee-4423-af07-b4bffe6ccdd6","Type":"ContainerDied","Data":"5c4a51cb9cbd7f17d0fb2eb0f0c1727217bdb20fff7e51bfcc65922888b3e97d"} Jul 07 19:35:20.745002 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:35:20.744586 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-2e6bb8-openshift-default-5c59fbb68f-n4n8h" Jul 07 19:35:20.745002 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:35:20.744622 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-2e6bb8-openshift-default-5c59fbb68f-n4n8h" event={"ID":"3132cd7e-44ee-4423-af07-b4bffe6ccdd6","Type":"ContainerDied","Data":"3199563223c862a3c2ed44af879e40a82940ed825169998414f2b69be52b691c"} Jul 07 19:35:20.745002 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:35:20.744644 2563 scope.go:117] "RemoveContainer" containerID="5c4a51cb9cbd7f17d0fb2eb0f0c1727217bdb20fff7e51bfcc65922888b3e97d" Jul 07 19:35:20.754120 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:35:20.754104 2563 scope.go:117] "RemoveContainer" containerID="362d52c84266b5a0312c3b3ae45a12445d8eaea763b272cc7d9d156145ba60e6" Jul 07 19:35:20.761744 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:35:20.761725 2563 scope.go:117] "RemoveContainer" containerID="5c4a51cb9cbd7f17d0fb2eb0f0c1727217bdb20fff7e51bfcc65922888b3e97d" Jul 07 19:35:20.761998 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:35:20.761980 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c4a51cb9cbd7f17d0fb2eb0f0c1727217bdb20fff7e51bfcc65922888b3e97d\": container with ID starting with 5c4a51cb9cbd7f17d0fb2eb0f0c1727217bdb20fff7e51bfcc65922888b3e97d not found: ID does not exist" containerID="5c4a51cb9cbd7f17d0fb2eb0f0c1727217bdb20fff7e51bfcc65922888b3e97d" Jul 07 19:35:20.762079 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:35:20.762005 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c4a51cb9cbd7f17d0fb2eb0f0c1727217bdb20fff7e51bfcc65922888b3e97d"} err="failed to get container status \"5c4a51cb9cbd7f17d0fb2eb0f0c1727217bdb20fff7e51bfcc65922888b3e97d\": rpc error: code = NotFound desc = could not find container \"5c4a51cb9cbd7f17d0fb2eb0f0c1727217bdb20fff7e51bfcc65922888b3e97d\": container with ID starting with 5c4a51cb9cbd7f17d0fb2eb0f0c1727217bdb20fff7e51bfcc65922888b3e97d not found: ID does not exist" Jul 07 19:35:20.762079 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:35:20.762021 2563 scope.go:117] "RemoveContainer" containerID="362d52c84266b5a0312c3b3ae45a12445d8eaea763b272cc7d9d156145ba60e6" Jul 07 19:35:20.762253 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:35:20.762238 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"362d52c84266b5a0312c3b3ae45a12445d8eaea763b272cc7d9d156145ba60e6\": container with ID starting with 362d52c84266b5a0312c3b3ae45a12445d8eaea763b272cc7d9d156145ba60e6 not found: ID does not exist" containerID="362d52c84266b5a0312c3b3ae45a12445d8eaea763b272cc7d9d156145ba60e6" Jul 07 19:35:20.762294 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:35:20.762257 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"362d52c84266b5a0312c3b3ae45a12445d8eaea763b272cc7d9d156145ba60e6"} err="failed to get container status \"362d52c84266b5a0312c3b3ae45a12445d8eaea763b272cc7d9d156145ba60e6\": rpc error: code = NotFound desc = could not find container \"362d52c84266b5a0312c3b3ae45a12445d8eaea763b272cc7d9d156145ba60e6\": container with ID starting with 362d52c84266b5a0312c3b3ae45a12445d8eaea763b272cc7d9d156145ba60e6 not found: ID does not exist" Jul 07 19:35:20.766699 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:35:20.766673 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-2e6bb8-openshift-default-5c59fbb68f-n4n8h"] Jul 07 19:35:20.777420 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:35:20.777397 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-2e6bb8-openshift-default-5c59fbb68f-n4n8h"] Jul 07 19:35:21.477568 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:35:21.477532 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3132cd7e-44ee-4423-af07-b4bffe6ccdd6" path="/var/lib/kubelet/pods/3132cd7e-44ee-4423-af07-b4bffe6ccdd6/volumes" Jul 07 19:35:22.287846 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:35:22.287814 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:35:22.388688 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:35:22.388657 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:35:22.529527 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:35:22.529489 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-89748e-openshift-default-794dd46c5f-qp8hk"] Jul 07 19:35:22.529799 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:35:22.529776 2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-89748e-openshift-default-794dd46c5f-qp8hk" podUID="8d9ba960-8f2e-4920-b787-3da2276be176" containerName="istio-proxy" containerID="cri-o://6a4628c57545963d71d30837f4fe0c9ca69eea25b0375b4a15ddc594ba10d57b" gracePeriod=30 Jul 07 19:35:22.558606 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:35:22.558523 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:35:22.779699 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:35:22.779661 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:35:23.108913 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:35:23.108880 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:35:23.410164 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:35:23.410064 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:35:23.522507 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:35:23.522465 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:35:23.864302 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:35:23.864268 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:35:26.071803 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:35:26.071768 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:35:26.179160 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:35:26.179130 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:35:26.312172 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:35:26.312142 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:35:26.616919 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:35:26.616890 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:35:26.889381 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:35:26.889302 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:35:26.987155 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:35:26.987124 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:35:27.104376 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:35:27.104337 2563 patch_prober.go:28] interesting pod/e2e-api-b-89748e-openshift-default-794dd46c5f-qp8hk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 07 19:35:27.104751 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:35:27.104408 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-89748e-openshift-default-794dd46c5f-qp8hk" podUID="8d9ba960-8f2e-4920-b787-3da2276be176" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 07 19:35:27.785561 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:35:27.785530 2563 generic.go:356] "Generic (PLEG): container finished" podID="8d9ba960-8f2e-4920-b787-3da2276be176" containerID="6a4628c57545963d71d30837f4fe0c9ca69eea25b0375b4a15ddc594ba10d57b" exitCode=0 Jul 07 19:35:27.785707 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:35:27.785609 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-89748e-openshift-default-794dd46c5f-qp8hk" event={"ID":"8d9ba960-8f2e-4920-b787-3da2276be176","Type":"ContainerDied","Data":"6a4628c57545963d71d30837f4fe0c9ca69eea25b0375b4a15ddc594ba10d57b"} Jul 07 19:35:27.785707 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:35:27.785677 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-89748e-openshift-default-794dd46c5f-qp8hk" event={"ID":"8d9ba960-8f2e-4920-b787-3da2276be176","Type":"ContainerDied","Data":"9de5651b7a8b65b6097c9b8fb28b0e54e552c9cf3e86d7b02c5085968ffba698"} Jul 07 19:35:27.785707 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:35:27.785700 2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9de5651b7a8b65b6097c9b8fb28b0e54e552c9cf3e86d7b02c5085968ffba698" Jul 07 19:35:27.785866 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:35:27.785717 2563 scope.go:117] "RemoveContainer" containerID="35c1b7e56a932f9b20f921ee6c4acd01cd967824151c965342a27c5cd35aab27" Jul 07 19:35:27.790563 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:35:27.790517 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-89748e-openshift-default-794dd46c5f-qp8hk" Jul 07 19:35:27.905465 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:35:27.905431 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d9ba960-8f2e-4920-b787-3da2276be176-istio-envoy\") pod \"8d9ba960-8f2e-4920-b787-3da2276be176\" (UID: \"8d9ba960-8f2e-4920-b787-3da2276be176\") " Jul 07 19:35:27.905621 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:35:27.905480 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d9ba960-8f2e-4920-b787-3da2276be176-credential-socket\") pod \"8d9ba960-8f2e-4920-b787-3da2276be176\" (UID: \"8d9ba960-8f2e-4920-b787-3da2276be176\") " Jul 07 19:35:27.905621 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:35:27.905507 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d9ba960-8f2e-4920-b787-3da2276be176-istio-token\") pod \"8d9ba960-8f2e-4920-b787-3da2276be176\" (UID: \"8d9ba960-8f2e-4920-b787-3da2276be176\") " Jul 07 19:35:27.905621 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:35:27.905561 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d9ba960-8f2e-4920-b787-3da2276be176-istio-podinfo\") pod \"8d9ba960-8f2e-4920-b787-3da2276be176\" (UID: \"8d9ba960-8f2e-4920-b787-3da2276be176\") " Jul 07 19:35:27.905621 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:35:27.905609 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d9ba960-8f2e-4920-b787-3da2276be176-workload-socket\") pod \"8d9ba960-8f2e-4920-b787-3da2276be176\" (UID: \"8d9ba960-8f2e-4920-b787-3da2276be176\") " Jul 07 19:35:27.905847 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:35:27.905645 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tlbr6\" (UniqueName: \"kubernetes.io/projected/8d9ba960-8f2e-4920-b787-3da2276be176-kube-api-access-tlbr6\") pod \"8d9ba960-8f2e-4920-b787-3da2276be176\" (UID: \"8d9ba960-8f2e-4920-b787-3da2276be176\") " Jul 07 19:35:27.905847 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:35:27.905671 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d9ba960-8f2e-4920-b787-3da2276be176-istiod-ca-cert\") pod \"8d9ba960-8f2e-4920-b787-3da2276be176\" (UID: \"8d9ba960-8f2e-4920-b787-3da2276be176\") " Jul 07 19:35:27.905847 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:35:27.905702 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d9ba960-8f2e-4920-b787-3da2276be176-workload-certs\") pod \"8d9ba960-8f2e-4920-b787-3da2276be176\" (UID: \"8d9ba960-8f2e-4920-b787-3da2276be176\") " Jul 07 19:35:27.905847 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:35:27.905809 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d9ba960-8f2e-4920-b787-3da2276be176-istio-data\") pod \"8d9ba960-8f2e-4920-b787-3da2276be176\" (UID: \"8d9ba960-8f2e-4920-b787-3da2276be176\") " Jul 07 19:35:27.906173 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:35:27.905803 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d9ba960-8f2e-4920-b787-3da2276be176-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "8d9ba960-8f2e-4920-b787-3da2276be176" (UID: "8d9ba960-8f2e-4920-b787-3da2276be176"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:35:27.906173 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:35:27.905940 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d9ba960-8f2e-4920-b787-3da2276be176-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "8d9ba960-8f2e-4920-b787-3da2276be176" (UID: "8d9ba960-8f2e-4920-b787-3da2276be176"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:35:27.906370 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:35:27.906197 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d9ba960-8f2e-4920-b787-3da2276be176-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "8d9ba960-8f2e-4920-b787-3da2276be176" (UID: "8d9ba960-8f2e-4920-b787-3da2276be176"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:35:27.906370 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:35:27.906223 2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d9ba960-8f2e-4920-b787-3da2276be176-credential-socket\") on node \"ip-10-0-142-134.ec2.internal\" DevicePath \"\"" Jul 07 19:35:27.906370 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:35:27.906241 2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d9ba960-8f2e-4920-b787-3da2276be176-workload-certs\") on node \"ip-10-0-142-134.ec2.internal\" DevicePath \"\"" Jul 07 19:35:27.906370 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:35:27.906239 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d9ba960-8f2e-4920-b787-3da2276be176-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "8d9ba960-8f2e-4920-b787-3da2276be176" (UID: "8d9ba960-8f2e-4920-b787-3da2276be176"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 19:35:27.906743 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:35:27.906720 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d9ba960-8f2e-4920-b787-3da2276be176-istio-data" (OuterVolumeSpecName: "istio-data") pod "8d9ba960-8f2e-4920-b787-3da2276be176" (UID: "8d9ba960-8f2e-4920-b787-3da2276be176"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:35:27.907727 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:35:27.907703 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d9ba960-8f2e-4920-b787-3da2276be176-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "8d9ba960-8f2e-4920-b787-3da2276be176" (UID: "8d9ba960-8f2e-4920-b787-3da2276be176"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:35:27.907868 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:35:27.907852 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d9ba960-8f2e-4920-b787-3da2276be176-istio-token" (OuterVolumeSpecName: "istio-token") pod "8d9ba960-8f2e-4920-b787-3da2276be176" (UID: "8d9ba960-8f2e-4920-b787-3da2276be176"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 19:35:27.908197 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:35:27.908182 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d9ba960-8f2e-4920-b787-3da2276be176-kube-api-access-tlbr6" (OuterVolumeSpecName: "kube-api-access-tlbr6") pod "8d9ba960-8f2e-4920-b787-3da2276be176" (UID: "8d9ba960-8f2e-4920-b787-3da2276be176"). InnerVolumeSpecName "kube-api-access-tlbr6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 19:35:27.908311 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:35:27.908292 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8d9ba960-8f2e-4920-b787-3da2276be176-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "8d9ba960-8f2e-4920-b787-3da2276be176" (UID: "8d9ba960-8f2e-4920-b787-3da2276be176"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 07 19:35:28.007229 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:35:28.007185 2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d9ba960-8f2e-4920-b787-3da2276be176-istio-data\") on node \"ip-10-0-142-134.ec2.internal\" DevicePath \"\"" Jul 07 19:35:28.007229 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:35:28.007218 2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d9ba960-8f2e-4920-b787-3da2276be176-istio-envoy\") on node \"ip-10-0-142-134.ec2.internal\" DevicePath \"\"" Jul 07 19:35:28.007229 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:35:28.007226 2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d9ba960-8f2e-4920-b787-3da2276be176-istio-token\") on node \"ip-10-0-142-134.ec2.internal\" DevicePath \"\"" Jul 07 19:35:28.007229 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:35:28.007234 2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d9ba960-8f2e-4920-b787-3da2276be176-istio-podinfo\") on node \"ip-10-0-142-134.ec2.internal\" DevicePath \"\"" Jul 07 19:35:28.007229 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:35:28.007241 2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d9ba960-8f2e-4920-b787-3da2276be176-workload-socket\") on node \"ip-10-0-142-134.ec2.internal\" DevicePath \"\"" Jul 07 19:35:28.007531 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:35:28.007250 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tlbr6\" (UniqueName: \"kubernetes.io/projected/8d9ba960-8f2e-4920-b787-3da2276be176-kube-api-access-tlbr6\") on node \"ip-10-0-142-134.ec2.internal\" DevicePath \"\"" Jul 07 19:35:28.007531 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:35:28.007259 2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d9ba960-8f2e-4920-b787-3da2276be176-istiod-ca-cert\") on node \"ip-10-0-142-134.ec2.internal\" DevicePath \"\"" Jul 07 19:35:28.792143 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:35:28.792108 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-89748e-openshift-default-794dd46c5f-qp8hk" Jul 07 19:35:28.849945 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:35:28.849910 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-89748e-openshift-default-794dd46c5f-qp8hk"] Jul 07 19:35:28.856237 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:35:28.856210 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-89748e-openshift-default-794dd46c5f-qp8hk"] Jul 07 19:35:29.068937 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:35:29.068856 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:35:29.476057 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:35:29.475981 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8d9ba960-8f2e-4920-b787-3da2276be176" path="/var/lib/kubelet/pods/8d9ba960-8f2e-4920-b787-3da2276be176/volumes" Jul 07 19:35:30.564184 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:35:30.564149 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:35:30.952714 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:35:30.952624 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:35:31.207179 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:35:31.207105 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-836c8d-openshift-default-65c87bb9d7-n97cr"] Jul 07 19:35:31.207511 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:35:31.207498 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3132cd7e-44ee-4423-af07-b4bffe6ccdd6" containerName="istio-proxy" Jul 07 19:35:31.207550 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:35:31.207512 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="3132cd7e-44ee-4423-af07-b4bffe6ccdd6" containerName="istio-proxy" Jul 07 19:35:31.207550 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:35:31.207520 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d9ba960-8f2e-4920-b787-3da2276be176" containerName="istio-proxy" Jul 07 19:35:31.207550 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:35:31.207526 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d9ba960-8f2e-4920-b787-3da2276be176" containerName="istio-proxy" Jul 07 19:35:31.207550 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:35:31.207541 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d9ba960-8f2e-4920-b787-3da2276be176" containerName="istio-proxy" Jul 07 19:35:31.207550 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:35:31.207546 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d9ba960-8f2e-4920-b787-3da2276be176" containerName="istio-proxy" Jul 07 19:35:31.207697 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:35:31.207610 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="8d9ba960-8f2e-4920-b787-3da2276be176" containerName="istio-proxy" Jul 07 19:35:31.207697 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:35:31.207617 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="3132cd7e-44ee-4423-af07-b4bffe6ccdd6" containerName="istio-proxy" Jul 07 19:35:31.207697 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:35:31.207623 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="3132cd7e-44ee-4423-af07-b4bffe6ccdd6" containerName="istio-proxy" Jul 07 19:35:31.207697 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:35:31.207633 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="8d9ba960-8f2e-4920-b787-3da2276be176" containerName="istio-proxy" Jul 07 19:35:31.212020 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:35:31.211998 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-836c8d-openshift-default-65c87bb9d7-n97cr" Jul 07 19:35:31.214163 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:35:31.214143 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-836c8d-openshift-default-dockercfg-pz28b\"" Jul 07 19:35:31.226274 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:35:31.226250 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-836c8d-openshift-default-65c87bb9d7-n97cr"] Jul 07 19:35:31.268801 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:35:31.268771 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:35:31.344367 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:35:31.344327 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88c48a48-4ff5-41eb-9e13-97993d7bf2cb-istio-envoy\") pod \"e2e-shared-b-836c8d-openshift-default-65c87bb9d7-n97cr\" (UID: \"88c48a48-4ff5-41eb-9e13-97993d7bf2cb\") " pod="openshift-ingress/e2e-shared-b-836c8d-openshift-default-65c87bb9d7-n97cr" Jul 07 19:35:31.344367 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:35:31.344370 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88c48a48-4ff5-41eb-9e13-97993d7bf2cb-istiod-ca-cert\") pod \"e2e-shared-b-836c8d-openshift-default-65c87bb9d7-n97cr\" (UID: \"88c48a48-4ff5-41eb-9e13-97993d7bf2cb\") " pod="openshift-ingress/e2e-shared-b-836c8d-openshift-default-65c87bb9d7-n97cr" Jul 07 19:35:31.344609 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:35:31.344471 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88c48a48-4ff5-41eb-9e13-97993d7bf2cb-workload-socket\") pod \"e2e-shared-b-836c8d-openshift-default-65c87bb9d7-n97cr\" (UID: \"88c48a48-4ff5-41eb-9e13-97993d7bf2cb\") " pod="openshift-ingress/e2e-shared-b-836c8d-openshift-default-65c87bb9d7-n97cr" Jul 07 19:35:31.344609 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:35:31.344518 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5fwh\" (UniqueName: \"kubernetes.io/projected/88c48a48-4ff5-41eb-9e13-97993d7bf2cb-kube-api-access-d5fwh\") pod \"e2e-shared-b-836c8d-openshift-default-65c87bb9d7-n97cr\" (UID: \"88c48a48-4ff5-41eb-9e13-97993d7bf2cb\") " pod="openshift-ingress/e2e-shared-b-836c8d-openshift-default-65c87bb9d7-n97cr" Jul 07 19:35:31.344609 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:35:31.344548 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88c48a48-4ff5-41eb-9e13-97993d7bf2cb-workload-certs\") pod \"e2e-shared-b-836c8d-openshift-default-65c87bb9d7-n97cr\" (UID: \"88c48a48-4ff5-41eb-9e13-97993d7bf2cb\") " pod="openshift-ingress/e2e-shared-b-836c8d-openshift-default-65c87bb9d7-n97cr" Jul 07 19:35:31.344609 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:35:31.344586 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88c48a48-4ff5-41eb-9e13-97993d7bf2cb-istio-token\") pod \"e2e-shared-b-836c8d-openshift-default-65c87bb9d7-n97cr\" (UID: \"88c48a48-4ff5-41eb-9e13-97993d7bf2cb\") " pod="openshift-ingress/e2e-shared-b-836c8d-openshift-default-65c87bb9d7-n97cr" Jul 07 19:35:31.344743 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:35:31.344709 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88c48a48-4ff5-41eb-9e13-97993d7bf2cb-istio-data\") pod \"e2e-shared-b-836c8d-openshift-default-65c87bb9d7-n97cr\" (UID: \"88c48a48-4ff5-41eb-9e13-97993d7bf2cb\") " pod="openshift-ingress/e2e-shared-b-836c8d-openshift-default-65c87bb9d7-n97cr" Jul 07 19:35:31.344782 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:35:31.344753 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88c48a48-4ff5-41eb-9e13-97993d7bf2cb-istio-podinfo\") pod \"e2e-shared-b-836c8d-openshift-default-65c87bb9d7-n97cr\" (UID: \"88c48a48-4ff5-41eb-9e13-97993d7bf2cb\") " pod="openshift-ingress/e2e-shared-b-836c8d-openshift-default-65c87bb9d7-n97cr" Jul 07 19:35:31.344816 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:35:31.344803 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88c48a48-4ff5-41eb-9e13-97993d7bf2cb-credential-socket\") pod \"e2e-shared-b-836c8d-openshift-default-65c87bb9d7-n97cr\" (UID: \"88c48a48-4ff5-41eb-9e13-97993d7bf2cb\") " pod="openshift-ingress/e2e-shared-b-836c8d-openshift-default-65c87bb9d7-n97cr" Jul 07 19:35:31.445625 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:35:31.445583 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88c48a48-4ff5-41eb-9e13-97993d7bf2cb-workload-socket\") pod \"e2e-shared-b-836c8d-openshift-default-65c87bb9d7-n97cr\" (UID: \"88c48a48-4ff5-41eb-9e13-97993d7bf2cb\") " pod="openshift-ingress/e2e-shared-b-836c8d-openshift-default-65c87bb9d7-n97cr" Jul 07 19:35:31.445625 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:35:31.445632 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5fwh\" (UniqueName: \"kubernetes.io/projected/88c48a48-4ff5-41eb-9e13-97993d7bf2cb-kube-api-access-d5fwh\") pod \"e2e-shared-b-836c8d-openshift-default-65c87bb9d7-n97cr\" (UID: \"88c48a48-4ff5-41eb-9e13-97993d7bf2cb\") " pod="openshift-ingress/e2e-shared-b-836c8d-openshift-default-65c87bb9d7-n97cr" Jul 07 19:35:31.445892 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:35:31.445659 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88c48a48-4ff5-41eb-9e13-97993d7bf2cb-workload-certs\") pod \"e2e-shared-b-836c8d-openshift-default-65c87bb9d7-n97cr\" (UID: \"88c48a48-4ff5-41eb-9e13-97993d7bf2cb\") " pod="openshift-ingress/e2e-shared-b-836c8d-openshift-default-65c87bb9d7-n97cr" Jul 07 19:35:31.445892 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:35:31.445707 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88c48a48-4ff5-41eb-9e13-97993d7bf2cb-istio-token\") pod \"e2e-shared-b-836c8d-openshift-default-65c87bb9d7-n97cr\" (UID: \"88c48a48-4ff5-41eb-9e13-97993d7bf2cb\") " pod="openshift-ingress/e2e-shared-b-836c8d-openshift-default-65c87bb9d7-n97cr" Jul 07 19:35:31.445892 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:35:31.445788 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88c48a48-4ff5-41eb-9e13-97993d7bf2cb-istio-data\") pod \"e2e-shared-b-836c8d-openshift-default-65c87bb9d7-n97cr\" (UID: \"88c48a48-4ff5-41eb-9e13-97993d7bf2cb\") " pod="openshift-ingress/e2e-shared-b-836c8d-openshift-default-65c87bb9d7-n97cr" Jul 07 19:35:31.445892 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:35:31.445816 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88c48a48-4ff5-41eb-9e13-97993d7bf2cb-istio-podinfo\") pod \"e2e-shared-b-836c8d-openshift-default-65c87bb9d7-n97cr\" (UID: \"88c48a48-4ff5-41eb-9e13-97993d7bf2cb\") " pod="openshift-ingress/e2e-shared-b-836c8d-openshift-default-65c87bb9d7-n97cr" Jul 07 19:35:31.445892 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:35:31.445846 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88c48a48-4ff5-41eb-9e13-97993d7bf2cb-credential-socket\") pod \"e2e-shared-b-836c8d-openshift-default-65c87bb9d7-n97cr\" (UID: \"88c48a48-4ff5-41eb-9e13-97993d7bf2cb\") " pod="openshift-ingress/e2e-shared-b-836c8d-openshift-default-65c87bb9d7-n97cr" Jul 07 19:35:31.446169 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:35:31.445903 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88c48a48-4ff5-41eb-9e13-97993d7bf2cb-istio-envoy\") pod \"e2e-shared-b-836c8d-openshift-default-65c87bb9d7-n97cr\" (UID: \"88c48a48-4ff5-41eb-9e13-97993d7bf2cb\") " pod="openshift-ingress/e2e-shared-b-836c8d-openshift-default-65c87bb9d7-n97cr" Jul 07 19:35:31.446169 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:35:31.445936 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88c48a48-4ff5-41eb-9e13-97993d7bf2cb-istiod-ca-cert\") pod \"e2e-shared-b-836c8d-openshift-default-65c87bb9d7-n97cr\" (UID: \"88c48a48-4ff5-41eb-9e13-97993d7bf2cb\") " pod="openshift-ingress/e2e-shared-b-836c8d-openshift-default-65c87bb9d7-n97cr" Jul 07 19:35:31.446169 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:35:31.446015 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88c48a48-4ff5-41eb-9e13-97993d7bf2cb-workload-socket\") pod \"e2e-shared-b-836c8d-openshift-default-65c87bb9d7-n97cr\" (UID: \"88c48a48-4ff5-41eb-9e13-97993d7bf2cb\") " pod="openshift-ingress/e2e-shared-b-836c8d-openshift-default-65c87bb9d7-n97cr" Jul 07 19:35:31.446169 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:35:31.446116 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88c48a48-4ff5-41eb-9e13-97993d7bf2cb-workload-certs\") pod \"e2e-shared-b-836c8d-openshift-default-65c87bb9d7-n97cr\" (UID: \"88c48a48-4ff5-41eb-9e13-97993d7bf2cb\") " pod="openshift-ingress/e2e-shared-b-836c8d-openshift-default-65c87bb9d7-n97cr" Jul 07 19:35:31.446388 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:35:31.446279 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88c48a48-4ff5-41eb-9e13-97993d7bf2cb-istio-data\") pod \"e2e-shared-b-836c8d-openshift-default-65c87bb9d7-n97cr\" (UID: \"88c48a48-4ff5-41eb-9e13-97993d7bf2cb\") " pod="openshift-ingress/e2e-shared-b-836c8d-openshift-default-65c87bb9d7-n97cr" Jul 07 19:35:31.446388 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:35:31.446278 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88c48a48-4ff5-41eb-9e13-97993d7bf2cb-credential-socket\") pod \"e2e-shared-b-836c8d-openshift-default-65c87bb9d7-n97cr\" (UID: \"88c48a48-4ff5-41eb-9e13-97993d7bf2cb\") " pod="openshift-ingress/e2e-shared-b-836c8d-openshift-default-65c87bb9d7-n97cr" Jul 07 19:35:31.446706 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:35:31.446684 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88c48a48-4ff5-41eb-9e13-97993d7bf2cb-istiod-ca-cert\") pod \"e2e-shared-b-836c8d-openshift-default-65c87bb9d7-n97cr\" (UID: \"88c48a48-4ff5-41eb-9e13-97993d7bf2cb\") " pod="openshift-ingress/e2e-shared-b-836c8d-openshift-default-65c87bb9d7-n97cr" Jul 07 19:35:31.448144 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:35:31.448125 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88c48a48-4ff5-41eb-9e13-97993d7bf2cb-istio-envoy\") pod \"e2e-shared-b-836c8d-openshift-default-65c87bb9d7-n97cr\" (UID: \"88c48a48-4ff5-41eb-9e13-97993d7bf2cb\") " pod="openshift-ingress/e2e-shared-b-836c8d-openshift-default-65c87bb9d7-n97cr" Jul 07 19:35:31.448373 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:35:31.448352 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88c48a48-4ff5-41eb-9e13-97993d7bf2cb-istio-podinfo\") pod \"e2e-shared-b-836c8d-openshift-default-65c87bb9d7-n97cr\" (UID: \"88c48a48-4ff5-41eb-9e13-97993d7bf2cb\") " pod="openshift-ingress/e2e-shared-b-836c8d-openshift-default-65c87bb9d7-n97cr" Jul 07 19:35:31.455866 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:35:31.455838 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88c48a48-4ff5-41eb-9e13-97993d7bf2cb-istio-token\") pod \"e2e-shared-b-836c8d-openshift-default-65c87bb9d7-n97cr\" (UID: \"88c48a48-4ff5-41eb-9e13-97993d7bf2cb\") " pod="openshift-ingress/e2e-shared-b-836c8d-openshift-default-65c87bb9d7-n97cr" Jul 07 19:35:31.456044 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:35:31.456024 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5fwh\" (UniqueName: \"kubernetes.io/projected/88c48a48-4ff5-41eb-9e13-97993d7bf2cb-kube-api-access-d5fwh\") pod \"e2e-shared-b-836c8d-openshift-default-65c87bb9d7-n97cr\" (UID: \"88c48a48-4ff5-41eb-9e13-97993d7bf2cb\") " pod="openshift-ingress/e2e-shared-b-836c8d-openshift-default-65c87bb9d7-n97cr" Jul 07 19:35:31.523144 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:35:31.523046 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-836c8d-openshift-default-65c87bb9d7-n97cr" Jul 07 19:35:31.605668 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:35:31.605636 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:35:31.672783 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:35:31.672754 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-836c8d-openshift-default-65c87bb9d7-n97cr"] Jul 07 19:35:31.675036 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:35:31.675006 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88c48a48_4ff5_41eb_9e13_97993d7bf2cb.slice/crio-ea574ba3a788bff8cef3c2248b4a29bf88e5d22fffdeaa8f9b5fc8c9f391ec07 WatchSource:0}: Error finding container ea574ba3a788bff8cef3c2248b4a29bf88e5d22fffdeaa8f9b5fc8c9f391ec07: Status 404 returned error can't find the container with id ea574ba3a788bff8cef3c2248b4a29bf88e5d22fffdeaa8f9b5fc8c9f391ec07 Jul 07 19:35:31.677088 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:35:31.677052 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 07 19:35:31.677207 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:35:31.677133 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 07 19:35:31.677207 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:35:31.677179 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 07 19:35:31.808728 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:35:31.808691 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-836c8d-openshift-default-65c87bb9d7-n97cr" event={"ID":"88c48a48-4ff5-41eb-9e13-97993d7bf2cb","Type":"ContainerStarted","Data":"07c50230ba4ca120f3467d115c1036b275d6e06f2abb1a5a67c8c435ba2cf3eb"} Jul 07 19:35:31.808970 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:35:31.808760 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-836c8d-openshift-default-65c87bb9d7-n97cr" event={"ID":"88c48a48-4ff5-41eb-9e13-97993d7bf2cb","Type":"ContainerStarted","Data":"ea574ba3a788bff8cef3c2248b4a29bf88e5d22fffdeaa8f9b5fc8c9f391ec07"} Jul 07 19:35:31.835760 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:35:31.835696 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-836c8d-openshift-default-65c87bb9d7-n97cr" podStartSLOduration=0.835674225 podStartE2EDuration="835.674225ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:35:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:35:31.833697612 +0000 UTC m=+2114.849701795" watchObservedRunningTime="2026-07-07 19:35:31.835674225 +0000 UTC m=+2114.851678405" Jul 07 19:35:32.523962 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:35:32.523917 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-836c8d-openshift-default-65c87bb9d7-n97cr" Jul 07 19:35:32.528679 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:35:32.528640 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-836c8d-openshift-default-65c87bb9d7-n97cr" Jul 07 19:35:32.813570 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:35:32.813528 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-836c8d-openshift-default-65c87bb9d7-n97cr" Jul 07 19:35:32.814617 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:35:32.814597 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-836c8d-openshift-default-65c87bb9d7-n97cr" Jul 07 19:35:33.867633 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:35:33.867597 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:35:34.007321 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:35:34.007284 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:35:34.145123 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:35:34.145037 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:35:34.289648 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:35:34.289605 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:35:34.782113 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:35:34.782078 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:35:34.967171 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:35:34.967139 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:35:38.341722 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:35:38.341689 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:35:41.792705 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:35:41.792674 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:35:44.877642 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:35:44.877609 2563 generic.go:356] "Generic (PLEG): container finished" podID="88c48a48-4ff5-41eb-9e13-97993d7bf2cb" containerID="07c50230ba4ca120f3467d115c1036b275d6e06f2abb1a5a67c8c435ba2cf3eb" exitCode=0 Jul 07 19:35:44.878090 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:35:44.877684 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-836c8d-openshift-default-65c87bb9d7-n97cr" event={"ID":"88c48a48-4ff5-41eb-9e13-97993d7bf2cb","Type":"ContainerDied","Data":"07c50230ba4ca120f3467d115c1036b275d6e06f2abb1a5a67c8c435ba2cf3eb"} Jul 07 19:35:44.878135 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:35:44.878090 2563 scope.go:117] "RemoveContainer" containerID="07c50230ba4ca120f3467d115c1036b275d6e06f2abb1a5a67c8c435ba2cf3eb" Jul 07 19:35:44.879057 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:35:44.879024 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 07 19:35:44.879170 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:35:44.879090 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 07 19:35:44.879170 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:35:44.879125 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 07 19:35:45.887160 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:35:45.887125 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-836c8d-openshift-default-65c87bb9d7-n97cr" event={"ID":"88c48a48-4ff5-41eb-9e13-97993d7bf2cb","Type":"ContainerStarted","Data":"18d2d31af4a753da83f1c2562593b4f9bfdbece2ae693c63acbd4321e366b777"} Jul 07 19:35:46.523381 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:35:46.523341 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-836c8d-openshift-default-65c87bb9d7-n97cr" Jul 07 19:35:46.524485 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:35:46.524460 2563 patch_prober.go:28] interesting pod/e2e-shared-b-836c8d-openshift-default-65c87bb9d7-n97cr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body= Jul 07 19:35:46.524610 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:35:46.524520 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-836c8d-openshift-default-65c87bb9d7-n97cr" podUID="88c48a48-4ff5-41eb-9e13-97993d7bf2cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" Jul 07 19:35:47.523890 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:35:47.523842 2563 patch_prober.go:28] interesting pod/e2e-shared-b-836c8d-openshift-default-65c87bb9d7-n97cr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body= Jul 07 19:35:47.524308 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:35:47.523909 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-836c8d-openshift-default-65c87bb9d7-n97cr" podUID="88c48a48-4ff5-41eb-9e13-97993d7bf2cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" Jul 07 19:35:47.813805 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:35:47.813773 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-836c8d-openshift-default-65c87bb9d7-n97cr" Jul 07 19:35:48.523877 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:35:48.523842 2563 patch_prober.go:28] interesting pod/e2e-shared-b-836c8d-openshift-default-65c87bb9d7-n97cr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body= Jul 07 19:35:48.524061 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:35:48.523909 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-836c8d-openshift-default-65c87bb9d7-n97cr" podUID="88c48a48-4ff5-41eb-9e13-97993d7bf2cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" Jul 07 19:35:48.762785 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:35:48.762731 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:35:48.860794 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:35:48.860749 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:35:49.523751 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:35:49.523715 2563 patch_prober.go:28] interesting pod/e2e-shared-b-836c8d-openshift-default-65c87bb9d7-n97cr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body= Jul 07 19:35:49.523941 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:35:49.523772 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-836c8d-openshift-default-65c87bb9d7-n97cr" podUID="88c48a48-4ff5-41eb-9e13-97993d7bf2cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" Jul 07 19:35:50.527296 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:35:50.527267 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-836c8d-openshift-default-65c87bb9d7-n97cr" Jul 07 19:35:50.528218 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:35:50.528202 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-836c8d-openshift-default-65c87bb9d7-n97cr" Jul 07 19:35:55.565588 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:35:55.565543 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:35:55.664211 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:35:55.664177 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:36:10.756647 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:36:10.756571 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-836c8d/auth-test-model-836c8d-kserve-5fd8d6c875-sl6vd"] Jul 07 19:36:10.757048 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:36:10.756945 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3132cd7e-44ee-4423-af07-b4bffe6ccdd6" containerName="istio-proxy" Jul 07 19:36:10.757048 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:36:10.756974 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="3132cd7e-44ee-4423-af07-b4bffe6ccdd6" containerName="istio-proxy" Jul 07 19:36:10.760888 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:36:10.760861 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-836c8d/auth-test-model-836c8d-kserve-5fd8d6c875-sl6vd" Jul 07 19:36:10.763864 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:36:10.763833 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-836c8d\"/\"default-dockercfg-frq88\"" Jul 07 19:36:10.764122 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:36:10.764092 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-836c8d\"/\"auth-test-model-836c8d-kserve-self-signed-certs\"" Jul 07 19:36:10.764591 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:36:10.764390 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-836c8d\"/\"openshift-service-ca.crt\"" Jul 07 19:36:10.764591 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:36:10.764415 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-836c8d\"/\"kube-root-ca.crt\"" Jul 07 19:36:10.769341 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:36:10.769315 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-836c8d/auth-test-model-836c8d-kserve-5fd8d6c875-sl6vd"] Jul 07 19:36:10.796489 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:36:10.796456 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:36:10.942939 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:36:10.942878 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/adb1aa98-64e6-4994-ba60-d66d209da759-home\") pod \"auth-test-model-836c8d-kserve-5fd8d6c875-sl6vd\" (UID: \"adb1aa98-64e6-4994-ba60-d66d209da759\") " pod="ai-tenant-e2e-shared-b-836c8d/auth-test-model-836c8d-kserve-5fd8d6c875-sl6vd" Jul 07 19:36:10.942939 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:36:10.942946 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pzbkw\" (UniqueName: \"kubernetes.io/projected/adb1aa98-64e6-4994-ba60-d66d209da759-kube-api-access-pzbkw\") pod \"auth-test-model-836c8d-kserve-5fd8d6c875-sl6vd\" (UID: \"adb1aa98-64e6-4994-ba60-d66d209da759\") " pod="ai-tenant-e2e-shared-b-836c8d/auth-test-model-836c8d-kserve-5fd8d6c875-sl6vd" Jul 07 19:36:10.943222 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:36:10.943112 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/adb1aa98-64e6-4994-ba60-d66d209da759-tls-certs\") pod \"auth-test-model-836c8d-kserve-5fd8d6c875-sl6vd\" (UID: \"adb1aa98-64e6-4994-ba60-d66d209da759\") " pod="ai-tenant-e2e-shared-b-836c8d/auth-test-model-836c8d-kserve-5fd8d6c875-sl6vd" Jul 07 19:36:10.943222 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:36:10.943145 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/adb1aa98-64e6-4994-ba60-d66d209da759-tmp-dir\") pod \"auth-test-model-836c8d-kserve-5fd8d6c875-sl6vd\" (UID: \"adb1aa98-64e6-4994-ba60-d66d209da759\") " pod="ai-tenant-e2e-shared-b-836c8d/auth-test-model-836c8d-kserve-5fd8d6c875-sl6vd" Jul 07 19:36:10.943222 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:36:10.943174 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/adb1aa98-64e6-4994-ba60-d66d209da759-model-cache\") pod \"auth-test-model-836c8d-kserve-5fd8d6c875-sl6vd\" (UID: \"adb1aa98-64e6-4994-ba60-d66d209da759\") " pod="ai-tenant-e2e-shared-b-836c8d/auth-test-model-836c8d-kserve-5fd8d6c875-sl6vd" Jul 07 19:36:10.943350 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:36:10.943238 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/adb1aa98-64e6-4994-ba60-d66d209da759-dshm\") pod \"auth-test-model-836c8d-kserve-5fd8d6c875-sl6vd\" (UID: \"adb1aa98-64e6-4994-ba60-d66d209da759\") " pod="ai-tenant-e2e-shared-b-836c8d/auth-test-model-836c8d-kserve-5fd8d6c875-sl6vd" Jul 07 19:36:11.044702 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:36:11.044653 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/adb1aa98-64e6-4994-ba60-d66d209da759-model-cache\") pod \"auth-test-model-836c8d-kserve-5fd8d6c875-sl6vd\" (UID: \"adb1aa98-64e6-4994-ba60-d66d209da759\") " pod="ai-tenant-e2e-shared-b-836c8d/auth-test-model-836c8d-kserve-5fd8d6c875-sl6vd" Jul 07 19:36:11.044879 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:36:11.044734 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/adb1aa98-64e6-4994-ba60-d66d209da759-dshm\") pod \"auth-test-model-836c8d-kserve-5fd8d6c875-sl6vd\" (UID: \"adb1aa98-64e6-4994-ba60-d66d209da759\") " pod="ai-tenant-e2e-shared-b-836c8d/auth-test-model-836c8d-kserve-5fd8d6c875-sl6vd" Jul 07 19:36:11.044879 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:36:11.044770 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/adb1aa98-64e6-4994-ba60-d66d209da759-home\") pod \"auth-test-model-836c8d-kserve-5fd8d6c875-sl6vd\" (UID: \"adb1aa98-64e6-4994-ba60-d66d209da759\") " pod="ai-tenant-e2e-shared-b-836c8d/auth-test-model-836c8d-kserve-5fd8d6c875-sl6vd" Jul 07 19:36:11.044879 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:36:11.044789 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzbkw\" (UniqueName: \"kubernetes.io/projected/adb1aa98-64e6-4994-ba60-d66d209da759-kube-api-access-pzbkw\") pod \"auth-test-model-836c8d-kserve-5fd8d6c875-sl6vd\" (UID: \"adb1aa98-64e6-4994-ba60-d66d209da759\") " pod="ai-tenant-e2e-shared-b-836c8d/auth-test-model-836c8d-kserve-5fd8d6c875-sl6vd" Jul 07 19:36:11.044879 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:36:11.044858 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/adb1aa98-64e6-4994-ba60-d66d209da759-tls-certs\") pod \"auth-test-model-836c8d-kserve-5fd8d6c875-sl6vd\" (UID: \"adb1aa98-64e6-4994-ba60-d66d209da759\") " pod="ai-tenant-e2e-shared-b-836c8d/auth-test-model-836c8d-kserve-5fd8d6c875-sl6vd" Jul 07 19:36:11.044879 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:36:11.044877 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/adb1aa98-64e6-4994-ba60-d66d209da759-tmp-dir\") pod \"auth-test-model-836c8d-kserve-5fd8d6c875-sl6vd\" (UID: \"adb1aa98-64e6-4994-ba60-d66d209da759\") " pod="ai-tenant-e2e-shared-b-836c8d/auth-test-model-836c8d-kserve-5fd8d6c875-sl6vd" Jul 07 19:36:11.045175 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:36:11.045120 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/adb1aa98-64e6-4994-ba60-d66d209da759-model-cache\") pod \"auth-test-model-836c8d-kserve-5fd8d6c875-sl6vd\" (UID: \"adb1aa98-64e6-4994-ba60-d66d209da759\") " pod="ai-tenant-e2e-shared-b-836c8d/auth-test-model-836c8d-kserve-5fd8d6c875-sl6vd" Jul 07 19:36:11.045175 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:36:11.045144 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/adb1aa98-64e6-4994-ba60-d66d209da759-home\") pod \"auth-test-model-836c8d-kserve-5fd8d6c875-sl6vd\" (UID: \"adb1aa98-64e6-4994-ba60-d66d209da759\") " pod="ai-tenant-e2e-shared-b-836c8d/auth-test-model-836c8d-kserve-5fd8d6c875-sl6vd" Jul 07 19:36:11.045355 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:36:11.045327 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/adb1aa98-64e6-4994-ba60-d66d209da759-tmp-dir\") pod \"auth-test-model-836c8d-kserve-5fd8d6c875-sl6vd\" (UID: \"adb1aa98-64e6-4994-ba60-d66d209da759\") " pod="ai-tenant-e2e-shared-b-836c8d/auth-test-model-836c8d-kserve-5fd8d6c875-sl6vd" Jul 07 19:36:11.047200 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:36:11.047180 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/adb1aa98-64e6-4994-ba60-d66d209da759-dshm\") pod \"auth-test-model-836c8d-kserve-5fd8d6c875-sl6vd\" (UID: \"adb1aa98-64e6-4994-ba60-d66d209da759\") " pod="ai-tenant-e2e-shared-b-836c8d/auth-test-model-836c8d-kserve-5fd8d6c875-sl6vd" Jul 07 19:36:11.047375 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:36:11.047358 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/adb1aa98-64e6-4994-ba60-d66d209da759-tls-certs\") pod \"auth-test-model-836c8d-kserve-5fd8d6c875-sl6vd\" (UID: \"adb1aa98-64e6-4994-ba60-d66d209da759\") " pod="ai-tenant-e2e-shared-b-836c8d/auth-test-model-836c8d-kserve-5fd8d6c875-sl6vd" Jul 07 19:36:11.055547 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:36:11.055524 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pzbkw\" (UniqueName: \"kubernetes.io/projected/adb1aa98-64e6-4994-ba60-d66d209da759-kube-api-access-pzbkw\") pod \"auth-test-model-836c8d-kserve-5fd8d6c875-sl6vd\" (UID: \"adb1aa98-64e6-4994-ba60-d66d209da759\") " pod="ai-tenant-e2e-shared-b-836c8d/auth-test-model-836c8d-kserve-5fd8d6c875-sl6vd" Jul 07 19:36:11.072788 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:36:11.072761 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-836c8d/auth-test-model-836c8d-kserve-5fd8d6c875-sl6vd" Jul 07 19:36:11.215005 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:36:11.214977 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-836c8d/auth-test-model-836c8d-kserve-5fd8d6c875-sl6vd"] Jul 07 19:36:11.216647 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:36:11.216608 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podadb1aa98_64e6_4994_ba60_d66d209da759.slice/crio-bb01e4c1fe094dee8441e4b353faba4cc051cd7c95e970d4020d6cd9dab1924e WatchSource:0}: Error finding container bb01e4c1fe094dee8441e4b353faba4cc051cd7c95e970d4020d6cd9dab1924e: Status 404 returned error can't find the container with id bb01e4c1fe094dee8441e4b353faba4cc051cd7c95e970d4020d6cd9dab1924e Jul 07 19:36:12.024785 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:36:12.024753 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-836c8d/auth-test-model-836c8d-kserve-5fd8d6c875-sl6vd" event={"ID":"adb1aa98-64e6-4994-ba60-d66d209da759","Type":"ContainerStarted","Data":"d64216118fc8c3fcbcb7cf514890f030430e761946b666aac514e9ec9e58b4a0"} Jul 07 19:36:12.024785 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:36:12.024790 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-836c8d/auth-test-model-836c8d-kserve-5fd8d6c875-sl6vd" event={"ID":"adb1aa98-64e6-4994-ba60-d66d209da759","Type":"ContainerStarted","Data":"bb01e4c1fe094dee8441e4b353faba4cc051cd7c95e970d4020d6cd9dab1924e"} Jul 07 19:36:12.044112 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:36:12.044061 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-836c8d/auth-test-model-836c8d-kserve-5fd8d6c875-sl6vd" podStartSLOduration=2.044047041 podStartE2EDuration="2.044047041s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:36:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:36:12.042457428 +0000 UTC m=+2155.058461611" watchObservedRunningTime="2026-07-07 19:36:12.044047041 +0000 UTC m=+2155.060051287" Jul 07 19:36:16.408664 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:36:16.408632 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:36:17.706187 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:36:17.706158 2563 scope.go:117] "RemoveContainer" containerID="96643a40295df8cafdf50455bb2eb0943ca8dfe1b197c20a22f681923f565d95" Jul 07 19:36:21.073458 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:36:21.073421 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-836c8d/auth-test-model-836c8d-kserve-5fd8d6c875-sl6vd" Jul 07 19:36:21.073871 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:36:21.073474 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-836c8d/auth-test-model-836c8d-kserve-5fd8d6c875-sl6vd" Jul 07 19:36:21.086090 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:36:21.086060 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-836c8d/auth-test-model-836c8d-kserve-5fd8d6c875-sl6vd" Jul 07 19:36:22.086670 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:36:22.086640 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-836c8d/auth-test-model-836c8d-kserve-5fd8d6c875-sl6vd" Jul 07 19:36:26.410200 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:36:26.410166 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:36:26.489354 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:36:26.489319 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:38:09.275725 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:38:09.275690 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:38:10.082526 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:38:10.082494 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-836c8d/auth-test-model-836c8d-kserve-5fd8d6c875-sl6vd"] Jul 07 19:38:10.476105 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:38:10.476023 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:38:25.103721 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:38:25.103666 2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-836c8d/auth-test-model-836c8d-kserve-5fd8d6c875-sl6vd" podUID="adb1aa98-64e6-4994-ba60-d66d209da759" containerName="main" containerID="cri-o://d64216118fc8c3fcbcb7cf514890f030430e761946b666aac514e9ec9e58b4a0" gracePeriod=45 Jul 07 19:38:25.355774 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:38:25.355720 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-836c8d/auth-test-model-836c8d-kserve-5fd8d6c875-sl6vd" Jul 07 19:38:25.487867 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:38:25.487830 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pzbkw\" (UniqueName: \"kubernetes.io/projected/adb1aa98-64e6-4994-ba60-d66d209da759-kube-api-access-pzbkw\") pod \"adb1aa98-64e6-4994-ba60-d66d209da759\" (UID: \"adb1aa98-64e6-4994-ba60-d66d209da759\") " Jul 07 19:38:25.488073 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:38:25.487892 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/adb1aa98-64e6-4994-ba60-d66d209da759-model-cache\") pod \"adb1aa98-64e6-4994-ba60-d66d209da759\" (UID: \"adb1aa98-64e6-4994-ba60-d66d209da759\") " Jul 07 19:38:25.488073 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:38:25.487921 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/adb1aa98-64e6-4994-ba60-d66d209da759-home\") pod \"adb1aa98-64e6-4994-ba60-d66d209da759\" (UID: \"adb1aa98-64e6-4994-ba60-d66d209da759\") " Jul 07 19:38:25.488073 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:38:25.488031 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/adb1aa98-64e6-4994-ba60-d66d209da759-dshm\") pod \"adb1aa98-64e6-4994-ba60-d66d209da759\" (UID: \"adb1aa98-64e6-4994-ba60-d66d209da759\") " Jul 07 19:38:25.488209 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:38:25.488080 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/adb1aa98-64e6-4994-ba60-d66d209da759-tmp-dir\") pod \"adb1aa98-64e6-4994-ba60-d66d209da759\" (UID: \"adb1aa98-64e6-4994-ba60-d66d209da759\") " Jul 07 19:38:25.488209 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:38:25.488130 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/adb1aa98-64e6-4994-ba60-d66d209da759-tls-certs\") pod \"adb1aa98-64e6-4994-ba60-d66d209da759\" (UID: \"adb1aa98-64e6-4994-ba60-d66d209da759\") " Jul 07 19:38:25.488323 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:38:25.488186 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/adb1aa98-64e6-4994-ba60-d66d209da759-model-cache" (OuterVolumeSpecName: "model-cache") pod "adb1aa98-64e6-4994-ba60-d66d209da759" (UID: "adb1aa98-64e6-4994-ba60-d66d209da759"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:38:25.488323 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:38:25.488260 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/adb1aa98-64e6-4994-ba60-d66d209da759-home" (OuterVolumeSpecName: "home") pod "adb1aa98-64e6-4994-ba60-d66d209da759" (UID: "adb1aa98-64e6-4994-ba60-d66d209da759"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:38:25.488433 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:38:25.488395 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/adb1aa98-64e6-4994-ba60-d66d209da759-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "adb1aa98-64e6-4994-ba60-d66d209da759" (UID: "adb1aa98-64e6-4994-ba60-d66d209da759"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:38:25.488573 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:38:25.488534 2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/adb1aa98-64e6-4994-ba60-d66d209da759-tmp-dir\") on node \"ip-10-0-142-134.ec2.internal\" DevicePath \"\"" Jul 07 19:38:25.488573 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:38:25.488552 2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/adb1aa98-64e6-4994-ba60-d66d209da759-model-cache\") on node \"ip-10-0-142-134.ec2.internal\" DevicePath \"\"" Jul 07 19:38:25.488573 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:38:25.488565 2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/adb1aa98-64e6-4994-ba60-d66d209da759-home\") on node \"ip-10-0-142-134.ec2.internal\" DevicePath \"\"" Jul 07 19:38:25.490174 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:38:25.490142 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/adb1aa98-64e6-4994-ba60-d66d209da759-dshm" (OuterVolumeSpecName: "dshm") pod "adb1aa98-64e6-4994-ba60-d66d209da759" (UID: "adb1aa98-64e6-4994-ba60-d66d209da759"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:38:25.490275 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:38:25.490223 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/adb1aa98-64e6-4994-ba60-d66d209da759-kube-api-access-pzbkw" (OuterVolumeSpecName: "kube-api-access-pzbkw") pod "adb1aa98-64e6-4994-ba60-d66d209da759" (UID: "adb1aa98-64e6-4994-ba60-d66d209da759"). InnerVolumeSpecName "kube-api-access-pzbkw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 19:38:25.490275 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:38:25.490232 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/adb1aa98-64e6-4994-ba60-d66d209da759-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "adb1aa98-64e6-4994-ba60-d66d209da759" (UID: "adb1aa98-64e6-4994-ba60-d66d209da759"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 19:38:25.589538 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:38:25.589501 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pzbkw\" (UniqueName: \"kubernetes.io/projected/adb1aa98-64e6-4994-ba60-d66d209da759-kube-api-access-pzbkw\") on node \"ip-10-0-142-134.ec2.internal\" DevicePath \"\"" Jul 07 19:38:25.589538 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:38:25.589538 2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/adb1aa98-64e6-4994-ba60-d66d209da759-dshm\") on node \"ip-10-0-142-134.ec2.internal\" DevicePath \"\"" Jul 07 19:38:25.589538 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:38:25.589552 2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/adb1aa98-64e6-4994-ba60-d66d209da759-tls-certs\") on node \"ip-10-0-142-134.ec2.internal\" DevicePath \"\"" Jul 07 19:38:25.705289 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:38:25.705193 2563 generic.go:356] "Generic (PLEG): container finished" podID="adb1aa98-64e6-4994-ba60-d66d209da759" containerID="d64216118fc8c3fcbcb7cf514890f030430e761946b666aac514e9ec9e58b4a0" exitCode=0 Jul 07 19:38:25.705289 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:38:25.705240 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-836c8d/auth-test-model-836c8d-kserve-5fd8d6c875-sl6vd" Jul 07 19:38:25.705477 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:38:25.705283 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-836c8d/auth-test-model-836c8d-kserve-5fd8d6c875-sl6vd" event={"ID":"adb1aa98-64e6-4994-ba60-d66d209da759","Type":"ContainerDied","Data":"d64216118fc8c3fcbcb7cf514890f030430e761946b666aac514e9ec9e58b4a0"} Jul 07 19:38:25.705477 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:38:25.705329 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-836c8d/auth-test-model-836c8d-kserve-5fd8d6c875-sl6vd" event={"ID":"adb1aa98-64e6-4994-ba60-d66d209da759","Type":"ContainerDied","Data":"bb01e4c1fe094dee8441e4b353faba4cc051cd7c95e970d4020d6cd9dab1924e"} Jul 07 19:38:25.705477 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:38:25.705346 2563 scope.go:117] "RemoveContainer" containerID="d64216118fc8c3fcbcb7cf514890f030430e761946b666aac514e9ec9e58b4a0" Jul 07 19:38:25.715217 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:38:25.715196 2563 scope.go:117] "RemoveContainer" containerID="d64216118fc8c3fcbcb7cf514890f030430e761946b666aac514e9ec9e58b4a0" Jul 07 19:38:25.715487 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:38:25.715468 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d64216118fc8c3fcbcb7cf514890f030430e761946b666aac514e9ec9e58b4a0\": container with ID starting with d64216118fc8c3fcbcb7cf514890f030430e761946b666aac514e9ec9e58b4a0 not found: ID does not exist" containerID="d64216118fc8c3fcbcb7cf514890f030430e761946b666aac514e9ec9e58b4a0" Jul 07 19:38:25.715527 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:38:25.715497 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d64216118fc8c3fcbcb7cf514890f030430e761946b666aac514e9ec9e58b4a0"} err="failed to get container status \"d64216118fc8c3fcbcb7cf514890f030430e761946b666aac514e9ec9e58b4a0\": rpc error: code = NotFound desc = could not find container \"d64216118fc8c3fcbcb7cf514890f030430e761946b666aac514e9ec9e58b4a0\": container with ID starting with d64216118fc8c3fcbcb7cf514890f030430e761946b666aac514e9ec9e58b4a0 not found: ID does not exist" Jul 07 19:38:25.729656 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:38:25.729624 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-836c8d/auth-test-model-836c8d-kserve-5fd8d6c875-sl6vd"] Jul 07 19:38:25.735216 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:38:25.735186 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-836c8d/auth-test-model-836c8d-kserve-5fd8d6c875-sl6vd"] Jul 07 19:38:27.477370 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:38:27.477336 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="adb1aa98-64e6-4994-ba60-d66d209da759" path="/var/lib/kubelet/pods/adb1aa98-64e6-4994-ba60-d66d209da759/volumes" Jul 07 19:38:38.205266 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:38:38.205233 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:39:00.243341 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:39:00.243307 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-836c8d/sub-test-model-836c8d-kserve-694d77dc95-94cdd"] Jul 07 19:39:00.243864 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:39:00.243695 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="adb1aa98-64e6-4994-ba60-d66d209da759" containerName="main" Jul 07 19:39:00.243864 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:39:00.243705 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="adb1aa98-64e6-4994-ba60-d66d209da759" containerName="main" Jul 07 19:39:00.243864 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:39:00.243777 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="adb1aa98-64e6-4994-ba60-d66d209da759" containerName="main" Jul 07 19:39:00.246593 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:39:00.246567 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-836c8d/sub-test-model-836c8d-kserve-694d77dc95-94cdd" Jul 07 19:39:00.248679 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:39:00.248658 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-836c8d\"/\"kube-root-ca.crt\"" Jul 07 19:39:00.249149 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:39:00.249123 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-836c8d\"/\"default-dockercfg-frq88\"" Jul 07 19:39:00.249149 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:39:00.249143 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-836c8d\"/\"openshift-service-ca.crt\"" Jul 07 19:39:00.249275 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:39:00.249163 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-836c8d\"/\"sub-test-model-836c8d-kserve-self-signed-certs\"" Jul 07 19:39:00.257615 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:39:00.257582 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-836c8d/sub-test-model-836c8d-kserve-694d77dc95-94cdd"] Jul 07 19:39:00.333672 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:39:00.333636 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2grkp\" (UniqueName: \"kubernetes.io/projected/21803477-3e8d-446c-8e9f-d4a74f71f491-kube-api-access-2grkp\") pod \"sub-test-model-836c8d-kserve-694d77dc95-94cdd\" (UID: \"21803477-3e8d-446c-8e9f-d4a74f71f491\") " pod="ai-tenant-e2e-shared-b-836c8d/sub-test-model-836c8d-kserve-694d77dc95-94cdd" Jul 07 19:39:00.333877 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:39:00.333715 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/21803477-3e8d-446c-8e9f-d4a74f71f491-tls-certs\") pod \"sub-test-model-836c8d-kserve-694d77dc95-94cdd\" (UID: \"21803477-3e8d-446c-8e9f-d4a74f71f491\") " pod="ai-tenant-e2e-shared-b-836c8d/sub-test-model-836c8d-kserve-694d77dc95-94cdd" Jul 07 19:39:00.333877 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:39:00.333757 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/21803477-3e8d-446c-8e9f-d4a74f71f491-dshm\") pod \"sub-test-model-836c8d-kserve-694d77dc95-94cdd\" (UID: \"21803477-3e8d-446c-8e9f-d4a74f71f491\") " pod="ai-tenant-e2e-shared-b-836c8d/sub-test-model-836c8d-kserve-694d77dc95-94cdd" Jul 07 19:39:00.333877 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:39:00.333795 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/21803477-3e8d-446c-8e9f-d4a74f71f491-model-cache\") pod \"sub-test-model-836c8d-kserve-694d77dc95-94cdd\" (UID: \"21803477-3e8d-446c-8e9f-d4a74f71f491\") " pod="ai-tenant-e2e-shared-b-836c8d/sub-test-model-836c8d-kserve-694d77dc95-94cdd" Jul 07 19:39:00.333877 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:39:00.333822 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/21803477-3e8d-446c-8e9f-d4a74f71f491-home\") pod \"sub-test-model-836c8d-kserve-694d77dc95-94cdd\" (UID: \"21803477-3e8d-446c-8e9f-d4a74f71f491\") " pod="ai-tenant-e2e-shared-b-836c8d/sub-test-model-836c8d-kserve-694d77dc95-94cdd" Jul 07 19:39:00.334094 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:39:00.333887 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/21803477-3e8d-446c-8e9f-d4a74f71f491-tmp-dir\") pod \"sub-test-model-836c8d-kserve-694d77dc95-94cdd\" (UID: \"21803477-3e8d-446c-8e9f-d4a74f71f491\") " pod="ai-tenant-e2e-shared-b-836c8d/sub-test-model-836c8d-kserve-694d77dc95-94cdd" Jul 07 19:39:00.362444 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:39:00.362406 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:39:00.435378 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:39:00.435342 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/21803477-3e8d-446c-8e9f-d4a74f71f491-dshm\") pod \"sub-test-model-836c8d-kserve-694d77dc95-94cdd\" (UID: \"21803477-3e8d-446c-8e9f-d4a74f71f491\") " pod="ai-tenant-e2e-shared-b-836c8d/sub-test-model-836c8d-kserve-694d77dc95-94cdd" Jul 07 19:39:00.435559 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:39:00.435388 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/21803477-3e8d-446c-8e9f-d4a74f71f491-model-cache\") pod \"sub-test-model-836c8d-kserve-694d77dc95-94cdd\" (UID: \"21803477-3e8d-446c-8e9f-d4a74f71f491\") " pod="ai-tenant-e2e-shared-b-836c8d/sub-test-model-836c8d-kserve-694d77dc95-94cdd" Jul 07 19:39:00.435559 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:39:00.435410 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/21803477-3e8d-446c-8e9f-d4a74f71f491-home\") pod \"sub-test-model-836c8d-kserve-694d77dc95-94cdd\" (UID: \"21803477-3e8d-446c-8e9f-d4a74f71f491\") " pod="ai-tenant-e2e-shared-b-836c8d/sub-test-model-836c8d-kserve-694d77dc95-94cdd" Jul 07 19:39:00.435559 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:39:00.435445 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/21803477-3e8d-446c-8e9f-d4a74f71f491-tmp-dir\") pod \"sub-test-model-836c8d-kserve-694d77dc95-94cdd\" (UID: \"21803477-3e8d-446c-8e9f-d4a74f71f491\") " pod="ai-tenant-e2e-shared-b-836c8d/sub-test-model-836c8d-kserve-694d77dc95-94cdd" Jul 07 19:39:00.435559 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:39:00.435494 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2grkp\" (UniqueName: \"kubernetes.io/projected/21803477-3e8d-446c-8e9f-d4a74f71f491-kube-api-access-2grkp\") pod \"sub-test-model-836c8d-kserve-694d77dc95-94cdd\" (UID: \"21803477-3e8d-446c-8e9f-d4a74f71f491\") " pod="ai-tenant-e2e-shared-b-836c8d/sub-test-model-836c8d-kserve-694d77dc95-94cdd" Jul 07 19:39:00.435559 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:39:00.435548 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/21803477-3e8d-446c-8e9f-d4a74f71f491-tls-certs\") pod \"sub-test-model-836c8d-kserve-694d77dc95-94cdd\" (UID: \"21803477-3e8d-446c-8e9f-d4a74f71f491\") " pod="ai-tenant-e2e-shared-b-836c8d/sub-test-model-836c8d-kserve-694d77dc95-94cdd" Jul 07 19:39:00.435876 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:39:00.435841 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/21803477-3e8d-446c-8e9f-d4a74f71f491-model-cache\") pod \"sub-test-model-836c8d-kserve-694d77dc95-94cdd\" (UID: \"21803477-3e8d-446c-8e9f-d4a74f71f491\") " pod="ai-tenant-e2e-shared-b-836c8d/sub-test-model-836c8d-kserve-694d77dc95-94cdd" Jul 07 19:39:00.436017 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:39:00.435853 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/21803477-3e8d-446c-8e9f-d4a74f71f491-home\") pod \"sub-test-model-836c8d-kserve-694d77dc95-94cdd\" (UID: \"21803477-3e8d-446c-8e9f-d4a74f71f491\") " pod="ai-tenant-e2e-shared-b-836c8d/sub-test-model-836c8d-kserve-694d77dc95-94cdd" Jul 07 19:39:00.436017 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:39:00.435920 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/21803477-3e8d-446c-8e9f-d4a74f71f491-tmp-dir\") pod \"sub-test-model-836c8d-kserve-694d77dc95-94cdd\" (UID: \"21803477-3e8d-446c-8e9f-d4a74f71f491\") " pod="ai-tenant-e2e-shared-b-836c8d/sub-test-model-836c8d-kserve-694d77dc95-94cdd" Jul 07 19:39:00.437677 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:39:00.437650 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/21803477-3e8d-446c-8e9f-d4a74f71f491-dshm\") pod \"sub-test-model-836c8d-kserve-694d77dc95-94cdd\" (UID: \"21803477-3e8d-446c-8e9f-d4a74f71f491\") " pod="ai-tenant-e2e-shared-b-836c8d/sub-test-model-836c8d-kserve-694d77dc95-94cdd" Jul 07 19:39:00.438088 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:39:00.438062 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/21803477-3e8d-446c-8e9f-d4a74f71f491-tls-certs\") pod \"sub-test-model-836c8d-kserve-694d77dc95-94cdd\" (UID: \"21803477-3e8d-446c-8e9f-d4a74f71f491\") " pod="ai-tenant-e2e-shared-b-836c8d/sub-test-model-836c8d-kserve-694d77dc95-94cdd" Jul 07 19:39:00.446528 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:39:00.446494 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2grkp\" (UniqueName: \"kubernetes.io/projected/21803477-3e8d-446c-8e9f-d4a74f71f491-kube-api-access-2grkp\") pod \"sub-test-model-836c8d-kserve-694d77dc95-94cdd\" (UID: \"21803477-3e8d-446c-8e9f-d4a74f71f491\") " pod="ai-tenant-e2e-shared-b-836c8d/sub-test-model-836c8d-kserve-694d77dc95-94cdd" Jul 07 19:39:00.557135 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:39:00.557104 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-836c8d/sub-test-model-836c8d-kserve-694d77dc95-94cdd" Jul 07 19:39:00.695106 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:39:00.694929 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-836c8d/sub-test-model-836c8d-kserve-694d77dc95-94cdd"] Jul 07 19:39:00.698436 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:39:00.698400 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod21803477_3e8d_446c_8e9f_d4a74f71f491.slice/crio-1bea7c230b1acd4a94aa8fbca477d0f33ff32cea3d188237dbeae9cddcb09eeb WatchSource:0}: Error finding container 1bea7c230b1acd4a94aa8fbca477d0f33ff32cea3d188237dbeae9cddcb09eeb: Status 404 returned error can't find the container with id 1bea7c230b1acd4a94aa8fbca477d0f33ff32cea3d188237dbeae9cddcb09eeb Jul 07 19:39:00.885740 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:39:00.885648 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-836c8d/sub-test-model-836c8d-kserve-694d77dc95-94cdd" event={"ID":"21803477-3e8d-446c-8e9f-d4a74f71f491","Type":"ContainerStarted","Data":"fed21f0f639487b08146ae5836d309158dc9cc3e6fa129a43a7f983251b6488f"} Jul 07 19:39:00.885740 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:39:00.885702 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-836c8d/sub-test-model-836c8d-kserve-694d77dc95-94cdd" event={"ID":"21803477-3e8d-446c-8e9f-d4a74f71f491","Type":"ContainerStarted","Data":"1bea7c230b1acd4a94aa8fbca477d0f33ff32cea3d188237dbeae9cddcb09eeb"} Jul 07 19:39:00.904835 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:39:00.904759 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-836c8d/sub-test-model-836c8d-kserve-694d77dc95-94cdd" podStartSLOduration=0.904742475 podStartE2EDuration="904.742475ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:39:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:39:00.903130578 +0000 UTC m=+2323.919134759" watchObservedRunningTime="2026-07-07 19:39:00.904742475 +0000 UTC m=+2323.920746697" Jul 07 19:39:10.558289 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:39:10.558203 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-836c8d/sub-test-model-836c8d-kserve-694d77dc95-94cdd" Jul 07 19:39:10.558289 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:39:10.558252 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-836c8d/sub-test-model-836c8d-kserve-694d77dc95-94cdd" Jul 07 19:39:10.571238 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:39:10.571209 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-836c8d/sub-test-model-836c8d-kserve-694d77dc95-94cdd" Jul 07 19:39:10.944889 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:39:10.944801 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-836c8d/sub-test-model-836c8d-kserve-694d77dc95-94cdd" Jul 07 19:39:27.555484 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:39:27.555439 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:39:27.658468 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:39:27.658433 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:39:34.107129 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:39:34.107093 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:39:34.989433 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:39:34.989389 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-836c8d/sub-test-model-836c8d-kserve-694d77dc95-94cdd"] Jul 07 19:39:35.014458 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:39:35.014414 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:39:50.011859 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:39:50.011819 2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-836c8d/sub-test-model-836c8d-kserve-694d77dc95-94cdd" podUID="21803477-3e8d-446c-8e9f-d4a74f71f491" containerName="main" containerID="cri-o://fed21f0f639487b08146ae5836d309158dc9cc3e6fa129a43a7f983251b6488f" gracePeriod=45 Jul 07 19:39:50.144833 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:39:50.144803 2563 generic.go:356] "Generic (PLEG): container finished" podID="21803477-3e8d-446c-8e9f-d4a74f71f491" containerID="fed21f0f639487b08146ae5836d309158dc9cc3e6fa129a43a7f983251b6488f" exitCode=0 Jul 07 19:39:50.145027 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:39:50.144869 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-836c8d/sub-test-model-836c8d-kserve-694d77dc95-94cdd" event={"ID":"21803477-3e8d-446c-8e9f-d4a74f71f491","Type":"ContainerDied","Data":"fed21f0f639487b08146ae5836d309158dc9cc3e6fa129a43a7f983251b6488f"} Jul 07 19:39:50.277377 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:39:50.277304 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-836c8d/sub-test-model-836c8d-kserve-694d77dc95-94cdd" Jul 07 19:39:50.342815 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:39:50.342775 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/21803477-3e8d-446c-8e9f-d4a74f71f491-tls-certs\") pod \"21803477-3e8d-446c-8e9f-d4a74f71f491\" (UID: \"21803477-3e8d-446c-8e9f-d4a74f71f491\") " Jul 07 19:39:50.342815 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:39:50.342825 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2grkp\" (UniqueName: \"kubernetes.io/projected/21803477-3e8d-446c-8e9f-d4a74f71f491-kube-api-access-2grkp\") pod \"21803477-3e8d-446c-8e9f-d4a74f71f491\" (UID: \"21803477-3e8d-446c-8e9f-d4a74f71f491\") " Jul 07 19:39:50.343135 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:39:50.342854 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/21803477-3e8d-446c-8e9f-d4a74f71f491-home\") pod \"21803477-3e8d-446c-8e9f-d4a74f71f491\" (UID: \"21803477-3e8d-446c-8e9f-d4a74f71f491\") " Jul 07 19:39:50.343135 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:39:50.342938 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/21803477-3e8d-446c-8e9f-d4a74f71f491-dshm\") pod \"21803477-3e8d-446c-8e9f-d4a74f71f491\" (UID: \"21803477-3e8d-446c-8e9f-d4a74f71f491\") " Jul 07 19:39:50.343135 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:39:50.343032 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/21803477-3e8d-446c-8e9f-d4a74f71f491-model-cache\") pod \"21803477-3e8d-446c-8e9f-d4a74f71f491\" (UID: \"21803477-3e8d-446c-8e9f-d4a74f71f491\") " Jul 07 19:39:50.343135 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:39:50.343053 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/21803477-3e8d-446c-8e9f-d4a74f71f491-tmp-dir\") pod \"21803477-3e8d-446c-8e9f-d4a74f71f491\" (UID: \"21803477-3e8d-446c-8e9f-d4a74f71f491\") " Jul 07 19:39:50.343345 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:39:50.343220 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21803477-3e8d-446c-8e9f-d4a74f71f491-home" (OuterVolumeSpecName: "home") pod "21803477-3e8d-446c-8e9f-d4a74f71f491" (UID: "21803477-3e8d-446c-8e9f-d4a74f71f491"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:39:50.343388 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:39:50.343309 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21803477-3e8d-446c-8e9f-d4a74f71f491-model-cache" (OuterVolumeSpecName: "model-cache") pod "21803477-3e8d-446c-8e9f-d4a74f71f491" (UID: "21803477-3e8d-446c-8e9f-d4a74f71f491"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:39:50.343430 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:39:50.343383 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21803477-3e8d-446c-8e9f-d4a74f71f491-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "21803477-3e8d-446c-8e9f-d4a74f71f491" (UID: "21803477-3e8d-446c-8e9f-d4a74f71f491"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:39:50.343500 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:39:50.343481 2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/21803477-3e8d-446c-8e9f-d4a74f71f491-model-cache\") on node \"ip-10-0-142-134.ec2.internal\" DevicePath \"\"" Jul 07 19:39:50.343544 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:39:50.343509 2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/21803477-3e8d-446c-8e9f-d4a74f71f491-tmp-dir\") on node \"ip-10-0-142-134.ec2.internal\" DevicePath \"\"" Jul 07 19:39:50.343544 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:39:50.343526 2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/21803477-3e8d-446c-8e9f-d4a74f71f491-home\") on node \"ip-10-0-142-134.ec2.internal\" DevicePath \"\"" Jul 07 19:39:50.345099 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:39:50.345076 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21803477-3e8d-446c-8e9f-d4a74f71f491-dshm" (OuterVolumeSpecName: "dshm") pod "21803477-3e8d-446c-8e9f-d4a74f71f491" (UID: "21803477-3e8d-446c-8e9f-d4a74f71f491"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:39:50.345204 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:39:50.345115 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21803477-3e8d-446c-8e9f-d4a74f71f491-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "21803477-3e8d-446c-8e9f-d4a74f71f491" (UID: "21803477-3e8d-446c-8e9f-d4a74f71f491"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 19:39:50.345204 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:39:50.345129 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21803477-3e8d-446c-8e9f-d4a74f71f491-kube-api-access-2grkp" (OuterVolumeSpecName: "kube-api-access-2grkp") pod "21803477-3e8d-446c-8e9f-d4a74f71f491" (UID: "21803477-3e8d-446c-8e9f-d4a74f71f491"). InnerVolumeSpecName "kube-api-access-2grkp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 19:39:50.444917 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:39:50.444874 2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/21803477-3e8d-446c-8e9f-d4a74f71f491-dshm\") on node \"ip-10-0-142-134.ec2.internal\" DevicePath \"\"" Jul 07 19:39:50.444917 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:39:50.444908 2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/21803477-3e8d-446c-8e9f-d4a74f71f491-tls-certs\") on node \"ip-10-0-142-134.ec2.internal\" DevicePath \"\"" Jul 07 19:39:50.444917 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:39:50.444921 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2grkp\" (UniqueName: \"kubernetes.io/projected/21803477-3e8d-446c-8e9f-d4a74f71f491-kube-api-access-2grkp\") on node \"ip-10-0-142-134.ec2.internal\" DevicePath \"\"" Jul 07 19:39:51.158557 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:39:51.158527 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-836c8d/sub-test-model-836c8d-kserve-694d77dc95-94cdd" Jul 07 19:39:51.159074 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:39:51.158526 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-836c8d/sub-test-model-836c8d-kserve-694d77dc95-94cdd" event={"ID":"21803477-3e8d-446c-8e9f-d4a74f71f491","Type":"ContainerDied","Data":"1bea7c230b1acd4a94aa8fbca477d0f33ff32cea3d188237dbeae9cddcb09eeb"} Jul 07 19:39:51.159074 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:39:51.158645 2563 scope.go:117] "RemoveContainer" containerID="fed21f0f639487b08146ae5836d309158dc9cc3e6fa129a43a7f983251b6488f" Jul 07 19:39:51.180424 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:39:51.180401 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-836c8d/sub-test-model-836c8d-kserve-694d77dc95-94cdd"] Jul 07 19:39:51.185040 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:39:51.185021 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-836c8d/sub-test-model-836c8d-kserve-694d77dc95-94cdd"] Jul 07 19:39:51.476334 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:39:51.476253 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="21803477-3e8d-446c-8e9f-d4a74f71f491" path="/var/lib/kubelet/pods/21803477-3e8d-446c-8e9f-d4a74f71f491/volumes" Jul 07 19:39:55.853879 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:39:55.853845 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-0e4903/rate-test-model-0e4903-kserve-56cbfdfb67-lhljm"] Jul 07 19:39:55.854303 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:39:55.854280 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="21803477-3e8d-446c-8e9f-d4a74f71f491" containerName="main" Jul 07 19:39:55.854303 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:39:55.854294 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="21803477-3e8d-446c-8e9f-d4a74f71f491" containerName="main" Jul 07 19:39:55.854382 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:39:55.854361 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="21803477-3e8d-446c-8e9f-d4a74f71f491" containerName="main" Jul 07 19:39:55.857530 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:39:55.857511 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-0e4903/rate-test-model-0e4903-kserve-56cbfdfb67-lhljm" Jul 07 19:39:55.860061 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:39:55.860040 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-0e4903\"/\"kube-root-ca.crt\"" Jul 07 19:39:55.860146 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:39:55.860058 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-0e4903\"/\"rate-test-model-0e4903-kserve-self-signed-certs\"" Jul 07 19:39:55.860146 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:39:55.860073 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-0e4903\"/\"openshift-service-ca.crt\"" Jul 07 19:39:55.860146 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:39:55.860052 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-0e4903\"/\"default-dockercfg-b9fbb\"" Jul 07 19:39:55.868662 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:39:55.868633 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-0e4903/rate-test-model-0e4903-kserve-56cbfdfb67-lhljm"] Jul 07 19:39:55.898183 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:39:55.898143 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4p4d\" (UniqueName: \"kubernetes.io/projected/efc984fa-8288-4774-98eb-4ffb7c82d0fc-kube-api-access-p4p4d\") pod \"rate-test-model-0e4903-kserve-56cbfdfb67-lhljm\" (UID: \"efc984fa-8288-4774-98eb-4ffb7c82d0fc\") " pod="ai-tenant-e2e-shared-a-0e4903/rate-test-model-0e4903-kserve-56cbfdfb67-lhljm" Jul 07 19:39:55.898401 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:39:55.898199 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/efc984fa-8288-4774-98eb-4ffb7c82d0fc-model-cache\") pod \"rate-test-model-0e4903-kserve-56cbfdfb67-lhljm\" (UID: \"efc984fa-8288-4774-98eb-4ffb7c82d0fc\") " pod="ai-tenant-e2e-shared-a-0e4903/rate-test-model-0e4903-kserve-56cbfdfb67-lhljm" Jul 07 19:39:55.898401 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:39:55.898264 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/efc984fa-8288-4774-98eb-4ffb7c82d0fc-dshm\") pod \"rate-test-model-0e4903-kserve-56cbfdfb67-lhljm\" (UID: \"efc984fa-8288-4774-98eb-4ffb7c82d0fc\") " pod="ai-tenant-e2e-shared-a-0e4903/rate-test-model-0e4903-kserve-56cbfdfb67-lhljm" Jul 07 19:39:55.898401 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:39:55.898309 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/efc984fa-8288-4774-98eb-4ffb7c82d0fc-home\") pod \"rate-test-model-0e4903-kserve-56cbfdfb67-lhljm\" (UID: \"efc984fa-8288-4774-98eb-4ffb7c82d0fc\") " pod="ai-tenant-e2e-shared-a-0e4903/rate-test-model-0e4903-kserve-56cbfdfb67-lhljm" Jul 07 19:39:55.898401 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:39:55.898338 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/efc984fa-8288-4774-98eb-4ffb7c82d0fc-tmp-dir\") pod \"rate-test-model-0e4903-kserve-56cbfdfb67-lhljm\" (UID: \"efc984fa-8288-4774-98eb-4ffb7c82d0fc\") " pod="ai-tenant-e2e-shared-a-0e4903/rate-test-model-0e4903-kserve-56cbfdfb67-lhljm" Jul 07 19:39:55.898401 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:39:55.898359 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/efc984fa-8288-4774-98eb-4ffb7c82d0fc-tls-certs\") pod \"rate-test-model-0e4903-kserve-56cbfdfb67-lhljm\" (UID: \"efc984fa-8288-4774-98eb-4ffb7c82d0fc\") " pod="ai-tenant-e2e-shared-a-0e4903/rate-test-model-0e4903-kserve-56cbfdfb67-lhljm" Jul 07 19:39:55.907316 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:39:55.907257 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:39:55.999658 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:39:55.999622 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/efc984fa-8288-4774-98eb-4ffb7c82d0fc-dshm\") pod \"rate-test-model-0e4903-kserve-56cbfdfb67-lhljm\" (UID: \"efc984fa-8288-4774-98eb-4ffb7c82d0fc\") " pod="ai-tenant-e2e-shared-a-0e4903/rate-test-model-0e4903-kserve-56cbfdfb67-lhljm" Jul 07 19:39:55.999853 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:39:55.999683 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/efc984fa-8288-4774-98eb-4ffb7c82d0fc-home\") pod \"rate-test-model-0e4903-kserve-56cbfdfb67-lhljm\" (UID: \"efc984fa-8288-4774-98eb-4ffb7c82d0fc\") " pod="ai-tenant-e2e-shared-a-0e4903/rate-test-model-0e4903-kserve-56cbfdfb67-lhljm" Jul 07 19:39:55.999853 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:39:55.999716 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/efc984fa-8288-4774-98eb-4ffb7c82d0fc-tmp-dir\") pod \"rate-test-model-0e4903-kserve-56cbfdfb67-lhljm\" (UID: \"efc984fa-8288-4774-98eb-4ffb7c82d0fc\") " pod="ai-tenant-e2e-shared-a-0e4903/rate-test-model-0e4903-kserve-56cbfdfb67-lhljm" Jul 07 19:39:55.999853 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:39:55.999745 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/efc984fa-8288-4774-98eb-4ffb7c82d0fc-tls-certs\") pod \"rate-test-model-0e4903-kserve-56cbfdfb67-lhljm\" (UID: \"efc984fa-8288-4774-98eb-4ffb7c82d0fc\") " pod="ai-tenant-e2e-shared-a-0e4903/rate-test-model-0e4903-kserve-56cbfdfb67-lhljm" Jul 07 19:39:55.999853 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:39:55.999815 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4p4d\" (UniqueName: \"kubernetes.io/projected/efc984fa-8288-4774-98eb-4ffb7c82d0fc-kube-api-access-p4p4d\") pod \"rate-test-model-0e4903-kserve-56cbfdfb67-lhljm\" (UID: \"efc984fa-8288-4774-98eb-4ffb7c82d0fc\") " pod="ai-tenant-e2e-shared-a-0e4903/rate-test-model-0e4903-kserve-56cbfdfb67-lhljm" Jul 07 19:39:56.000130 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:39:55.999864 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/efc984fa-8288-4774-98eb-4ffb7c82d0fc-model-cache\") pod \"rate-test-model-0e4903-kserve-56cbfdfb67-lhljm\" (UID: \"efc984fa-8288-4774-98eb-4ffb7c82d0fc\") " pod="ai-tenant-e2e-shared-a-0e4903/rate-test-model-0e4903-kserve-56cbfdfb67-lhljm" Jul 07 19:39:56.000209 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:39:56.000183 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/efc984fa-8288-4774-98eb-4ffb7c82d0fc-home\") pod \"rate-test-model-0e4903-kserve-56cbfdfb67-lhljm\" (UID: \"efc984fa-8288-4774-98eb-4ffb7c82d0fc\") " pod="ai-tenant-e2e-shared-a-0e4903/rate-test-model-0e4903-kserve-56cbfdfb67-lhljm" Jul 07 19:39:56.000271 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:39:56.000220 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/efc984fa-8288-4774-98eb-4ffb7c82d0fc-tmp-dir\") pod \"rate-test-model-0e4903-kserve-56cbfdfb67-lhljm\" (UID: \"efc984fa-8288-4774-98eb-4ffb7c82d0fc\") " pod="ai-tenant-e2e-shared-a-0e4903/rate-test-model-0e4903-kserve-56cbfdfb67-lhljm" Jul 07 19:39:56.000271 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:39:56.000234 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/efc984fa-8288-4774-98eb-4ffb7c82d0fc-model-cache\") pod \"rate-test-model-0e4903-kserve-56cbfdfb67-lhljm\" (UID: \"efc984fa-8288-4774-98eb-4ffb7c82d0fc\") " pod="ai-tenant-e2e-shared-a-0e4903/rate-test-model-0e4903-kserve-56cbfdfb67-lhljm" Jul 07 19:39:56.002017 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:39:56.001996 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/efc984fa-8288-4774-98eb-4ffb7c82d0fc-dshm\") pod \"rate-test-model-0e4903-kserve-56cbfdfb67-lhljm\" (UID: \"efc984fa-8288-4774-98eb-4ffb7c82d0fc\") " pod="ai-tenant-e2e-shared-a-0e4903/rate-test-model-0e4903-kserve-56cbfdfb67-lhljm" Jul 07 19:39:56.002235 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:39:56.002218 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/efc984fa-8288-4774-98eb-4ffb7c82d0fc-tls-certs\") pod \"rate-test-model-0e4903-kserve-56cbfdfb67-lhljm\" (UID: \"efc984fa-8288-4774-98eb-4ffb7c82d0fc\") " pod="ai-tenant-e2e-shared-a-0e4903/rate-test-model-0e4903-kserve-56cbfdfb67-lhljm" Jul 07 19:39:56.011053 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:39:56.011023 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4p4d\" (UniqueName: \"kubernetes.io/projected/efc984fa-8288-4774-98eb-4ffb7c82d0fc-kube-api-access-p4p4d\") pod \"rate-test-model-0e4903-kserve-56cbfdfb67-lhljm\" (UID: \"efc984fa-8288-4774-98eb-4ffb7c82d0fc\") " pod="ai-tenant-e2e-shared-a-0e4903/rate-test-model-0e4903-kserve-56cbfdfb67-lhljm" Jul 07 19:39:56.167711 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:39:56.167620 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-0e4903/rate-test-model-0e4903-kserve-56cbfdfb67-lhljm" Jul 07 19:39:56.309010 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:39:56.308984 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-0e4903/rate-test-model-0e4903-kserve-56cbfdfb67-lhljm"] Jul 07 19:39:56.311223 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:39:56.311194 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podefc984fa_8288_4774_98eb_4ffb7c82d0fc.slice/crio-54b1c3c6f58aa090780ead912a499800a876fa776bcd0fd2deb9363e3babdd77 WatchSource:0}: Error finding container 54b1c3c6f58aa090780ead912a499800a876fa776bcd0fd2deb9363e3babdd77: Status 404 returned error can't find the container with id 54b1c3c6f58aa090780ead912a499800a876fa776bcd0fd2deb9363e3babdd77 Jul 07 19:39:57.192461 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:39:57.192422 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-0e4903/rate-test-model-0e4903-kserve-56cbfdfb67-lhljm" event={"ID":"efc984fa-8288-4774-98eb-4ffb7c82d0fc","Type":"ContainerStarted","Data":"4c37186446e50e9b535696750f8771f4cca43652030a38036910c4bc235df9f3"} Jul 07 19:39:57.192897 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:39:57.192470 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-0e4903/rate-test-model-0e4903-kserve-56cbfdfb67-lhljm" event={"ID":"efc984fa-8288-4774-98eb-4ffb7c82d0fc","Type":"ContainerStarted","Data":"54b1c3c6f58aa090780ead912a499800a876fa776bcd0fd2deb9363e3babdd77"} Jul 07 19:39:57.211220 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:39:57.211156 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-0e4903/rate-test-model-0e4903-kserve-56cbfdfb67-lhljm" podStartSLOduration=2.21113928 podStartE2EDuration="2.21113928s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:39:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:39:57.209816284 +0000 UTC m=+2380.225820465" watchObservedRunningTime="2026-07-07 19:39:57.21113928 +0000 UTC m=+2380.227143460" Jul 07 19:40:00.982158 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:40:00.982103 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:40:06.168459 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:40:06.168420 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-0e4903/rate-test-model-0e4903-kserve-56cbfdfb67-lhljm" Jul 07 19:40:06.169031 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:40:06.168474 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-0e4903/rate-test-model-0e4903-kserve-56cbfdfb67-lhljm" Jul 07 19:40:06.180931 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:40:06.180900 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-0e4903/rate-test-model-0e4903-kserve-56cbfdfb67-lhljm" Jul 07 19:40:06.251029 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:40:06.251000 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-0e4903/rate-test-model-0e4903-kserve-56cbfdfb67-lhljm" Jul 07 19:40:11.873068 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:40:11.873036 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:40:17.646562 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:40:17.646533 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-frt7c_e25ab82f-72fb-4cde-89c4-0a0dd80f5feb/console-operator/2.log" Jul 07 19:40:17.653016 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:40:17.652991 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zzwbq_4f0c5d5c-147f-426a-965d-8886e5e79984/ovn-acl-logging/0.log" Jul 07 19:40:17.655913 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:40:17.655889 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-frt7c_e25ab82f-72fb-4cde-89c4-0a0dd80f5feb/console-operator/2.log" Jul 07 19:40:17.662091 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:40:17.662069 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zzwbq_4f0c5d5c-147f-426a-965d-8886e5e79984/ovn-acl-logging/0.log" Jul 07 19:40:20.408280 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:40:20.408246 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:40:49.576047 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:40:49.576011 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:40:49.674188 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:40:49.674156 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:40:56.697316 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:40:56.697277 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:40:56.797574 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:40:56.797535 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:41:01.615755 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:41:01.615722 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-0e4903/rate-test-model-0e4903-kserve-56cbfdfb67-lhljm"] Jul 07 19:41:01.653058 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:41:01.653028 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:41:02.498525 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:41:02.498459 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:41:03.615821 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:41:03.615782 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:41:04.003887 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:41:04.003785 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:41:04.128578 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:41:04.128540 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:41:04.226595 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:41:04.226560 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:41:04.430572 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:41:04.430540 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:41:09.161935 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:41:09.161899 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:41:16.638824 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:41:16.638767 2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-0e4903/rate-test-model-0e4903-kserve-56cbfdfb67-lhljm" podUID="efc984fa-8288-4774-98eb-4ffb7c82d0fc" containerName="main" containerID="cri-o://4c37186446e50e9b535696750f8771f4cca43652030a38036910c4bc235df9f3" gracePeriod=45 Jul 07 19:41:16.900305 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:41:16.900235 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-0e4903/rate-test-model-0e4903-kserve-56cbfdfb67-lhljm" Jul 07 19:41:17.012162 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:41:17.012121 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/efc984fa-8288-4774-98eb-4ffb7c82d0fc-tmp-dir\") pod \"efc984fa-8288-4774-98eb-4ffb7c82d0fc\" (UID: \"efc984fa-8288-4774-98eb-4ffb7c82d0fc\") " Jul 07 19:41:17.012339 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:41:17.012179 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/efc984fa-8288-4774-98eb-4ffb7c82d0fc-home\") pod \"efc984fa-8288-4774-98eb-4ffb7c82d0fc\" (UID: \"efc984fa-8288-4774-98eb-4ffb7c82d0fc\") " Jul 07 19:41:17.012339 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:41:17.012266 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/efc984fa-8288-4774-98eb-4ffb7c82d0fc-tls-certs\") pod \"efc984fa-8288-4774-98eb-4ffb7c82d0fc\" (UID: \"efc984fa-8288-4774-98eb-4ffb7c82d0fc\") " Jul 07 19:41:17.012339 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:41:17.012303 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/efc984fa-8288-4774-98eb-4ffb7c82d0fc-dshm\") pod \"efc984fa-8288-4774-98eb-4ffb7c82d0fc\" (UID: \"efc984fa-8288-4774-98eb-4ffb7c82d0fc\") " Jul 07 19:41:17.012536 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:41:17.012363 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/efc984fa-8288-4774-98eb-4ffb7c82d0fc-model-cache\") pod \"efc984fa-8288-4774-98eb-4ffb7c82d0fc\" (UID: \"efc984fa-8288-4774-98eb-4ffb7c82d0fc\") " Jul 07 19:41:17.012536 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:41:17.012403 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p4p4d\" (UniqueName: \"kubernetes.io/projected/efc984fa-8288-4774-98eb-4ffb7c82d0fc-kube-api-access-p4p4d\") pod \"efc984fa-8288-4774-98eb-4ffb7c82d0fc\" (UID: \"efc984fa-8288-4774-98eb-4ffb7c82d0fc\") " Jul 07 19:41:17.012536 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:41:17.012440 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efc984fa-8288-4774-98eb-4ffb7c82d0fc-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "efc984fa-8288-4774-98eb-4ffb7c82d0fc" (UID: "efc984fa-8288-4774-98eb-4ffb7c82d0fc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:41:17.012536 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:41:17.012457 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efc984fa-8288-4774-98eb-4ffb7c82d0fc-home" (OuterVolumeSpecName: "home") pod "efc984fa-8288-4774-98eb-4ffb7c82d0fc" (UID: "efc984fa-8288-4774-98eb-4ffb7c82d0fc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:41:17.012748 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:41:17.012683 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efc984fa-8288-4774-98eb-4ffb7c82d0fc-model-cache" (OuterVolumeSpecName: "model-cache") pod "efc984fa-8288-4774-98eb-4ffb7c82d0fc" (UID: "efc984fa-8288-4774-98eb-4ffb7c82d0fc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:41:17.012787 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:41:17.012746 2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/efc984fa-8288-4774-98eb-4ffb7c82d0fc-tmp-dir\") on node \"ip-10-0-142-134.ec2.internal\" DevicePath \"\"" Jul 07 19:41:17.012787 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:41:17.012765 2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/efc984fa-8288-4774-98eb-4ffb7c82d0fc-home\") on node \"ip-10-0-142-134.ec2.internal\" DevicePath \"\"" Jul 07 19:41:17.014617 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:41:17.014595 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/efc984fa-8288-4774-98eb-4ffb7c82d0fc-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "efc984fa-8288-4774-98eb-4ffb7c82d0fc" (UID: "efc984fa-8288-4774-98eb-4ffb7c82d0fc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 19:41:17.014698 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:41:17.014645 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efc984fa-8288-4774-98eb-4ffb7c82d0fc-dshm" (OuterVolumeSpecName: "dshm") pod "efc984fa-8288-4774-98eb-4ffb7c82d0fc" (UID: "efc984fa-8288-4774-98eb-4ffb7c82d0fc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:41:17.014738 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:41:17.014696 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/efc984fa-8288-4774-98eb-4ffb7c82d0fc-kube-api-access-p4p4d" (OuterVolumeSpecName: "kube-api-access-p4p4d") pod "efc984fa-8288-4774-98eb-4ffb7c82d0fc" (UID: "efc984fa-8288-4774-98eb-4ffb7c82d0fc"). InnerVolumeSpecName "kube-api-access-p4p4d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 19:41:17.113850 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:41:17.113803 2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/efc984fa-8288-4774-98eb-4ffb7c82d0fc-tls-certs\") on node \"ip-10-0-142-134.ec2.internal\" DevicePath \"\"" Jul 07 19:41:17.113850 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:41:17.113833 2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/efc984fa-8288-4774-98eb-4ffb7c82d0fc-dshm\") on node \"ip-10-0-142-134.ec2.internal\" DevicePath \"\"" Jul 07 19:41:17.113850 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:41:17.113841 2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/efc984fa-8288-4774-98eb-4ffb7c82d0fc-model-cache\") on node \"ip-10-0-142-134.ec2.internal\" DevicePath \"\"" Jul 07 19:41:17.113850 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:41:17.113850 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p4p4d\" (UniqueName: \"kubernetes.io/projected/efc984fa-8288-4774-98eb-4ffb7c82d0fc-kube-api-access-p4p4d\") on node \"ip-10-0-142-134.ec2.internal\" DevicePath \"\"" Jul 07 19:41:17.616036 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:41:17.615991 2563 generic.go:356] "Generic (PLEG): container finished" podID="efc984fa-8288-4774-98eb-4ffb7c82d0fc" containerID="4c37186446e50e9b535696750f8771f4cca43652030a38036910c4bc235df9f3" exitCode=0 Jul 07 19:41:17.616229 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:41:17.616057 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-0e4903/rate-test-model-0e4903-kserve-56cbfdfb67-lhljm" Jul 07 19:41:17.616229 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:41:17.616083 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-0e4903/rate-test-model-0e4903-kserve-56cbfdfb67-lhljm" event={"ID":"efc984fa-8288-4774-98eb-4ffb7c82d0fc","Type":"ContainerDied","Data":"4c37186446e50e9b535696750f8771f4cca43652030a38036910c4bc235df9f3"} Jul 07 19:41:17.616229 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:41:17.616131 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-0e4903/rate-test-model-0e4903-kserve-56cbfdfb67-lhljm" event={"ID":"efc984fa-8288-4774-98eb-4ffb7c82d0fc","Type":"ContainerDied","Data":"54b1c3c6f58aa090780ead912a499800a876fa776bcd0fd2deb9363e3babdd77"} Jul 07 19:41:17.616229 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:41:17.616153 2563 scope.go:117] "RemoveContainer" containerID="4c37186446e50e9b535696750f8771f4cca43652030a38036910c4bc235df9f3" Jul 07 19:41:17.625250 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:41:17.625226 2563 scope.go:117] "RemoveContainer" containerID="4c37186446e50e9b535696750f8771f4cca43652030a38036910c4bc235df9f3" Jul 07 19:41:17.625554 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:41:17.625515 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c37186446e50e9b535696750f8771f4cca43652030a38036910c4bc235df9f3\": container with ID starting with 4c37186446e50e9b535696750f8771f4cca43652030a38036910c4bc235df9f3 not found: ID does not exist" containerID="4c37186446e50e9b535696750f8771f4cca43652030a38036910c4bc235df9f3" Jul 07 19:41:17.625620 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:41:17.625561 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c37186446e50e9b535696750f8771f4cca43652030a38036910c4bc235df9f3"} err="failed to get container status \"4c37186446e50e9b535696750f8771f4cca43652030a38036910c4bc235df9f3\": rpc error: code = NotFound desc = could not find container \"4c37186446e50e9b535696750f8771f4cca43652030a38036910c4bc235df9f3\": container with ID starting with 4c37186446e50e9b535696750f8771f4cca43652030a38036910c4bc235df9f3 not found: ID does not exist" Jul 07 19:41:17.639357 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:41:17.639327 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-0e4903/rate-test-model-0e4903-kserve-56cbfdfb67-lhljm"] Jul 07 19:41:17.645747 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:41:17.645720 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-0e4903/rate-test-model-0e4903-kserve-56cbfdfb67-lhljm"] Jul 07 19:41:17.769637 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:41:17.769555 2563 scope.go:117] "RemoveContainer" containerID="6a4628c57545963d71d30837f4fe0c9ca69eea25b0375b4a15ddc594ba10d57b" Jul 07 19:41:19.476599 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:41:19.476524 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="efc984fa-8288-4774-98eb-4ffb7c82d0fc" path="/var/lib/kubelet/pods/efc984fa-8288-4774-98eb-4ffb7c82d0fc/volumes" Jul 07 19:41:47.825076 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:41:47.825030 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:41:48.107496 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:41:48.107411 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:41:48.334268 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:41:48.334231 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:41:48.514798 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:41:48.514712 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:41:48.898477 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:41:48.898436 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:41:48.998997 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:41:48.998968 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:41:49.099003 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:41:49.098976 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:41:49.287802 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:41:49.287759 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:41:56.684036 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:41:56.684005 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:41:56.784046 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:41:56.783998 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:41:56.952339 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:41:56.952252 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-836c8d-openshift-default-65c87bb9d7-n97cr"] Jul 07 19:41:56.952540 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:41:56.952518 2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-836c8d-openshift-default-65c87bb9d7-n97cr" podUID="88c48a48-4ff5-41eb-9e13-97993d7bf2cb" containerName="istio-proxy" containerID="cri-o://18d2d31af4a753da83f1c2562593b4f9bfdbece2ae693c63acbd4321e366b777" gracePeriod=30 Jul 07 19:41:56.979564 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:41:56.979524 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:41:57.237451 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:41:57.237355 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:41:59.262705 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:41:59.262674 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:42:02.219174 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:42:02.219102 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-836c8d-openshift-default-65c87bb9d7-n97cr" Jul 07 19:42:02.310536 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:42:02.310500 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-zl9w5_f71dae8a-03c2-4f96-ad16-895ff28cd975/manager/0.log" Jul 07 19:42:02.363994 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:42:02.363933 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88c48a48-4ff5-41eb-9e13-97993d7bf2cb-istio-token\") pod \"88c48a48-4ff5-41eb-9e13-97993d7bf2cb\" (UID: \"88c48a48-4ff5-41eb-9e13-97993d7bf2cb\") " Jul 07 19:42:02.363994 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:42:02.364002 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88c48a48-4ff5-41eb-9e13-97993d7bf2cb-credential-socket\") pod \"88c48a48-4ff5-41eb-9e13-97993d7bf2cb\" (UID: \"88c48a48-4ff5-41eb-9e13-97993d7bf2cb\") " Jul 07 19:42:02.364209 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:42:02.364079 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88c48a48-4ff5-41eb-9e13-97993d7bf2cb-istiod-ca-cert\") pod \"88c48a48-4ff5-41eb-9e13-97993d7bf2cb\" (UID: \"88c48a48-4ff5-41eb-9e13-97993d7bf2cb\") " Jul 07 19:42:02.364209 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:42:02.364105 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d5fwh\" (UniqueName: \"kubernetes.io/projected/88c48a48-4ff5-41eb-9e13-97993d7bf2cb-kube-api-access-d5fwh\") pod \"88c48a48-4ff5-41eb-9e13-97993d7bf2cb\" (UID: \"88c48a48-4ff5-41eb-9e13-97993d7bf2cb\") " Jul 07 19:42:02.364209 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:42:02.364146 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88c48a48-4ff5-41eb-9e13-97993d7bf2cb-workload-socket\") pod \"88c48a48-4ff5-41eb-9e13-97993d7bf2cb\" (UID: \"88c48a48-4ff5-41eb-9e13-97993d7bf2cb\") " Jul 07 19:42:02.364209 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:42:02.364170 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88c48a48-4ff5-41eb-9e13-97993d7bf2cb-istio-envoy\") pod \"88c48a48-4ff5-41eb-9e13-97993d7bf2cb\" (UID: \"88c48a48-4ff5-41eb-9e13-97993d7bf2cb\") " Jul 07 19:42:02.364209 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:42:02.364197 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88c48a48-4ff5-41eb-9e13-97993d7bf2cb-workload-certs\") pod \"88c48a48-4ff5-41eb-9e13-97993d7bf2cb\" (UID: \"88c48a48-4ff5-41eb-9e13-97993d7bf2cb\") " Jul 07 19:42:02.364454 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:42:02.364220 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88c48a48-4ff5-41eb-9e13-97993d7bf2cb-istio-data\") pod \"88c48a48-4ff5-41eb-9e13-97993d7bf2cb\" (UID: \"88c48a48-4ff5-41eb-9e13-97993d7bf2cb\") " Jul 07 19:42:02.364454 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:42:02.364268 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88c48a48-4ff5-41eb-9e13-97993d7bf2cb-istio-podinfo\") pod \"88c48a48-4ff5-41eb-9e13-97993d7bf2cb\" (UID: \"88c48a48-4ff5-41eb-9e13-97993d7bf2cb\") " Jul 07 19:42:02.364454 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:42:02.364316 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88c48a48-4ff5-41eb-9e13-97993d7bf2cb-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "88c48a48-4ff5-41eb-9e13-97993d7bf2cb" (UID: "88c48a48-4ff5-41eb-9e13-97993d7bf2cb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:42:02.364640 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:42:02.364516 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88c48a48-4ff5-41eb-9e13-97993d7bf2cb-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "88c48a48-4ff5-41eb-9e13-97993d7bf2cb" (UID: "88c48a48-4ff5-41eb-9e13-97993d7bf2cb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:42:02.364640 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:42:02.364593 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88c48a48-4ff5-41eb-9e13-97993d7bf2cb-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "88c48a48-4ff5-41eb-9e13-97993d7bf2cb" (UID: "88c48a48-4ff5-41eb-9e13-97993d7bf2cb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 19:42:02.364640 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:42:02.364628 2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88c48a48-4ff5-41eb-9e13-97993d7bf2cb-workload-certs\") on node \"ip-10-0-142-134.ec2.internal\" DevicePath \"\"" Jul 07 19:42:02.364795 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:42:02.364648 2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88c48a48-4ff5-41eb-9e13-97993d7bf2cb-credential-socket\") on node \"ip-10-0-142-134.ec2.internal\" DevicePath \"\"" Jul 07 19:42:02.364795 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:42:02.364768 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88c48a48-4ff5-41eb-9e13-97993d7bf2cb-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "88c48a48-4ff5-41eb-9e13-97993d7bf2cb" (UID: "88c48a48-4ff5-41eb-9e13-97993d7bf2cb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:42:02.365193 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:42:02.365167 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88c48a48-4ff5-41eb-9e13-97993d7bf2cb-istio-data" (OuterVolumeSpecName: "istio-data") pod "88c48a48-4ff5-41eb-9e13-97993d7bf2cb" (UID: "88c48a48-4ff5-41eb-9e13-97993d7bf2cb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:42:02.366329 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:42:02.366303 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88c48a48-4ff5-41eb-9e13-97993d7bf2cb-istio-token" (OuterVolumeSpecName: "istio-token") pod "88c48a48-4ff5-41eb-9e13-97993d7bf2cb" (UID: "88c48a48-4ff5-41eb-9e13-97993d7bf2cb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 19:42:02.366453 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:42:02.366396 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88c48a48-4ff5-41eb-9e13-97993d7bf2cb-kube-api-access-d5fwh" (OuterVolumeSpecName: "kube-api-access-d5fwh") pod "88c48a48-4ff5-41eb-9e13-97993d7bf2cb" (UID: "88c48a48-4ff5-41eb-9e13-97993d7bf2cb"). InnerVolumeSpecName "kube-api-access-d5fwh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 19:42:02.366552 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:42:02.366536 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/88c48a48-4ff5-41eb-9e13-97993d7bf2cb-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "88c48a48-4ff5-41eb-9e13-97993d7bf2cb" (UID: "88c48a48-4ff5-41eb-9e13-97993d7bf2cb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 07 19:42:02.366596 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:42:02.366546 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88c48a48-4ff5-41eb-9e13-97993d7bf2cb-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "88c48a48-4ff5-41eb-9e13-97993d7bf2cb" (UID: "88c48a48-4ff5-41eb-9e13-97993d7bf2cb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:42:02.465902 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:42:02.465859 2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88c48a48-4ff5-41eb-9e13-97993d7bf2cb-istio-token\") on node \"ip-10-0-142-134.ec2.internal\" DevicePath \"\"" Jul 07 19:42:02.465902 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:42:02.465898 2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88c48a48-4ff5-41eb-9e13-97993d7bf2cb-istiod-ca-cert\") on node \"ip-10-0-142-134.ec2.internal\" DevicePath \"\"" Jul 07 19:42:02.465902 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:42:02.465911 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d5fwh\" (UniqueName: \"kubernetes.io/projected/88c48a48-4ff5-41eb-9e13-97993d7bf2cb-kube-api-access-d5fwh\") on node \"ip-10-0-142-134.ec2.internal\" DevicePath \"\"" Jul 07 19:42:02.466183 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:42:02.465924 2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88c48a48-4ff5-41eb-9e13-97993d7bf2cb-workload-socket\") on node \"ip-10-0-142-134.ec2.internal\" DevicePath \"\"" Jul 07 19:42:02.466183 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:42:02.465935 2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88c48a48-4ff5-41eb-9e13-97993d7bf2cb-istio-envoy\") on node \"ip-10-0-142-134.ec2.internal\" DevicePath \"\"" Jul 07 19:42:02.466183 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:42:02.465945 2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88c48a48-4ff5-41eb-9e13-97993d7bf2cb-istio-data\") on node \"ip-10-0-142-134.ec2.internal\" DevicePath \"\"" Jul 07 19:42:02.466183 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:42:02.465973 2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88c48a48-4ff5-41eb-9e13-97993d7bf2cb-istio-podinfo\") on node \"ip-10-0-142-134.ec2.internal\" DevicePath \"\"" Jul 07 19:42:02.837460 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:42:02.837425 2563 generic.go:356] "Generic (PLEG): container finished" podID="88c48a48-4ff5-41eb-9e13-97993d7bf2cb" containerID="18d2d31af4a753da83f1c2562593b4f9bfdbece2ae693c63acbd4321e366b777" exitCode=0 Jul 07 19:42:02.837664 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:42:02.837475 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-836c8d-openshift-default-65c87bb9d7-n97cr" Jul 07 19:42:02.837664 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:42:02.837494 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-836c8d-openshift-default-65c87bb9d7-n97cr" event={"ID":"88c48a48-4ff5-41eb-9e13-97993d7bf2cb","Type":"ContainerDied","Data":"18d2d31af4a753da83f1c2562593b4f9bfdbece2ae693c63acbd4321e366b777"} Jul 07 19:42:02.837664 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:42:02.837545 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-836c8d-openshift-default-65c87bb9d7-n97cr" event={"ID":"88c48a48-4ff5-41eb-9e13-97993d7bf2cb","Type":"ContainerDied","Data":"ea574ba3a788bff8cef3c2248b4a29bf88e5d22fffdeaa8f9b5fc8c9f391ec07"} Jul 07 19:42:02.837664 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:42:02.837577 2563 scope.go:117] "RemoveContainer" containerID="18d2d31af4a753da83f1c2562593b4f9bfdbece2ae693c63acbd4321e366b777" Jul 07 19:42:02.847059 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:42:02.847040 2563 scope.go:117] "RemoveContainer" containerID="07c50230ba4ca120f3467d115c1036b275d6e06f2abb1a5a67c8c435ba2cf3eb" Jul 07 19:42:02.857452 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:42:02.857429 2563 scope.go:117] "RemoveContainer" containerID="18d2d31af4a753da83f1c2562593b4f9bfdbece2ae693c63acbd4321e366b777" Jul 07 19:42:02.857828 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:42:02.857807 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18d2d31af4a753da83f1c2562593b4f9bfdbece2ae693c63acbd4321e366b777\": container with ID starting with 18d2d31af4a753da83f1c2562593b4f9bfdbece2ae693c63acbd4321e366b777 not found: ID does not exist" containerID="18d2d31af4a753da83f1c2562593b4f9bfdbece2ae693c63acbd4321e366b777" Jul 07 19:42:02.857879 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:42:02.857838 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18d2d31af4a753da83f1c2562593b4f9bfdbece2ae693c63acbd4321e366b777"} err="failed to get container status \"18d2d31af4a753da83f1c2562593b4f9bfdbece2ae693c63acbd4321e366b777\": rpc error: code = NotFound desc = could not find container \"18d2d31af4a753da83f1c2562593b4f9bfdbece2ae693c63acbd4321e366b777\": container with ID starting with 18d2d31af4a753da83f1c2562593b4f9bfdbece2ae693c63acbd4321e366b777 not found: ID does not exist" Jul 07 19:42:02.857879 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:42:02.857861 2563 scope.go:117] "RemoveContainer" containerID="07c50230ba4ca120f3467d115c1036b275d6e06f2abb1a5a67c8c435ba2cf3eb" Jul 07 19:42:02.858163 ip-10-0-142-134 kubenswrapper[2563]: E0707 19:42:02.858145 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07c50230ba4ca120f3467d115c1036b275d6e06f2abb1a5a67c8c435ba2cf3eb\": container with ID starting with 07c50230ba4ca120f3467d115c1036b275d6e06f2abb1a5a67c8c435ba2cf3eb not found: ID does not exist" containerID="07c50230ba4ca120f3467d115c1036b275d6e06f2abb1a5a67c8c435ba2cf3eb" Jul 07 19:42:02.858222 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:42:02.858169 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07c50230ba4ca120f3467d115c1036b275d6e06f2abb1a5a67c8c435ba2cf3eb"} err="failed to get container status \"07c50230ba4ca120f3467d115c1036b275d6e06f2abb1a5a67c8c435ba2cf3eb\": rpc error: code = NotFound desc = could not find container \"07c50230ba4ca120f3467d115c1036b275d6e06f2abb1a5a67c8c435ba2cf3eb\": container with ID starting with 07c50230ba4ca120f3467d115c1036b275d6e06f2abb1a5a67c8c435ba2cf3eb not found: ID does not exist" Jul 07 19:42:02.862546 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:42:02.862523 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-836c8d-openshift-default-65c87bb9d7-n97cr"] Jul 07 19:42:02.871010 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:42:02.870989 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-836c8d-openshift-default-65c87bb9d7-n97cr"] Jul 07 19:42:03.442227 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:42:03.442199 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-54cc9bdf65-hwc7h_db7846e2-43d5-4f0a-8cbd-3b9aa92e564a/copy-manifests/0.log" Jul 07 19:42:03.452149 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:42:03.452118 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-54cc9bdf65-hwc7h_db7846e2-43d5-4f0a-8cbd-3b9aa92e564a/manager/0.log" Jul 07 19:42:03.476893 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:42:03.476865 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="88c48a48-4ff5-41eb-9e13-97993d7bf2cb" path="/var/lib/kubelet/pods/88c48a48-4ff5-41eb-9e13-97993d7bf2cb/volumes" Jul 07 19:42:05.146859 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:42:05.146791 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-2tvsm_f242cd57-28f2-4e5b-917b-e47794e9b26e/manager/0.log" Jul 07 19:42:05.271286 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:42:05.271253 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-rhd4d_86db6a78-52ad-429e-a6df-2a623f06bf6b/manager/0.log" Jul 07 19:42:05.530773 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:42:05.530738 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-wfgtm_658ee1a1-f9fa-4fb6-ab7f-8acdab870d80/registry-server/0.log" Jul 07 19:42:05.921605 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:42:05.921506 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-q9d5b_1eb6d982-857c-49eb-a0ca-f489029bcc87/manager/0.log" Jul 07 19:42:06.286625 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:42:06.286596 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-554689689dtjt6l_1a0d31c0-5fe7-4d3e-9ee3-8266857f6931/istio-proxy/0.log" Jul 07 19:42:06.788278 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:42:06.788250 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-74949dcd89-rph7z_f9e6513c-1aa8-40da-9c2c-a0ba5ea11b69/istio-proxy/1.log" Jul 07 19:42:07.517096 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:42:07.517062 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-c622d_d5d65f82-5aca-41e4-aa46-0457819c4664/main/0.log" Jul 07 19:42:07.890102 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:42:07.890075 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db8mhk2_354ac1be-b1da-46ba-9eac-57e9cdcd7469/main/0.log" Jul 07 19:42:08.130825 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:42:08.130795 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5fc6fb456f-cg9mj_aee2eeec-b78e-481e-ad56-7577bda25769/main/0.log" Jul 07 19:42:12.116870 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:42:12.116789 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:42:12.218559 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:42:12.218517 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:42:12.498317 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:42:12.498231 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" limit=1024 Jul 07 19:42:16.031000 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:42:16.030945 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-lk7md_59f0c8fc-bc9a-4273-aac2-a8ee23bf6b63/global-pull-secret-syncer/0.log" Jul 07 19:42:16.167451 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:42:16.167417 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-dsvgs_4f25fdbd-3251-4ef6-ab69-e87cc915a469/konnectivity-agent/0.log" Jul 07 19:42:16.385615 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:42:16.385510 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-142-134.ec2.internal_d64b863190df7f2a681aceffe233fdc3/haproxy/0.log" Jul 07 19:42:21.496801 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:42:21.496766 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-2tvsm_f242cd57-28f2-4e5b-917b-e47794e9b26e/manager/0.log" Jul 07 19:42:21.589796 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:42:21.589755 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-rhd4d_86db6a78-52ad-429e-a6df-2a623f06bf6b/manager/0.log" Jul 07 19:42:21.749789 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:42:21.749692 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-wfgtm_658ee1a1-f9fa-4fb6-ab7f-8acdab870d80/registry-server/0.log" Jul 07 19:42:22.177714 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:42:22.177676 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-q9d5b_1eb6d982-857c-49eb-a0ca-f489029bcc87/manager/0.log" Jul 07 19:42:24.011908 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:42:24.011878 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_b41727ec-29e6-46d3-b63a-29141930d29b/alertmanager/0.log" Jul 07 19:42:24.066102 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:42:24.066068 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_b41727ec-29e6-46d3-b63a-29141930d29b/config-reloader/0.log" Jul 07 19:42:24.126797 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:42:24.126770 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_b41727ec-29e6-46d3-b63a-29141930d29b/kube-rbac-proxy-web/0.log" Jul 07 19:42:24.180486 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:42:24.180459 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_b41727ec-29e6-46d3-b63a-29141930d29b/kube-rbac-proxy/0.log" Jul 07 19:42:24.245806 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:42:24.245776 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_b41727ec-29e6-46d3-b63a-29141930d29b/kube-rbac-proxy-metric/0.log" Jul 07 19:42:24.298525 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:42:24.298499 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_b41727ec-29e6-46d3-b63a-29141930d29b/prom-label-proxy/0.log" Jul 07 19:42:24.342717 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:42:24.342634 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_b41727ec-29e6-46d3-b63a-29141930d29b/init-config-reloader/0.log" Jul 07 19:42:24.421260 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:42:24.421230 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-64786d9cbb-pnvkh_ecead548-45c5-4195-95e8-a5c058d5958b/cluster-monitoring-operator/0.log" Jul 07 19:42:24.600030 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:42:24.599917 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-5cb97f8f7b-stvw8_6f2e13c9-5c07-429e-850a-3816f2c24a21/metrics-server/0.log" Jul 07 19:42:25.001273 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:42:25.001192 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-vjxrs_9b37d03d-9be0-4dc0-a77a-7384e5814a4f/node-exporter/0.log" Jul 07 19:42:25.051884 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:42:25.051848 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-vjxrs_9b37d03d-9be0-4dc0-a77a-7384e5814a4f/kube-rbac-proxy/0.log" Jul 07 19:42:25.093687 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:42:25.093662 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-vjxrs_9b37d03d-9be0-4dc0-a77a-7384e5814a4f/init-textfile/0.log" Jul 07 19:42:25.262429 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:42:25.262353 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_fe61befe-7a98-40cf-b142-a435f8804125/prometheus/0.log" Jul 07 19:42:25.319080 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:42:25.319050 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_fe61befe-7a98-40cf-b142-a435f8804125/config-reloader/0.log" Jul 07 19:42:25.377017 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:42:25.376993 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_fe61befe-7a98-40cf-b142-a435f8804125/thanos-sidecar/0.log" Jul 07 19:42:25.433152 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:42:25.433113 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_fe61befe-7a98-40cf-b142-a435f8804125/kube-rbac-proxy-web/0.log" Jul 07 19:42:25.483761 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:42:25.483727 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_fe61befe-7a98-40cf-b142-a435f8804125/kube-rbac-proxy/0.log" Jul 07 19:42:25.535121 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:42:25.535098 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_fe61befe-7a98-40cf-b142-a435f8804125/kube-rbac-proxy-thanos/0.log" Jul 07 19:42:25.600004 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:42:25.599971 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_fe61befe-7a98-40cf-b142-a435f8804125/init-config-reloader/0.log" Jul 07 19:42:25.617601 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:42:25.617572 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-bhqz8/perf-node-gather-daemonset-rvjnj"] Jul 07 19:42:25.617969 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:42:25.617937 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="88c48a48-4ff5-41eb-9e13-97993d7bf2cb" containerName="istio-proxy" Jul 07 19:42:25.617969 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:42:25.617963 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="88c48a48-4ff5-41eb-9e13-97993d7bf2cb" containerName="istio-proxy" Jul 07 19:42:25.618062 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:42:25.617984 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="efc984fa-8288-4774-98eb-4ffb7c82d0fc" containerName="main" Jul 07 19:42:25.618062 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:42:25.617991 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="efc984fa-8288-4774-98eb-4ffb7c82d0fc" containerName="main" Jul 07 19:42:25.618137 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:42:25.618066 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="88c48a48-4ff5-41eb-9e13-97993d7bf2cb" containerName="istio-proxy" Jul 07 19:42:25.618137 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:42:25.618073 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="efc984fa-8288-4774-98eb-4ffb7c82d0fc" containerName="main" Jul 07 19:42:25.621250 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:42:25.621233 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-bhqz8/perf-node-gather-daemonset-rvjnj" Jul 07 19:42:25.624240 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:42:25.624212 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-bhqz8\"/\"openshift-service-ca.crt\"" Jul 07 19:42:25.624819 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:42:25.624795 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-bhqz8\"/\"default-dockercfg-b4nc4\"" Jul 07 19:42:25.624819 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:42:25.624813 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-bhqz8\"/\"kube-root-ca.crt\"" Jul 07 19:42:25.639714 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:42:25.639679 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-bhqz8/perf-node-gather-daemonset-rvjnj"] Jul 07 19:42:25.671774 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:42:25.671748 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-567f7b8b7c-dj4zk_53739ab1-4901-467f-9965-1a7f6888b580/prometheus-operator/0.log" Jul 07 19:42:25.680315 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:42:25.680292 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/1501a2d4-cd2c-4f31-b70d-0974271c3ee9-proc\") pod \"perf-node-gather-daemonset-rvjnj\" (UID: \"1501a2d4-cd2c-4f31-b70d-0974271c3ee9\") " pod="openshift-must-gather-bhqz8/perf-node-gather-daemonset-rvjnj" Jul 07 19:42:25.680426 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:42:25.680329 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1501a2d4-cd2c-4f31-b70d-0974271c3ee9-sys\") pod \"perf-node-gather-daemonset-rvjnj\" (UID: \"1501a2d4-cd2c-4f31-b70d-0974271c3ee9\") " pod="openshift-must-gather-bhqz8/perf-node-gather-daemonset-rvjnj" Jul 07 19:42:25.680426 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:42:25.680352 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tj6fw\" (UniqueName: \"kubernetes.io/projected/1501a2d4-cd2c-4f31-b70d-0974271c3ee9-kube-api-access-tj6fw\") pod \"perf-node-gather-daemonset-rvjnj\" (UID: \"1501a2d4-cd2c-4f31-b70d-0974271c3ee9\") " pod="openshift-must-gather-bhqz8/perf-node-gather-daemonset-rvjnj" Jul 07 19:42:25.680497 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:42:25.680434 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/1501a2d4-cd2c-4f31-b70d-0974271c3ee9-podres\") pod \"perf-node-gather-daemonset-rvjnj\" (UID: \"1501a2d4-cd2c-4f31-b70d-0974271c3ee9\") " pod="openshift-must-gather-bhqz8/perf-node-gather-daemonset-rvjnj" Jul 07 19:42:25.680497 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:42:25.680461 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1501a2d4-cd2c-4f31-b70d-0974271c3ee9-lib-modules\") pod \"perf-node-gather-daemonset-rvjnj\" (UID: \"1501a2d4-cd2c-4f31-b70d-0974271c3ee9\") " pod="openshift-must-gather-bhqz8/perf-node-gather-daemonset-rvjnj" Jul 07 19:42:25.709991 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:42:25.709961 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-567f7b8b7c-dj4zk_53739ab1-4901-467f-9965-1a7f6888b580/kube-rbac-proxy/0.log" Jul 07 19:42:25.781421 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:42:25.781382 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/1501a2d4-cd2c-4f31-b70d-0974271c3ee9-proc\") pod \"perf-node-gather-daemonset-rvjnj\" (UID: \"1501a2d4-cd2c-4f31-b70d-0974271c3ee9\") " pod="openshift-must-gather-bhqz8/perf-node-gather-daemonset-rvjnj" Jul 07 19:42:25.781596 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:42:25.781430 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1501a2d4-cd2c-4f31-b70d-0974271c3ee9-sys\") pod \"perf-node-gather-daemonset-rvjnj\" (UID: \"1501a2d4-cd2c-4f31-b70d-0974271c3ee9\") " pod="openshift-must-gather-bhqz8/perf-node-gather-daemonset-rvjnj" Jul 07 19:42:25.781596 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:42:25.781512 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/1501a2d4-cd2c-4f31-b70d-0974271c3ee9-proc\") pod \"perf-node-gather-daemonset-rvjnj\" (UID: \"1501a2d4-cd2c-4f31-b70d-0974271c3ee9\") " pod="openshift-must-gather-bhqz8/perf-node-gather-daemonset-rvjnj" Jul 07 19:42:25.781596 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:42:25.781554 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1501a2d4-cd2c-4f31-b70d-0974271c3ee9-sys\") pod \"perf-node-gather-daemonset-rvjnj\" (UID: \"1501a2d4-cd2c-4f31-b70d-0974271c3ee9\") " pod="openshift-must-gather-bhqz8/perf-node-gather-daemonset-rvjnj" Jul 07 19:42:25.781596 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:42:25.781580 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tj6fw\" (UniqueName: \"kubernetes.io/projected/1501a2d4-cd2c-4f31-b70d-0974271c3ee9-kube-api-access-tj6fw\") pod \"perf-node-gather-daemonset-rvjnj\" (UID: \"1501a2d4-cd2c-4f31-b70d-0974271c3ee9\") " pod="openshift-must-gather-bhqz8/perf-node-gather-daemonset-rvjnj" Jul 07 19:42:25.781756 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:42:25.781624 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/1501a2d4-cd2c-4f31-b70d-0974271c3ee9-podres\") pod \"perf-node-gather-daemonset-rvjnj\" (UID: \"1501a2d4-cd2c-4f31-b70d-0974271c3ee9\") " pod="openshift-must-gather-bhqz8/perf-node-gather-daemonset-rvjnj" Jul 07 19:42:25.781756 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:42:25.781645 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1501a2d4-cd2c-4f31-b70d-0974271c3ee9-lib-modules\") pod \"perf-node-gather-daemonset-rvjnj\" (UID: \"1501a2d4-cd2c-4f31-b70d-0974271c3ee9\") " pod="openshift-must-gather-bhqz8/perf-node-gather-daemonset-rvjnj" Jul 07 19:42:25.781756 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:42:25.781737 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/1501a2d4-cd2c-4f31-b70d-0974271c3ee9-podres\") pod \"perf-node-gather-daemonset-rvjnj\" (UID: \"1501a2d4-cd2c-4f31-b70d-0974271c3ee9\") " pod="openshift-must-gather-bhqz8/perf-node-gather-daemonset-rvjnj" Jul 07 19:42:25.781852 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:42:25.781800 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1501a2d4-cd2c-4f31-b70d-0974271c3ee9-lib-modules\") pod \"perf-node-gather-daemonset-rvjnj\" (UID: \"1501a2d4-cd2c-4f31-b70d-0974271c3ee9\") " pod="openshift-must-gather-bhqz8/perf-node-gather-daemonset-rvjnj" Jul 07 19:42:25.787664 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:42:25.787590 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5d4848bbfb-w9dk4_b43c0ae2-9e13-4beb-a30b-e2147f3d4e41/telemeter-client/0.log" Jul 07 19:42:25.793221 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:42:25.793200 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tj6fw\" (UniqueName: \"kubernetes.io/projected/1501a2d4-cd2c-4f31-b70d-0974271c3ee9-kube-api-access-tj6fw\") pod \"perf-node-gather-daemonset-rvjnj\" (UID: \"1501a2d4-cd2c-4f31-b70d-0974271c3ee9\") " pod="openshift-must-gather-bhqz8/perf-node-gather-daemonset-rvjnj" Jul 07 19:42:25.817592 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:42:25.817563 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5d4848bbfb-w9dk4_b43c0ae2-9e13-4beb-a30b-e2147f3d4e41/reload/0.log" Jul 07 19:42:25.851303 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:42:25.851274 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5d4848bbfb-w9dk4_b43c0ae2-9e13-4beb-a30b-e2147f3d4e41/kube-rbac-proxy/0.log" Jul 07 19:42:25.931716 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:42:25.931671 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-bhqz8/perf-node-gather-daemonset-rvjnj" Jul 07 19:42:26.072440 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:42:26.072356 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-bhqz8/perf-node-gather-daemonset-rvjnj"] Jul 07 19:42:26.075576 ip-10-0-142-134 kubenswrapper[2563]: W0707 19:42:26.075546 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod1501a2d4_cd2c_4f31_b70d_0974271c3ee9.slice/crio-8621a11c2ffbf0ded81be4b72a5d9a462407ca303ba01929f2f4233d171f4714 WatchSource:0}: Error finding container 8621a11c2ffbf0ded81be4b72a5d9a462407ca303ba01929f2f4233d171f4714: Status 404 returned error can't find the container with id 8621a11c2ffbf0ded81be4b72a5d9a462407ca303ba01929f2f4233d171f4714 Jul 07 19:42:26.960086 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:42:26.960046 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bhqz8/perf-node-gather-daemonset-rvjnj" event={"ID":"1501a2d4-cd2c-4f31-b70d-0974271c3ee9","Type":"ContainerStarted","Data":"ef06b58f4537f2658125bb845db10fdca9f1eaa14533ca8af9d22e117e8461e1"} Jul 07 19:42:26.960404 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:42:26.960114 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bhqz8/perf-node-gather-daemonset-rvjnj" event={"ID":"1501a2d4-cd2c-4f31-b70d-0974271c3ee9","Type":"ContainerStarted","Data":"8621a11c2ffbf0ded81be4b72a5d9a462407ca303ba01929f2f4233d171f4714"} Jul 07 19:42:26.960404 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:42:26.960220 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-bhqz8/perf-node-gather-daemonset-rvjnj" Jul 07 19:42:26.994257 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:42:26.994195 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-bhqz8/perf-node-gather-daemonset-rvjnj" podStartSLOduration=1.994179675 podStartE2EDuration="1.994179675s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 19:42:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:42:26.992000413 +0000 UTC m=+2530.008004594" watchObservedRunningTime="2026-07-07 19:42:26.994179675 +0000 UTC m=+2530.010183896" Jul 07 19:42:27.869583 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:42:27.869551 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-frt7c_e25ab82f-72fb-4cde-89c4-0a0dd80f5feb/console-operator/2.log" Jul 07 19:42:27.877072 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:42:27.877047 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-frt7c_e25ab82f-72fb-4cde-89c4-0a0dd80f5feb/console-operator/3.log" Jul 07 19:42:30.365351 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:42:30.365288 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-hq4fc_c338a925-fe6b-40a0-b669-ce49a20cc370/dns/0.log" Jul 07 19:42:30.395470 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:42:30.395444 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-hq4fc_c338a925-fe6b-40a0-b669-ce49a20cc370/kube-rbac-proxy/0.log" Jul 07 19:42:30.632291 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:42:30.632168 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-whw2v_3953dadf-98ed-4933-a330-1191906b6cff/dns-node-resolver/0.log" Jul 07 19:42:31.343515 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:42:31.343485 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-s272w_a6abe188-2b1b-427a-ba0c-e93930187c7e/node-ca/0.log" Jul 07 19:42:32.535140 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:42:32.535111 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-554689689dtjt6l_1a0d31c0-5fe7-4d3e-9ee3-8266857f6931/istio-proxy/0.log" Jul 07 19:42:32.795423 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:42:32.795323 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-74949dcd89-rph7z_f9e6513c-1aa8-40da-9c2c-a0ba5ea11b69/istio-proxy/0.log" Jul 07 19:42:32.852811 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:42:32.852778 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-74949dcd89-rph7z_f9e6513c-1aa8-40da-9c2c-a0ba5ea11b69/istio-proxy/1.log" Jul 07 19:42:32.975733 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:42:32.975701 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-bhqz8/perf-node-gather-daemonset-rvjnj" Jul 07 19:42:33.837800 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:42:33.837770 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-28nms_a6f2e7e7-5e73-40a1-a05a-d5a338128dd7/serve-healthcheck-canary/0.log" Jul 07 19:42:34.689849 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:42:34.689819 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-mb8xx_8db445ca-fea3-4e5e-b946-b4509c6427c8/kube-rbac-proxy/0.log" Jul 07 19:42:34.749563 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:42:34.749525 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-mb8xx_8db445ca-fea3-4e5e-b946-b4509c6427c8/exporter/0.log" Jul 07 19:42:34.801793 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:42:34.801761 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-mb8xx_8db445ca-fea3-4e5e-b946-b4509c6427c8/extractor/0.log" Jul 07 19:42:36.397258 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:42:36.397162 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-554689689dtjt6l_1a0d31c0-5fe7-4d3e-9ee3-8266857f6931/istio-proxy/0.log" Jul 07 19:42:36.653966 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:42:36.653848 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-74949dcd89-rph7z_f9e6513c-1aa8-40da-9c2c-a0ba5ea11b69/istio-proxy/0.log" Jul 07 19:42:36.707483 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:42:36.707449 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-74949dcd89-rph7z_f9e6513c-1aa8-40da-9c2c-a0ba5ea11b69/istio-proxy/1.log" Jul 07 19:42:37.743291 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:42:37.743263 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-zl9w5_f71dae8a-03c2-4f96-ad16-895ff28cd975/manager/0.log" Jul 07 19:42:38.125863 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:42:38.125777 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-554689689dtjt6l_1a0d31c0-5fe7-4d3e-9ee3-8266857f6931/istio-proxy/0.log" Jul 07 19:42:38.374720 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:42:38.374677 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-74949dcd89-rph7z_f9e6513c-1aa8-40da-9c2c-a0ba5ea11b69/istio-proxy/0.log" Jul 07 19:42:38.427146 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:42:38.427048 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-74949dcd89-rph7z_f9e6513c-1aa8-40da-9c2c-a0ba5ea11b69/istio-proxy/1.log" Jul 07 19:42:38.780451 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:42:38.780428 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-54cc9bdf65-hwc7h_db7846e2-43d5-4f0a-8cbd-3b9aa92e564a/manager/0.log" Jul 07 19:42:38.843806 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:42:38.843779 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-54cc9bdf65-hwc7h_db7846e2-43d5-4f0a-8cbd-3b9aa92e564a/copy-manifests/0.log" Jul 07 19:42:40.441801 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:42:40.441774 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-5f4bfc584b-jcd55_c288a2e8-a2a9-4125-a538-552328cff0a9/manager/0.log" Jul 07 19:42:40.529204 ip-10-0-142-134 kubenswrapper[2563]: I0707 19:42:40.529168 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-7pw62_4a7f9314-da36-4f36-97f0-b87b49df7c92/openshift-lws-operator/0.log"