Jul 12 19:20:28.183752 ip-10-0-139-170 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Jul 12 19:20:28.183764 ip-10-0-139-170 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Jul 12 19:20:28.183774 ip-10-0-139-170 systemd[1]: kubelet.service: Failed with result 'resources'. Jul 12 19:20:28.184075 ip-10-0-139-170 systemd[1]: Failed to start Kubernetes Kubelet. Jul 12 19:20:38.370577 ip-10-0-139-170 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Jul 12 19:20:38.370596 ip-10-0-139-170 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 86ff458c43e9482980c27f43f0c1908b -- Jul 12 19:22:13.303494 ip-10-0-139-170 systemd[1]: Starting Kubernetes Kubelet... Jul 12 19:22:13.912389 ip-10-0-139-170 kubenswrapper[2565]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 12 19:22:13.912389 ip-10-0-139-170 kubenswrapper[2565]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 12 19:22:13.912389 ip-10-0-139-170 kubenswrapper[2565]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 12 19:22:13.912389 ip-10-0-139-170 kubenswrapper[2565]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 12 19:22:13.912389 ip-10-0-139-170 kubenswrapper[2565]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 12 19:22:13.912389 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:13.751875 2565 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 12 19:22:13.912389 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.761030 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 12 19:22:13.912389 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.761050 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 12 19:22:13.912389 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.761054 2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 12 19:22:13.912389 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.761057 2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 12 19:22:13.912389 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.761060 2565 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 12 19:22:13.912389 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.761063 2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 12 19:22:13.912389 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.761066 2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 12 19:22:13.912389 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.761069 2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 12 19:22:14.069328 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.761071 2565 feature_gate.go:328] unrecognized feature gate: Example2 Jul 12 19:22:14.069328 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.761073 2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 12 19:22:14.069328 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.761076 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 12 19:22:14.069328 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.761079 2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 12 19:22:14.069328 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.761081 2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 12 19:22:14.069328 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.761084 2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 12 19:22:14.069328 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.761086 2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 12 19:22:14.069328 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.761091 2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 12 19:22:14.069328 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.761096 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 12 19:22:14.069328 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.761098 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 12 19:22:14.069328 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.761102 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 12 19:22:14.069328 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.761104 2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 12 19:22:14.069328 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.761107 2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 12 19:22:14.069328 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.761110 2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 12 19:22:14.069328 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.761113 2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 12 19:22:14.069328 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.761116 2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 12 19:22:14.069328 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.761118 2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 12 19:22:14.069328 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.761120 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 12 19:22:14.069328 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.761126 2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 12 19:22:14.069328 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.761129 2565 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 12 19:22:13.833674 ip-10-0-139-170 systemd[1]: Started Kubernetes Kubelet. Jul 12 19:22:14.070635 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.761132 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 12 19:22:14.070635 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.761135 2565 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 12 19:22:14.070635 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.761137 2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 12 19:22:14.070635 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.761140 2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 12 19:22:14.070635 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.761143 2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 12 19:22:14.070635 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.761147 2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 12 19:22:14.070635 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.761149 2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 12 19:22:14.070635 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.761152 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 12 19:22:14.070635 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.761154 2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 12 19:22:14.070635 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.761157 2565 feature_gate.go:328] unrecognized feature gate: Example Jul 12 19:22:14.070635 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.761159 2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 12 19:22:14.070635 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.761162 2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 12 19:22:14.070635 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.761165 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 12 19:22:14.070635 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.761168 2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 12 19:22:14.070635 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.761171 2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 12 19:22:14.070635 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.761174 2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 12 19:22:14.070635 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.761177 2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 12 19:22:14.070635 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.761179 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 12 19:22:14.070635 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.761182 2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 12 19:22:14.070635 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.761185 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 12 19:22:14.071553 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.761187 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 12 19:22:14.071553 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.761189 2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 12 19:22:14.071553 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.761192 2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 12 19:22:14.071553 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.761194 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 12 19:22:14.071553 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.761196 2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 12 19:22:14.071553 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.761199 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 12 19:22:14.071553 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.761201 2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 12 19:22:14.071553 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.761204 2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 12 19:22:14.071553 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.761207 2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 12 19:22:14.071553 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.761213 2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 12 19:22:14.071553 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.761236 2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 12 19:22:14.071553 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.761239 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 12 19:22:14.071553 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.761242 2565 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 12 19:22:14.071553 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.761244 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 12 19:22:14.071553 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.761247 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 12 19:22:14.071553 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.761250 2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 12 19:22:14.071553 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.761253 2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 12 19:22:14.071553 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.761256 2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 12 19:22:14.071553 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.761258 2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 12 19:22:14.072138 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.761261 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 12 19:22:14.072138 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.761264 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 12 19:22:14.072138 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.761266 2565 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 12 19:22:14.072138 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.761269 2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 12 19:22:14.072138 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.761273 2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 12 19:22:14.072138 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.761276 2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 12 19:22:14.072138 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.761279 2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 12 19:22:14.072138 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.761282 2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 12 19:22:14.072138 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.761285 2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 12 19:22:14.072138 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.761287 2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 12 19:22:14.072138 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.761290 2565 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 12 19:22:14.072138 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.761292 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 12 19:22:14.072138 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.761295 2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 12 19:22:14.072138 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.761297 2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 12 19:22:14.072138 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.761299 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 12 19:22:14.072138 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.761302 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 12 19:22:14.072138 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.761305 2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 12 19:22:14.072138 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.761308 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 12 19:22:14.072138 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.761311 2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 12 19:22:14.072138 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.761313 2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 12 19:22:14.072762 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.761316 2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 12 19:22:14.072762 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.761318 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 12 19:22:14.072762 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.761321 2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 12 19:22:14.072762 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.761323 2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 12 19:22:14.072762 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.761326 2565 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 12 19:22:14.072762 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.761331 2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 12 19:22:14.072762 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.761333 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 12 19:22:14.072762 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.761335 2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 12 19:22:14.072762 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.761338 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 12 19:22:14.072762 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.761341 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 12 19:22:14.072762 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.761343 2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 12 19:22:14.072762 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.761346 2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 12 19:22:14.072762 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.762109 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 12 19:22:14.072762 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.762116 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 12 19:22:14.072762 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.762119 2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 12 19:22:14.072762 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.762122 2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 12 19:22:14.072762 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.762125 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 12 19:22:14.072762 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.762127 2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 12 19:22:14.072762 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.762130 2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 12 19:22:14.072762 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.762132 2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 12 19:22:14.073330 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.762135 2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 12 19:22:14.073330 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.762137 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 12 19:22:14.073330 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.762140 2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 12 19:22:14.073330 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.762142 2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 12 19:22:14.073330 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.762145 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 12 19:22:14.073330 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.762147 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 12 19:22:14.073330 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.762150 2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 12 19:22:14.073330 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.762153 2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 12 19:22:14.073330 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.762155 2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 12 19:22:14.073330 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.762165 2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 12 19:22:14.073330 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.762167 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 12 19:22:14.073330 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.762170 2565 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 12 19:22:14.073330 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.762173 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 12 19:22:14.073330 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.762176 2565 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 12 19:22:14.073330 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.762179 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 12 19:22:14.073330 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.762181 2565 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 12 19:22:14.073330 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.762184 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 12 19:22:14.073330 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.762189 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 12 19:22:14.073330 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.762191 2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 12 19:22:14.073330 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.762194 2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 12 19:22:14.073951 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.762196 2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 12 19:22:14.073951 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.762199 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 12 19:22:14.073951 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.762201 2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 12 19:22:14.073951 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.762204 2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 12 19:22:14.073951 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.762207 2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 12 19:22:14.073951 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.762209 2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 12 19:22:14.073951 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.762212 2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 12 19:22:14.073951 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.762215 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 12 19:22:14.073951 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.762236 2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 12 19:22:14.073951 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.762239 2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 12 19:22:14.073951 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.762241 2565 feature_gate.go:328] unrecognized feature gate: Example Jul 12 19:22:14.073951 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.762244 2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 12 19:22:14.073951 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.762246 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 12 19:22:14.073951 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.762249 2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 12 19:22:14.073951 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.762252 2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 12 19:22:14.073951 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.762254 2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 12 19:22:14.073951 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.762257 2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 12 19:22:14.073951 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.762259 2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 12 19:22:14.073951 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.762262 2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 12 19:22:14.074505 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.762264 2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 12 19:22:14.074505 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.762267 2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 12 19:22:14.074505 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.762270 2565 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 12 19:22:14.074505 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.762279 2565 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 12 19:22:14.074505 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.762282 2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 12 19:22:14.074505 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.762286 2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 12 19:22:14.074505 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.762290 2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 12 19:22:14.074505 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.762292 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 12 19:22:14.074505 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.762295 2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 12 19:22:14.074505 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.762298 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 12 19:22:14.074505 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.762302 2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 12 19:22:14.074505 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.762305 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 12 19:22:14.074505 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.762307 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 12 19:22:14.074505 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.762310 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 12 19:22:14.074505 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.762312 2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 12 19:22:14.074505 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.762315 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 12 19:22:14.074505 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.762317 2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 12 19:22:14.074505 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.762320 2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 12 19:22:14.074505 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.762322 2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 12 19:22:14.074505 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.762325 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 12 19:22:14.075097 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.762328 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 12 19:22:14.075097 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.762331 2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 12 19:22:14.075097 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.762333 2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 12 19:22:14.075097 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.762337 2565 feature_gate.go:328] unrecognized feature gate: Example2 Jul 12 19:22:14.075097 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.762339 2565 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 12 19:22:14.075097 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.762342 2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 12 19:22:14.075097 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.762344 2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 12 19:22:14.075097 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.762347 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 12 19:22:14.075097 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.762349 2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 12 19:22:14.075097 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.762352 2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 12 19:22:14.075097 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.762354 2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 12 19:22:14.075097 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.762357 2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 12 19:22:14.075097 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.762359 2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 12 19:22:14.075097 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.762362 2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 12 19:22:14.075097 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.762365 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 12 19:22:14.075097 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.762367 2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 12 19:22:14.075097 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.762377 2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 12 19:22:14.075097 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.762380 2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 12 19:22:14.075097 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.762382 2565 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 12 19:22:14.075097 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.762386 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 12 19:22:14.075669 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.762388 2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 12 19:22:14.075669 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.762391 2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 12 19:22:14.075669 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.762396 2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 12 19:22:14.075669 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.762398 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 12 19:22:14.075669 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.762402 2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 12 19:22:14.075669 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.762406 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 12 19:22:14.075669 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.762409 2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 12 19:22:14.075669 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.762411 2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 12 19:22:14.075669 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.762414 2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 12 19:22:14.075669 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.762418 2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 12 19:22:14.075669 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.762421 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 12 19:22:14.075669 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.762423 2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 12 19:22:14.075669 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.762880 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 12 19:22:14.075669 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.762885 2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 12 19:22:14.075669 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.762896 2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 12 19:22:14.075669 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.762900 2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 12 19:22:14.075669 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.762904 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 12 19:22:14.075669 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.762907 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 12 19:22:14.075669 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.762910 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 12 19:22:14.076259 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.762914 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 12 19:22:14.076259 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.762917 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 12 19:22:14.076259 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.762920 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 12 19:22:14.076259 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.762923 2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 12 19:22:14.076259 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.762925 2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 12 19:22:14.076259 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.762928 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 12 19:22:14.076259 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.762930 2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 12 19:22:14.076259 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.762933 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 12 19:22:14.076259 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.762935 2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 12 19:22:14.076259 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.762944 2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 12 19:22:14.076259 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.762947 2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 12 19:22:14.076259 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.762951 2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 12 19:22:14.076259 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.762954 2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 12 19:22:14.076259 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.762957 2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 12 19:22:14.076259 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.762959 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 12 19:22:14.076259 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.762965 2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 12 19:22:14.076259 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.762967 2565 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 12 19:22:14.076259 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.762970 2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 12 19:22:14.076259 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.762972 2565 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 12 19:22:14.076259 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.762975 2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 12 19:22:14.076815 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.762978 2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 12 19:22:14.076815 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.762980 2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 12 19:22:14.076815 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.762983 2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 12 19:22:14.076815 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.762985 2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 12 19:22:14.076815 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.762988 2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 12 19:22:14.076815 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.762990 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 12 19:22:14.076815 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.762992 2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 12 19:22:14.076815 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.762995 2565 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 12 19:22:14.076815 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.762997 2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 12 19:22:14.076815 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.763000 2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 12 19:22:14.076815 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.763002 2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 12 19:22:14.076815 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.763005 2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 12 19:22:14.076815 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.763007 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 12 19:22:14.076815 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.763010 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 12 19:22:14.076815 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.763012 2565 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 12 19:22:14.076815 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.763015 2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 12 19:22:14.076815 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.763017 2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 12 19:22:14.076815 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.763019 2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 12 19:22:14.076815 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.763022 2565 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 12 19:22:14.077461 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.763024 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 12 19:22:14.077461 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.763027 2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 12 19:22:14.077461 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.763029 2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 12 19:22:14.077461 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.763038 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 12 19:22:14.077461 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.763041 2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 12 19:22:14.077461 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.763043 2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 12 19:22:14.077461 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.763046 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 12 19:22:14.077461 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.763050 2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 12 19:22:14.077461 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.763056 2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 12 19:22:14.077461 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.763059 2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 12 19:22:14.077461 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.763061 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 12 19:22:14.077461 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.763064 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 12 19:22:14.077461 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.763067 2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 12 19:22:14.077461 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.763070 2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 12 19:22:14.077461 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.763072 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 12 19:22:14.077461 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.763075 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 12 19:22:14.077461 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.763078 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 12 19:22:14.077461 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.763080 2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 12 19:22:14.077461 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.763082 2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 12 19:22:14.077951 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.763085 2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 12 19:22:14.077951 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.763088 2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 12 19:22:14.077951 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.763091 2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 12 19:22:14.077951 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.763094 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 12 19:22:14.077951 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.763096 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 12 19:22:14.077951 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.763099 2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 12 19:22:14.077951 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.763101 2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 12 19:22:14.077951 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.763104 2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 12 19:22:14.077951 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.763106 2565 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 12 19:22:14.077951 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.763108 2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 12 19:22:14.077951 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.763115 2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 12 19:22:14.077951 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.763117 2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 12 19:22:14.077951 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.763120 2565 feature_gate.go:328] unrecognized feature gate: Example2 Jul 12 19:22:14.077951 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.763123 2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 12 19:22:14.077951 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.763125 2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 12 19:22:14.077951 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.763127 2565 feature_gate.go:328] unrecognized feature gate: Example Jul 12 19:22:14.077951 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.763131 2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 12 19:22:14.077951 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.763140 2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 12 19:22:14.077951 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.763143 2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 12 19:22:14.077951 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.763146 2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 12 19:22:14.078521 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.763149 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 12 19:22:14.078521 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.763154 2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 12 19:22:14.078521 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.763157 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 12 19:22:14.078521 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.763160 2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 12 19:22:14.078521 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.763162 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 12 19:22:14.078521 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.763165 2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 12 19:22:14.078521 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.763167 2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 12 19:22:14.078521 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.763170 2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 12 19:22:14.078521 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.763172 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 12 19:22:14.078521 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.763174 2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 12 19:22:14.078521 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.763177 2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 12 19:22:14.078521 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.763179 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 12 19:22:14.078521 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.763182 2565 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 12 19:22:14.078521 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.763184 2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 12 19:22:14.078521 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:13.763295 2565 flags.go:64] FLAG: --address="0.0.0.0" Jul 12 19:22:14.078521 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:13.763329 2565 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 12 19:22:14.078521 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:13.763341 2565 flags.go:64] FLAG: --anonymous-auth="true" Jul 12 19:22:14.078521 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:13.763348 2565 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 12 19:22:14.078521 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:13.763353 2565 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 12 19:22:14.078521 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:13.763356 2565 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 12 19:22:14.078521 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:13.763361 2565 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 12 19:22:14.078521 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:13.763366 2565 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 12 19:22:14.079086 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:13.763372 2565 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 12 19:22:14.079086 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:13.763375 2565 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 12 19:22:14.079086 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:13.763379 2565 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 12 19:22:14.079086 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:13.763382 2565 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 12 19:22:14.079086 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:13.763385 2565 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 12 19:22:14.079086 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:13.763388 2565 flags.go:64] FLAG: --cgroup-root="" Jul 12 19:22:14.079086 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:13.763391 2565 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 12 19:22:14.079086 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:13.763394 2565 flags.go:64] FLAG: --client-ca-file="" Jul 12 19:22:14.079086 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:13.763397 2565 flags.go:64] FLAG: --cloud-provider="external" Jul 12 19:22:14.079086 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:13.763400 2565 flags.go:64] FLAG: --cluster-dns="[]" Jul 12 19:22:14.079086 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:13.763404 2565 flags.go:64] FLAG: --cluster-domain="" Jul 12 19:22:14.079086 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:13.763408 2565 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 12 19:22:14.079086 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:13.763411 2565 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 12 19:22:14.079086 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:13.763414 2565 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 12 19:22:14.079086 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:13.763417 2565 flags.go:64] FLAG: --container-log-max-files="5" Jul 12 19:22:14.079086 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:13.763421 2565 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 12 19:22:14.079086 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:13.763424 2565 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 12 19:22:14.079086 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:13.763427 2565 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 12 19:22:14.079086 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:13.763430 2565 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 12 19:22:14.079086 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:13.763433 2565 flags.go:64] FLAG: --contention-profiling="false" Jul 12 19:22:14.079086 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:13.763436 2565 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 12 19:22:14.079086 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:13.763439 2565 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 12 19:22:14.079086 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:13.763443 2565 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 12 19:22:14.079086 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:13.763446 2565 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 12 19:22:14.079772 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:13.763451 2565 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 12 19:22:14.079772 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:13.763456 2565 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 12 19:22:14.079772 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:13.763459 2565 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 12 19:22:14.079772 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:13.763462 2565 flags.go:64] FLAG: --enable-load-reader="false" Jul 12 19:22:14.079772 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:13.763465 2565 flags.go:64] FLAG: --enable-server="true" Jul 12 19:22:14.079772 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:13.763467 2565 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 12 19:22:14.079772 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:13.763473 2565 flags.go:64] FLAG: --event-burst="100" Jul 12 19:22:14.079772 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:13.763476 2565 flags.go:64] FLAG: --event-qps="50" Jul 12 19:22:14.079772 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:13.763481 2565 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 12 19:22:14.079772 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:13.763484 2565 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 12 19:22:14.079772 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:13.763487 2565 flags.go:64] FLAG: --eviction-hard="" Jul 12 19:22:14.079772 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:13.763491 2565 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 12 19:22:14.079772 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:13.763493 2565 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 12 19:22:14.079772 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:13.763497 2565 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 12 19:22:14.079772 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:13.763500 2565 flags.go:64] FLAG: --eviction-soft="" Jul 12 19:22:14.079772 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:13.763503 2565 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 12 19:22:14.079772 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:13.763506 2565 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 12 19:22:14.079772 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:13.763509 2565 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 12 19:22:14.079772 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:13.763511 2565 flags.go:64] FLAG: --experimental-mounter-path="" Jul 12 19:22:14.079772 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:13.763514 2565 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 12 19:22:14.079772 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:13.763517 2565 flags.go:64] FLAG: --fail-swap-on="true" Jul 12 19:22:14.079772 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:13.763520 2565 flags.go:64] FLAG: --feature-gates="" Jul 12 19:22:14.079772 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:13.763524 2565 flags.go:64] FLAG: --file-check-frequency="20s" Jul 12 19:22:14.079772 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:13.763527 2565 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 12 19:22:14.079772 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:13.763530 2565 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 12 19:22:14.080508 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:13.763533 2565 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 12 19:22:14.080508 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:13.763536 2565 flags.go:64] FLAG: --healthz-port="10248" Jul 12 19:22:14.080508 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:13.763539 2565 flags.go:64] FLAG: --help="false" Jul 12 19:22:14.080508 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:13.763542 2565 flags.go:64] FLAG: --hostname-override="ip-10-0-139-170.ec2.internal" Jul 12 19:22:14.080508 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:13.763545 2565 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 12 19:22:14.080508 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:13.763548 2565 flags.go:64] FLAG: --http-check-frequency="20s" Jul 12 19:22:14.080508 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:13.763551 2565 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 12 19:22:14.080508 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:13.763555 2565 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 12 19:22:14.080508 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:13.763558 2565 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 12 19:22:14.080508 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:13.763562 2565 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 12 19:22:14.080508 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:13.763564 2565 flags.go:64] FLAG: --image-service-endpoint="" Jul 12 19:22:14.080508 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:13.763567 2565 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 12 19:22:14.080508 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:13.763569 2565 flags.go:64] FLAG: --kube-api-burst="100" Jul 12 19:22:14.080508 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:13.763572 2565 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 12 19:22:14.080508 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:13.763576 2565 flags.go:64] FLAG: --kube-api-qps="50" Jul 12 19:22:14.080508 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:13.763581 2565 flags.go:64] FLAG: --kube-reserved="" Jul 12 19:22:14.080508 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:13.763584 2565 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 12 19:22:14.080508 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:13.763587 2565 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 12 19:22:14.080508 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:13.763590 2565 flags.go:64] FLAG: --kubelet-cgroups="" Jul 12 19:22:14.080508 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:13.763592 2565 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 12 19:22:14.080508 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:13.763595 2565 flags.go:64] FLAG: --lock-file="" Jul 12 19:22:14.080508 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:13.763598 2565 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 12 19:22:14.080508 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:13.763601 2565 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 12 19:22:14.080508 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:13.763604 2565 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 12 19:22:14.081167 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:13.763609 2565 flags.go:64] FLAG: --log-json-split-stream="false" Jul 12 19:22:14.081167 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:13.763613 2565 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 12 19:22:14.081167 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:13.763615 2565 flags.go:64] FLAG: --log-text-split-stream="false" Jul 12 19:22:14.081167 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:13.763618 2565 flags.go:64] FLAG: --logging-format="text" Jul 12 19:22:14.081167 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:13.763621 2565 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 12 19:22:14.081167 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:13.763624 2565 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 12 19:22:14.081167 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:13.763627 2565 flags.go:64] FLAG: --manifest-url="" Jul 12 19:22:14.081167 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:13.763630 2565 flags.go:64] FLAG: --manifest-url-header="" Jul 12 19:22:14.081167 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:13.763635 2565 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 12 19:22:14.081167 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:13.763638 2565 flags.go:64] FLAG: --max-open-files="1000000" Jul 12 19:22:14.081167 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:13.763642 2565 flags.go:64] FLAG: --max-pods="110" Jul 12 19:22:14.081167 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:13.763645 2565 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 12 19:22:14.081167 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:13.763648 2565 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 12 19:22:14.081167 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:13.763651 2565 flags.go:64] FLAG: --memory-manager-policy="None" Jul 12 19:22:14.081167 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:13.763654 2565 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 12 19:22:14.081167 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:13.763657 2565 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 12 19:22:14.081167 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:13.763660 2565 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 12 19:22:14.081167 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:13.763663 2565 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 12 19:22:14.081167 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:13.763672 2565 flags.go:64] FLAG: --node-status-max-images="50" Jul 12 19:22:14.081167 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:13.763675 2565 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 12 19:22:14.081167 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:13.763678 2565 flags.go:64] FLAG: --oom-score-adj="-999" Jul 12 19:22:14.081167 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:13.763680 2565 flags.go:64] FLAG: --pod-cidr="" Jul 12 19:22:14.081167 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:13.763684 2565 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b04c0b1f73af4d021dd8f0b51bfeb302c24482c093de5f2c43654ab56c506926" Jul 12 19:22:14.082056 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:13.763691 2565 flags.go:64] FLAG: --pod-manifest-path="" Jul 12 19:22:14.082056 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:13.763694 2565 flags.go:64] FLAG: --pod-max-pids="-1" Jul 12 19:22:14.082056 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:13.763697 2565 flags.go:64] FLAG: --pods-per-core="0" Jul 12 19:22:14.082056 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:13.763700 2565 flags.go:64] FLAG: --port="10250" Jul 12 19:22:14.082056 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:13.763704 2565 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 12 19:22:14.082056 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:13.763707 2565 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-08a2fe02e2bc9034b" Jul 12 19:22:14.082056 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:13.763710 2565 flags.go:64] FLAG: --qos-reserved="" Jul 12 19:22:14.082056 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:13.763713 2565 flags.go:64] FLAG: --read-only-port="10255" Jul 12 19:22:14.082056 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:13.763716 2565 flags.go:64] FLAG: --register-node="true" Jul 12 19:22:14.082056 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:13.763719 2565 flags.go:64] FLAG: --register-with-taints="" Jul 12 19:22:14.082056 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:13.763723 2565 flags.go:64] FLAG: --registry-burst="10" Jul 12 19:22:14.082056 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:13.763725 2565 flags.go:64] FLAG: --registry-qps="5" Jul 12 19:22:14.082056 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:13.763729 2565 flags.go:64] FLAG: --reserved-cpus="" Jul 12 19:22:14.082056 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:13.763732 2565 flags.go:64] FLAG: --reserved-memory="" Jul 12 19:22:14.082056 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:13.763735 2565 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 12 19:22:14.082056 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:13.763739 2565 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 12 19:22:14.082056 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:13.763741 2565 flags.go:64] FLAG: --rotate-certificates="false" Jul 12 19:22:14.082056 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:13.763744 2565 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 12 19:22:14.082056 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:13.763747 2565 flags.go:64] FLAG: --runonce="false" Jul 12 19:22:14.082056 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:13.763750 2565 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 12 19:22:14.082056 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:13.763753 2565 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 12 19:22:14.082056 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:13.763756 2565 flags.go:64] FLAG: --seccomp-default="false" Jul 12 19:22:14.082056 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:13.763758 2565 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 12 19:22:14.082056 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:13.763761 2565 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 12 19:22:14.082056 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:13.763764 2565 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 12 19:22:14.082056 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:13.763767 2565 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 12 19:22:14.082782 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:13.763770 2565 flags.go:64] FLAG: --storage-driver-password="root" Jul 12 19:22:14.082782 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:13.763773 2565 flags.go:64] FLAG: --storage-driver-secure="false" Jul 12 19:22:14.082782 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:13.763777 2565 flags.go:64] FLAG: --storage-driver-table="stats" Jul 12 19:22:14.082782 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:13.763780 2565 flags.go:64] FLAG: --storage-driver-user="root" Jul 12 19:22:14.082782 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:13.763782 2565 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 12 19:22:14.082782 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:13.763785 2565 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 12 19:22:14.082782 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:13.763791 2565 flags.go:64] FLAG: --system-cgroups="" Jul 12 19:22:14.082782 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:13.763794 2565 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 12 19:22:14.082782 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:13.763801 2565 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 12 19:22:14.082782 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:13.763804 2565 flags.go:64] FLAG: --tls-cert-file="" Jul 12 19:22:14.082782 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:13.763807 2565 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 12 19:22:14.082782 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:13.763811 2565 flags.go:64] FLAG: --tls-min-version="" Jul 12 19:22:14.082782 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:13.763814 2565 flags.go:64] FLAG: --tls-private-key-file="" Jul 12 19:22:14.082782 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:13.763816 2565 flags.go:64] FLAG: --topology-manager-policy="none" Jul 12 19:22:14.082782 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:13.763819 2565 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 12 19:22:14.082782 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:13.763822 2565 flags.go:64] FLAG: --topology-manager-scope="container" Jul 12 19:22:14.082782 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:13.763825 2565 flags.go:64] FLAG: --v="2" Jul 12 19:22:14.082782 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:13.763829 2565 flags.go:64] FLAG: --version="false" Jul 12 19:22:14.082782 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:13.763833 2565 flags.go:64] FLAG: --vmodule="" Jul 12 19:22:14.082782 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:13.763837 2565 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 12 19:22:14.082782 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:13.763840 2565 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 12 19:22:14.082782 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.763969 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 12 19:22:14.082782 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.763974 2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 12 19:22:14.082782 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.763978 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 12 19:22:14.212813 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.763981 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 12 19:22:14.212813 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.763984 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 12 19:22:14.212813 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.763986 2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 12 19:22:14.212813 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.763989 2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 12 19:22:14.212813 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.763992 2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 12 19:22:14.212813 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.763995 2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 12 19:22:14.212813 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.763997 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 12 19:22:14.212813 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.764000 2565 feature_gate.go:328] unrecognized feature gate: Example2 Jul 12 19:22:14.212813 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.764002 2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 12 19:22:14.212813 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.764005 2565 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 12 19:22:14.212813 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.764008 2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 12 19:22:14.212813 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.764011 2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 12 19:22:14.212813 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.764014 2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 12 19:22:14.212813 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.764016 2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 12 19:22:14.212813 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.764021 2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 12 19:22:14.212813 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.764024 2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 12 19:22:14.212813 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.764027 2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 12 19:22:14.212813 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.764030 2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 12 19:22:14.212813 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.764032 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 12 19:22:14.212813 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.764035 2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 12 19:22:14.213534 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.764040 2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 12 19:22:14.213534 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.764043 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 12 19:22:14.213534 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.764046 2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 12 19:22:14.213534 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.764049 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 12 19:22:14.213534 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.764051 2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 12 19:22:14.213534 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.764054 2565 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 12 19:22:14.213534 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.764056 2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 12 19:22:14.213534 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.764059 2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 12 19:22:14.213534 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.764062 2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 12 19:22:14.213534 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.764064 2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 12 19:22:14.213534 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.764067 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 12 19:22:14.213534 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.764069 2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 12 19:22:14.213534 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.764072 2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 12 19:22:14.213534 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.764074 2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 12 19:22:14.213534 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.764077 2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 12 19:22:14.213534 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.764079 2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 12 19:22:14.213534 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.764082 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 12 19:22:14.213534 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.764085 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 12 19:22:14.213534 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.764087 2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 12 19:22:14.214071 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.764090 2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 12 19:22:14.214071 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.764092 2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 12 19:22:14.214071 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.764094 2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 12 19:22:14.214071 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.764097 2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 12 19:22:14.214071 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.764100 2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 12 19:22:14.214071 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.764103 2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 12 19:22:14.214071 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.764106 2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 12 19:22:14.214071 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.764113 2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 12 19:22:14.214071 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.764115 2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 12 19:22:14.214071 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.764118 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 12 19:22:14.214071 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.764121 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 12 19:22:14.214071 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.764123 2565 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 12 19:22:14.214071 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.764126 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 12 19:22:14.214071 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.764128 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 12 19:22:14.214071 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.764131 2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 12 19:22:14.214071 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.764133 2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 12 19:22:14.214071 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.764136 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 12 19:22:14.214071 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.764138 2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 12 19:22:14.214071 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.764141 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 12 19:22:14.214627 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.764144 2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 12 19:22:14.214627 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.764146 2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 12 19:22:14.214627 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.764148 2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 12 19:22:14.214627 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.764151 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 12 19:22:14.214627 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.764154 2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 12 19:22:14.214627 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.764156 2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 12 19:22:14.214627 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.764159 2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 12 19:22:14.214627 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.764161 2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 12 19:22:14.214627 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.764163 2565 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 12 19:22:14.214627 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.764166 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 12 19:22:14.214627 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.764168 2565 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 12 19:22:14.214627 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.764170 2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 12 19:22:14.214627 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.764173 2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 12 19:22:14.214627 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.764175 2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 12 19:22:14.214627 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.764178 2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 12 19:22:14.214627 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.764181 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 12 19:22:14.214627 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.764183 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 12 19:22:14.214627 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.764186 2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 12 19:22:14.214627 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.764188 2565 feature_gate.go:328] unrecognized feature gate: Example Jul 12 19:22:14.214627 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.764191 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 12 19:22:14.214627 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.764196 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 12 19:22:14.215167 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.764199 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 12 19:22:14.215167 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.764203 2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 12 19:22:14.215167 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.764206 2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 12 19:22:14.215167 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.764209 2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 12 19:22:14.215167 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.764211 2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 12 19:22:14.215167 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.764214 2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 12 19:22:14.215167 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.764232 2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 12 19:22:14.215167 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.764235 2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 12 19:22:14.215167 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.764237 2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 12 19:22:14.215167 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.764240 2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 12 19:22:14.215167 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.764243 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 12 19:22:14.215167 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.764245 2565 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 12 19:22:14.215167 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.764248 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 12 19:22:14.215167 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.764250 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 12 19:22:14.215167 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.764253 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 12 19:22:14.215167 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.764256 2565 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 12 19:22:14.215167 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.764258 2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 12 19:22:14.215648 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:13.764264 2565 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 12 19:22:14.215648 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:13.772868 2565 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 12 19:22:14.215648 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:13.772890 2565 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 12 19:22:14.215648 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:13.772910 2565 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 12 19:22:14.215648 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:13.772916 2565 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 12 19:22:14.215648 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.772992 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 12 19:22:14.215648 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.772999 2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 12 19:22:14.215648 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.773004 2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 12 19:22:14.215648 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.773007 2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 12 19:22:14.215648 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.773010 2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 12 19:22:14.215648 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.773013 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 12 19:22:14.215648 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.773016 2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 12 19:22:14.215648 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.773019 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 12 19:22:14.215648 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.773022 2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 12 19:22:14.215648 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.773024 2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 12 19:22:14.216050 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.773027 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 12 19:22:14.216050 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.773030 2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 12 19:22:14.216050 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.773035 2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 12 19:22:14.216050 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.773038 2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 12 19:22:14.216050 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.773040 2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 12 19:22:14.216050 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.773043 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 12 19:22:14.216050 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.773045 2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 12 19:22:14.216050 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.773048 2565 feature_gate.go:328] unrecognized feature gate: Example2 Jul 12 19:22:14.216050 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.773051 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 12 19:22:14.216050 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.773054 2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 12 19:22:14.216050 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.773056 2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 12 19:22:14.216050 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.773059 2565 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 12 19:22:14.216050 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.773062 2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 12 19:22:14.216050 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.773065 2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 12 19:22:14.216050 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.773070 2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 12 19:22:14.216050 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.773074 2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 12 19:22:14.216050 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.773076 2565 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 12 19:22:14.216050 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.773078 2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 12 19:22:14.216050 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.773081 2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 12 19:22:14.216050 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.773084 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 12 19:22:14.216604 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.773087 2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 12 19:22:14.216604 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.773091 2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 12 19:22:14.216604 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.773095 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 12 19:22:14.216604 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.773098 2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 12 19:22:14.216604 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.773102 2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 12 19:22:14.216604 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.773105 2565 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 12 19:22:14.216604 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.773108 2565 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 12 19:22:14.216604 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.773110 2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 12 19:22:14.216604 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.773113 2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 12 19:22:14.216604 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.773115 2565 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 12 19:22:14.216604 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.773118 2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 12 19:22:14.216604 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.773120 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 12 19:22:14.216604 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.773123 2565 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 12 19:22:14.216604 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.773126 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 12 19:22:14.216604 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.773128 2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 12 19:22:14.216604 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.773132 2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 12 19:22:14.216604 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.773134 2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 12 19:22:14.216604 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.773137 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 12 19:22:14.216604 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.773140 2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 12 19:22:14.216604 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.773142 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 12 19:22:14.217158 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.773145 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 12 19:22:14.217158 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.773147 2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 12 19:22:14.217158 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.773150 2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 12 19:22:14.217158 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.773152 2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 12 19:22:14.217158 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.773156 2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 12 19:22:14.217158 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.773158 2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 12 19:22:14.217158 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.773162 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 12 19:22:14.217158 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.773166 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 12 19:22:14.217158 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.773170 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 12 19:22:14.217158 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.773173 2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 12 19:22:14.217158 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.773176 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 12 19:22:14.217158 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.773178 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 12 19:22:14.217158 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.773181 2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 12 19:22:14.217158 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.773184 2565 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 12 19:22:14.217158 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.773186 2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 12 19:22:14.217158 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.773188 2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 12 19:22:14.217158 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.773193 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 12 19:22:14.217158 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.773195 2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 12 19:22:14.217158 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.773199 2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 12 19:22:14.217158 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.773203 2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 12 19:22:14.217707 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.773207 2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 12 19:22:14.217707 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.773210 2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 12 19:22:14.217707 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.773213 2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 12 19:22:14.217707 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.773230 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 12 19:22:14.217707 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.773234 2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 12 19:22:14.217707 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.773237 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 12 19:22:14.217707 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.773240 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 12 19:22:14.217707 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.773242 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 12 19:22:14.217707 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.773246 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 12 19:22:14.217707 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.773249 2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 12 19:22:14.217707 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.773251 2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 12 19:22:14.217707 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.773254 2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 12 19:22:14.217707 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.773257 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 12 19:22:14.217707 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.773259 2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 12 19:22:14.217707 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.773262 2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 12 19:22:14.217707 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.773265 2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 12 19:22:14.217707 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.773267 2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 12 19:22:14.217707 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.773270 2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 12 19:22:14.217707 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.773273 2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 12 19:22:14.218188 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.773276 2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 12 19:22:14.218188 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.773279 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 12 19:22:14.218188 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.773281 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 12 19:22:14.218188 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.773283 2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 12 19:22:14.218188 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.773286 2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 12 19:22:14.218188 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.773289 2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 12 19:22:14.218188 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.773291 2565 feature_gate.go:328] unrecognized feature gate: Example Jul 12 19:22:14.218188 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.773294 2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 12 19:22:14.218188 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.773296 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 12 19:22:14.218188 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.773299 2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 12 19:22:14.218188 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:13.773305 2565 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 12 19:22:14.218188 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.773443 2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 12 19:22:14.218188 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.773448 2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 12 19:22:14.218188 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.773451 2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 12 19:22:14.218601 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.773454 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 12 19:22:14.218601 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.773457 2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 12 19:22:14.218601 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.773459 2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 12 19:22:14.218601 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.773462 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 12 19:22:14.218601 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.773465 2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 12 19:22:14.218601 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.773469 2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 12 19:22:14.218601 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.773473 2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 12 19:22:14.218601 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.773476 2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 12 19:22:14.218601 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.773479 2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 12 19:22:14.218601 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.773482 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 12 19:22:14.218601 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.773485 2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 12 19:22:14.218601 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.773488 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 12 19:22:14.218601 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.773490 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 12 19:22:14.218601 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.773493 2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 12 19:22:14.218601 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.773496 2565 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 12 19:22:14.218601 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.773500 2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 12 19:22:14.218601 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.773502 2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 12 19:22:14.218601 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.773507 2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 12 19:22:14.218601 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.773510 2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 12 19:22:14.219097 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.773512 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 12 19:22:14.219097 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.773515 2565 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 12 19:22:14.219097 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.773517 2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 12 19:22:14.219097 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.773520 2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 12 19:22:14.219097 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.773522 2565 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 12 19:22:14.219097 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.773525 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 12 19:22:14.219097 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.773527 2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 12 19:22:14.219097 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.773530 2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 12 19:22:14.219097 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.773532 2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 12 19:22:14.219097 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.773536 2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 12 19:22:14.219097 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.773539 2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 12 19:22:14.219097 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.773541 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 12 19:22:14.219097 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.773544 2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 12 19:22:14.219097 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.773546 2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 12 19:22:14.219097 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.773548 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 12 19:22:14.219097 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.773551 2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 12 19:22:14.219097 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.773554 2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 12 19:22:14.219097 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.773556 2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 12 19:22:14.219097 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.773558 2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 12 19:22:14.219097 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.773562 2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 12 19:22:14.219650 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.773565 2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 12 19:22:14.219650 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.773567 2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 12 19:22:14.219650 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.773578 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 12 19:22:14.219650 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.773580 2565 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 12 19:22:14.219650 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.773583 2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 12 19:22:14.219650 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.773586 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 12 19:22:14.219650 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.773590 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 12 19:22:14.219650 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.773593 2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 12 19:22:14.219650 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.773597 2565 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 12 19:22:14.219650 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.773600 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 12 19:22:14.219650 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.773603 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 12 19:22:14.219650 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.773606 2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 12 19:22:14.219650 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.773609 2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 12 19:22:14.219650 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.773611 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 12 19:22:14.219650 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.773614 2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 12 19:22:14.219650 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.773616 2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 12 19:22:14.219650 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.773619 2565 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 12 19:22:14.219650 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.773622 2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 12 19:22:14.219650 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.773624 2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 12 19:22:14.219650 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.773627 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 12 19:22:14.220187 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.773629 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 12 19:22:14.220187 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.773633 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 12 19:22:14.220187 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.773636 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 12 19:22:14.220187 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.773638 2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 12 19:22:14.220187 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.773640 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 12 19:22:14.220187 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.773643 2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 12 19:22:14.220187 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.773646 2565 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 12 19:22:14.220187 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.773648 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 12 19:22:14.220187 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.773651 2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 12 19:22:14.220187 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.773654 2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 12 19:22:14.220187 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.773657 2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 12 19:22:14.220187 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.773660 2565 feature_gate.go:328] unrecognized feature gate: Example Jul 12 19:22:14.220187 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.773662 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 12 19:22:14.220187 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.773665 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 12 19:22:14.220187 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.773667 2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 12 19:22:14.220187 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.773676 2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 12 19:22:14.220187 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.773678 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 12 19:22:14.220187 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.773681 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 12 19:22:14.220187 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.773683 2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 12 19:22:14.220793 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.773686 2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 12 19:22:14.220793 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.773688 2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 12 19:22:14.220793 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.773691 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 12 19:22:14.220793 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.773695 2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 12 19:22:14.220793 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.773697 2565 feature_gate.go:328] unrecognized feature gate: Example2 Jul 12 19:22:14.220793 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.773700 2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 12 19:22:14.220793 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.773702 2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 12 19:22:14.220793 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.773705 2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 12 19:22:14.220793 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.773707 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 12 19:22:14.220793 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.773710 2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 12 19:22:14.220793 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.773712 2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 12 19:22:14.220793 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.773716 2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 12 19:22:14.220793 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.773720 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 12 19:22:14.220793 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.773723 2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 12 19:22:14.220793 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.773728 2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 12 19:22:14.220793 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.773732 2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 12 19:22:14.220793 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.773735 2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 12 19:22:14.220793 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:13.773740 2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 12 19:22:14.221275 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:13.773746 2565 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 12 19:22:14.221275 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:13.773921 2565 server.go:962] "Client rotation is on, will bootstrap in background" Jul 12 19:22:14.221275 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:13.777948 2565 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 12 19:22:14.221275 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:13.779583 2565 server.go:1019] "Starting client certificate rotation" Jul 12 19:22:14.221275 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:13.779695 2565 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 12 19:22:14.221275 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:13.779742 2565 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 12 19:22:14.221275 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:13.783325 2565 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 12 19:22:14.221275 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:13.785356 2565 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 12 19:22:14.221275 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:13.797481 2565 log.go:25] "Validated CRI v1 runtime API" Jul 12 19:22:14.221275 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:13.799887 2565 log.go:25] "Validated CRI v1 image API" Jul 12 19:22:14.221275 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:13.800689 2565 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 12 19:22:14.221275 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:13.806009 2565 fs.go:135] Filesystem UUIDs: map[04894c67-b242-45f4-b7d1-2d33af73b85c:/dev/nvme0n1p4 482f0591-5ba4-4f3d-8665-b4e1ad990c7a:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2] Jul 12 19:22:14.221635 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:13.806028 2565 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 12 19:22:14.221700 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:13.810345 2565 manager.go:216] Machine: {Timestamp:2026-07-12 19:22:13.809845721 +0000 UTC m=+0.376081337 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3196602 MemoryCapacity:33164492800 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec28630b72a8f9141c137fc4abac8f22 SystemUUID:ec28630b-72a8-f914-1c13-7fc4abac8f22 BootID:86ff458c-43e9-4829-80c2-7f43f0c1908b Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:c9:cf:67:e9:9b Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:c9:cf:67:e9:9b Speed:0 Mtu:9001} {Name:ovs-system MacAddress:a2:fa:3f:4e:4e:fc Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164492800 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 12 19:22:14.221700 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:13.810459 2565 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 12 19:22:14.221700 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:13.810549 2565 manager.go:232] Version: {KernelVersion:5.14.0-570.126.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260702-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 12 19:22:14.221700 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:13.811130 2565 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 12 19:22:14.221700 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:13.814504 2565 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 12 19:22:14.221700 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:13.814547 2565 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-139-170.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 12 19:22:14.221700 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:13.815450 2565 topology_manager.go:138] "Creating topology manager with none policy" Jul 12 19:22:14.221979 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:13.815462 2565 container_manager_linux.go:309] "Creating device plugin manager" Jul 12 19:22:14.221979 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:13.815472 2565 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 12 19:22:14.221979 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:13.815483 2565 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 12 19:22:14.221979 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:13.816611 2565 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 12 19:22:14.221979 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:13.820814 2565 state_mem.go:36] "Initialized new in-memory state store" Jul 12 19:22:14.221979 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:13.820940 2565 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 12 19:22:14.221979 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:13.820974 2565 kubelet.go:486] "Attempting to sync node with API server" Jul 12 19:22:14.221979 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:13.820984 2565 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 12 19:22:14.221979 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:13.820997 2565 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 12 19:22:14.221979 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:13.821006 2565 kubelet.go:398] "Adding apiserver pod source" Jul 12 19:22:14.221979 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:13.821015 2565 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 12 19:22:14.221979 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:13.821515 2565 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 12 19:22:14.221979 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:13.821531 2565 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 12 19:22:14.221979 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:13.825010 2565 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 12 19:22:14.221979 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:13.826881 2565 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 12 19:22:14.221979 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:13.826917 2565 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 12 19:22:14.221979 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:13.827791 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 12 19:22:14.221979 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:13.827805 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 12 19:22:14.221979 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:13.827812 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 12 19:22:14.221979 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:13.827817 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 12 19:22:14.221979 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:13.827827 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 12 19:22:14.221979 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:13.827842 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 12 19:22:14.221979 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:13.827851 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 12 19:22:14.221979 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:13.827856 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 12 19:22:14.221979 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:13.827862 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 12 19:22:14.221979 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:13.827867 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 12 19:22:14.221979 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:13.827882 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 12 19:22:14.221979 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:13.827889 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 12 19:22:14.221979 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:13.828786 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 12 19:22:14.221979 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:13.828792 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 12 19:22:14.221979 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:13.832500 2565 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-139-170.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 12 19:22:14.221979 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:13.832663 2565 server.go:1268] "Started kubelet" Jul 12 19:22:14.221979 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:13.832739 2565 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 12 19:22:14.221979 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:13.832785 2565 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 12 19:22:14.221979 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:13.832868 2565 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 12 19:22:14.221979 ip-10-0-139-170 kubenswrapper[2565]: E0712 19:22:13.833024 2565 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 12 19:22:14.221979 ip-10-0-139-170 kubenswrapper[2565]: E0712 19:22:13.833043 2565 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-139-170.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 12 19:22:14.221979 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:13.837580 2565 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 12 19:22:14.221979 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:13.838978 2565 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-rps6h" Jul 12 19:22:14.225544 ip-10-0-139-170 kubenswrapper[2565]: E0712 19:22:13.840065 2565 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 12 19:22:14.225544 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:13.840435 2565 server.go:310] "Adding debug handlers to kubelet server" Jul 12 19:22:14.225544 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:13.843266 2565 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 12 19:22:14.225544 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:13.843274 2565 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 12 19:22:14.225544 ip-10-0-139-170 kubenswrapper[2565]: E0712 19:22:13.843372 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-170.ec2.internal\" not found" Jul 12 19:22:14.225544 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:13.844411 2565 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-rps6h" Jul 12 19:22:14.225544 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:13.844532 2565 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 12 19:22:14.225544 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:13.844545 2565 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 12 19:22:14.225544 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:13.844625 2565 factory.go:55] Registering systemd factory Jul 12 19:22:14.225544 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:13.844641 2565 factory.go:223] Registration of the systemd container factory successfully Jul 12 19:22:14.225544 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:13.844682 2565 reconstruct.go:97] "Volume reconstruction finished" Jul 12 19:22:14.225544 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:13.844691 2565 reconciler.go:29] "Reconciler: start to sync state" Jul 12 19:22:14.225544 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:13.844743 2565 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 12 19:22:14.225544 ip-10-0-139-170 kubenswrapper[2565]: E0712 19:22:13.843327 2565 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-170.ec2.internal.18c1a12b15887672 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-170.ec2.internal,UID:ip-10-0-139-170.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-139-170.ec2.internal,},FirstTimestamp:2026-07-12 19:22:13.83235749 +0000 UTC m=+0.398593176,LastTimestamp:2026-07-12 19:22:13.83235749 +0000 UTC m=+0.398593176,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-170.ec2.internal,}" Jul 12 19:22:14.225544 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:13.845538 2565 factory.go:153] Registering CRI-O factory Jul 12 19:22:14.225544 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:13.845555 2565 factory.go:223] Registration of the crio container factory successfully Jul 12 19:22:14.225544 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:13.845611 2565 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 12 19:22:14.225544 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:13.845638 2565 factory.go:103] Registering Raw factory Jul 12 19:22:14.225544 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:13.845656 2565 manager.go:1196] Started watching for new ooms in manager Jul 12 19:22:14.225544 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:13.846632 2565 manager.go:349] Starting recovery of all containers Jul 12 19:22:14.225544 ip-10-0-139-170 kubenswrapper[2565]: E0712 19:22:13.851861 2565 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-139-170.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 12 19:22:14.225544 ip-10-0-139-170 kubenswrapper[2565]: E0712 19:22:13.852731 2565 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 12 19:22:14.225544 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:13.858506 2565 manager.go:354] Recovery completed Jul 12 19:22:14.225544 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:13.864058 2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 12 19:22:14.225544 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:13.867063 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-170.ec2.internal" event="NodeHasSufficientMemory" Jul 12 19:22:14.225544 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:13.867099 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-170.ec2.internal" event="NodeHasNoDiskPressure" Jul 12 19:22:14.225544 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:13.867114 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-170.ec2.internal" event="NodeHasSufficientPID" Jul 12 19:22:14.225544 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:13.867791 2565 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 12 19:22:14.225544 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:13.867801 2565 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 12 19:22:14.225544 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:13.867818 2565 state_mem.go:36] "Initialized new in-memory state store" Jul 12 19:22:14.225544 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:13.867803 2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 12 19:22:14.225544 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:13.868484 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-170.ec2.internal" event="NodeHasSufficientMemory" Jul 12 19:22:14.225544 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:13.868510 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-170.ec2.internal" event="NodeHasNoDiskPressure" Jul 12 19:22:14.376330 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:13.868523 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-170.ec2.internal" event="NodeHasSufficientPID" Jul 12 19:22:14.376330 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:13.869899 2565 policy_none.go:49] "None policy: Start" Jul 12 19:22:14.376330 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:13.869912 2565 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 12 19:22:14.376330 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:13.869922 2565 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 12 19:22:14.376330 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:13.871870 2565 policy_none.go:47] "Start" Jul 12 19:22:14.376330 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:13.939923 2565 manager.go:337] "Starting Device Plugin manager" Jul 12 19:22:14.376330 ip-10-0-139-170 kubenswrapper[2565]: E0712 19:22:13.939968 2565 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 12 19:22:14.376330 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:13.939983 2565 server.go:87] "Starting device plugin registration server" Jul 12 19:22:14.376330 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:13.940259 2565 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 12 19:22:14.376330 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:13.940270 2565 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 12 19:22:14.376330 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:13.940393 2565 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 12 19:22:14.376330 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:13.940476 2565 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 12 19:22:14.376330 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:13.940484 2565 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 12 19:22:14.376330 ip-10-0-139-170 kubenswrapper[2565]: E0712 19:22:13.940876 2565 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 12 19:22:14.376330 ip-10-0-139-170 kubenswrapper[2565]: E0712 19:22:13.940930 2565 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-139-170.ec2.internal\" not found" Jul 12 19:22:14.376330 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:14.016282 2565 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 12 19:22:14.376330 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:14.017514 2565 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 12 19:22:14.376330 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:14.017568 2565 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 12 19:22:14.376330 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:14.017601 2565 kubelet.go:2448] "Starting kubelet main sync loop" Jul 12 19:22:14.376330 ip-10-0-139-170 kubenswrapper[2565]: E0712 19:22:14.017641 2565 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 12 19:22:14.376330 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:14.019764 2565 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 12 19:22:14.376330 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:14.040574 2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 12 19:22:14.376330 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:14.041748 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-170.ec2.internal" event="NodeHasSufficientMemory" Jul 12 19:22:14.376330 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:14.041781 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-170.ec2.internal" event="NodeHasNoDiskPressure" Jul 12 19:22:14.376330 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:14.041795 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-170.ec2.internal" event="NodeHasSufficientPID" Jul 12 19:22:14.376330 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:14.041824 2565 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-139-170.ec2.internal" Jul 12 19:22:14.376330 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:14.049306 2565 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-139-170.ec2.internal" Jul 12 19:22:14.376330 ip-10-0-139-170 kubenswrapper[2565]: E0712 19:22:14.049342 2565 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-139-170.ec2.internal\": node \"ip-10-0-139-170.ec2.internal\" not found" Jul 12 19:22:14.376330 ip-10-0-139-170 kubenswrapper[2565]: E0712 19:22:14.067735 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-170.ec2.internal\" not found" Jul 12 19:22:14.376330 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:14.118311 2565 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-170.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-139-170.ec2.internal"] Jul 12 19:22:14.376330 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:14.119116 2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 12 19:22:14.376330 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:14.124886 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-170.ec2.internal" event="NodeHasSufficientMemory" Jul 12 19:22:14.376330 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:14.124921 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-170.ec2.internal" event="NodeHasNoDiskPressure" Jul 12 19:22:14.376330 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:14.124934 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-170.ec2.internal" event="NodeHasSufficientPID" Jul 12 19:22:14.376330 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:14.127194 2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 12 19:22:14.376330 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:14.127309 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-170.ec2.internal" Jul 12 19:22:14.376330 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:14.127340 2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 12 19:22:14.376330 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:14.127917 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-170.ec2.internal" event="NodeHasSufficientMemory" Jul 12 19:22:14.379952 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:14.127937 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-170.ec2.internal" event="NodeHasSufficientMemory" Jul 12 19:22:14.379952 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:14.127959 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-170.ec2.internal" event="NodeHasNoDiskPressure" Jul 12 19:22:14.379952 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:14.127970 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-170.ec2.internal" event="NodeHasSufficientPID" Jul 12 19:22:14.379952 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:14.127942 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-170.ec2.internal" event="NodeHasNoDiskPressure" Jul 12 19:22:14.379952 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:14.128032 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-170.ec2.internal" event="NodeHasSufficientPID" Jul 12 19:22:14.379952 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:14.130174 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-170.ec2.internal" Jul 12 19:22:14.379952 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:14.130196 2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 12 19:22:14.379952 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:14.130907 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-170.ec2.internal" event="NodeHasSufficientMemory" Jul 12 19:22:14.379952 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:14.130931 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-170.ec2.internal" event="NodeHasNoDiskPressure" Jul 12 19:22:14.379952 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:14.130942 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-170.ec2.internal" event="NodeHasSufficientPID" Jul 12 19:22:14.379952 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:14.146592 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/83a48f1c2ae61195e40b72789388b7f8-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-170.ec2.internal\" (UID: \"83a48f1c2ae61195e40b72789388b7f8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-170.ec2.internal" Jul 12 19:22:14.379952 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:14.146618 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/83a48f1c2ae61195e40b72789388b7f8-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-170.ec2.internal\" (UID: \"83a48f1c2ae61195e40b72789388b7f8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-170.ec2.internal" Jul 12 19:22:14.379952 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:14.146640 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2a4584a3f0966be0c3f50f97c2a5e31a-config\") pod \"kube-apiserver-proxy-ip-10-0-139-170.ec2.internal\" (UID: \"2a4584a3f0966be0c3f50f97c2a5e31a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-170.ec2.internal" Jul 12 19:22:14.379952 ip-10-0-139-170 kubenswrapper[2565]: E0712 19:22:14.151950 2565 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-170.ec2.internal\" not found" node="ip-10-0-139-170.ec2.internal" Jul 12 19:22:14.379952 ip-10-0-139-170 kubenswrapper[2565]: E0712 19:22:14.158372 2565 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-170.ec2.internal\" not found" node="ip-10-0-139-170.ec2.internal" Jul 12 19:22:14.379952 ip-10-0-139-170 kubenswrapper[2565]: E0712 19:22:14.167786 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-170.ec2.internal\" not found" Jul 12 19:22:14.379952 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:14.246864 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/83a48f1c2ae61195e40b72789388b7f8-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-170.ec2.internal\" (UID: \"83a48f1c2ae61195e40b72789388b7f8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-170.ec2.internal" Jul 12 19:22:14.379952 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:14.246918 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/83a48f1c2ae61195e40b72789388b7f8-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-170.ec2.internal\" (UID: \"83a48f1c2ae61195e40b72789388b7f8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-170.ec2.internal" Jul 12 19:22:14.379952 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:14.246936 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2a4584a3f0966be0c3f50f97c2a5e31a-config\") pod \"kube-apiserver-proxy-ip-10-0-139-170.ec2.internal\" (UID: \"2a4584a3f0966be0c3f50f97c2a5e31a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-170.ec2.internal" Jul 12 19:22:14.379952 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:14.246974 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2a4584a3f0966be0c3f50f97c2a5e31a-config\") pod \"kube-apiserver-proxy-ip-10-0-139-170.ec2.internal\" (UID: \"2a4584a3f0966be0c3f50f97c2a5e31a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-170.ec2.internal" Jul 12 19:22:14.380544 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:14.247575 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/83a48f1c2ae61195e40b72789388b7f8-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-170.ec2.internal\" (UID: \"83a48f1c2ae61195e40b72789388b7f8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-170.ec2.internal" Jul 12 19:22:14.380544 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:14.247790 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/83a48f1c2ae61195e40b72789388b7f8-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-170.ec2.internal\" (UID: \"83a48f1c2ae61195e40b72789388b7f8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-170.ec2.internal" Jul 12 19:22:14.380544 ip-10-0-139-170 kubenswrapper[2565]: E0712 19:22:14.268208 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-170.ec2.internal\" not found" Jul 12 19:22:14.380544 ip-10-0-139-170 kubenswrapper[2565]: E0712 19:22:14.368307 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-170.ec2.internal\" not found" Jul 12 19:22:14.555948 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:14.454865 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-170.ec2.internal" Jul 12 19:22:14.555948 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:14.461131 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-170.ec2.internal" Jul 12 19:22:14.555948 ip-10-0-139-170 kubenswrapper[2565]: E0712 19:22:14.468489 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-170.ec2.internal\" not found" Jul 12 19:22:16.581069 ip-10-0-139-170 kubenswrapper[2565]: E0712 19:22:14.569114 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-170.ec2.internal\" not found" Jul 12 19:22:16.581069 ip-10-0-139-170 kubenswrapper[2565]: E0712 19:22:16.503318 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-170.ec2.internal\" not found" Jul 12 19:22:16.758597 ip-10-0-139-170 kubenswrapper[2565]: E0712 19:22:16.603994 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-170.ec2.internal\" not found" Jul 12 19:22:16.758597 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:16.612148 2565 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 12 19:22:16.758597 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:16.612308 2565 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 12 19:22:16.758597 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:16.634793 2565 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 12 19:22:16.758597 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:16.685656 2565 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-11 19:17:15 +0000 UTC" deadline="2027-12-25 05:30:14.851906519 +0000 UTC" Jul 12 19:22:16.758597 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:16.685682 2565 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12730h7m58.166227714s" Jul 12 19:22:16.758597 ip-10-0-139-170 kubenswrapper[2565]: E0712 19:22:16.704810 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-170.ec2.internal\" not found" Jul 12 19:22:16.758597 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:16.731094 2565 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 12 19:22:16.900341 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:16.778051 2565 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-170.ec2.internal" Jul 12 19:22:16.900341 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:16.787657 2565 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 12 19:22:16.900341 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:16.789273 2565 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-170.ec2.internal" Jul 12 19:22:16.900341 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:16.797691 2565 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 12 19:22:16.900341 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:16.807273 2565 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 12 19:22:16.900341 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:16.859508 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83a48f1c2ae61195e40b72789388b7f8.slice/crio-47ba7d14b0c53865b1ee26bd94d25a5c5589ab7b75a2ed97ea256e7fbf78bd5e WatchSource:0}: Error finding container 47ba7d14b0c53865b1ee26bd94d25a5c5589ab7b75a2ed97ea256e7fbf78bd5e: Status 404 returned error can't find the container with id 47ba7d14b0c53865b1ee26bd94d25a5c5589ab7b75a2ed97ea256e7fbf78bd5e Jul 12 19:22:16.900341 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:16.865502 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a4584a3f0966be0c3f50f97c2a5e31a.slice/crio-5268b43e5f11755e1f696a7ca89028441497336049f4d47bd07a61d5e9beb792 WatchSource:0}: Error finding container 5268b43e5f11755e1f696a7ca89028441497336049f4d47bd07a61d5e9beb792: Status 404 returned error can't find the container with id 5268b43e5f11755e1f696a7ca89028441497336049f4d47bd07a61d5e9beb792 Jul 12 19:22:16.900341 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:16.868223 2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 12 19:22:17.475826 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.424285 2565 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 12 19:22:17.675852 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.655617 2565 apiserver.go:52] "Watching apiserver" Jul 12 19:22:17.675852 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.662374 2565 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 12 19:22:17.675852 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.662910 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-288bq","kube-system/konnectivity-agent-7vvbp","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bwmdr","openshift-cluster-node-tuning-operator/tuned-vsllh","openshift-image-registry/node-ca-c89jb","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-170.ec2.internal","openshift-multus/multus-additional-cni-plugins-prfcd","openshift-multus/network-metrics-daemon-2xmfq","kube-system/kube-apiserver-proxy-ip-10-0-139-170.ec2.internal","openshift-dns/node-resolver-zs4lw","openshift-multus/multus-9klzd","openshift-network-diagnostics/network-check-target-76gfm","openshift-network-operator/iptables-alerter-hrrhk","openshift-ovn-kubernetes/ovnkube-node-k5h9m"] Jul 12 19:22:17.675852 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.665542 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-prfcd" Jul 12 19:22:17.675852 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.667643 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 12 19:22:17.675852 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.667924 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-hrrhk" Jul 12 19:22:17.675852 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.668112 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 12 19:22:17.675852 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.668134 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-b4d9h\"" Jul 12 19:22:17.675852 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.668280 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 12 19:22:17.675852 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.668378 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 12 19:22:17.675852 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.668462 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 12 19:22:17.675852 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.669911 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 12 19:22:17.675852 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.670181 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 12 19:22:17.675852 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.670213 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 12 19:22:17.675852 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.670404 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-p98w2\"" Jul 12 19:22:17.675852 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.670443 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bwmdr" Jul 12 19:22:17.675852 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.671973 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-vsllh" Jul 12 19:22:17.675852 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.672402 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 12 19:22:17.675852 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.672414 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-cwfpd\"" Jul 12 19:22:17.675852 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.672431 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 12 19:22:17.675852 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.672569 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 12 19:22:17.675852 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.673902 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 12 19:22:17.675852 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.673959 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 12 19:22:17.675852 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.674199 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-c89jb" Jul 12 19:22:17.675852 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.674299 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-g58wf\"" Jul 12 19:22:17.677263 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.676013 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-tgsv7\"" Jul 12 19:22:17.677263 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.676398 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 12 19:22:17.677263 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.676409 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 12 19:22:17.677263 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.676643 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 12 19:22:17.677263 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.676723 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-k5h9m" Jul 12 19:22:17.677842 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.677625 2565 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 12 19:22:17.679484 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.679322 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 12 19:22:17.679484 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.679336 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 12 19:22:17.679484 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.679349 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2xmfq" Jul 12 19:22:17.679484 ip-10-0-139-170 kubenswrapper[2565]: E0712 19:22:17.679409 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2xmfq" podUID="e7722ae2-e293-478e-9ac8-ea376f8d583f" Jul 12 19:22:17.796576 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.684013 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-76gfm" Jul 12 19:22:17.796576 ip-10-0-139-170 kubenswrapper[2565]: E0712 19:22:17.684081 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-76gfm" podUID="10c67d44-bdd0-4530-9f5d-fa10238ee66e" Jul 12 19:22:17.796576 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.688639 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 12 19:22:17.796576 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.688834 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/8e016c2e-0c3a-49da-83d4-f526738c1cbd-log-socket\") pod \"ovnkube-node-k5h9m\" (UID: \"8e016c2e-0c3a-49da-83d4-f526738c1cbd\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5h9m" Jul 12 19:22:17.796576 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.688871 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/8e016c2e-0c3a-49da-83d4-f526738c1cbd-ovnkube-config\") pod \"ovnkube-node-k5h9m\" (UID: \"8e016c2e-0c3a-49da-83d4-f526738c1cbd\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5h9m" Jul 12 19:22:17.796576 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.688940 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 12 19:22:17.796576 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.689512 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-k296g\"" Jul 12 19:22:17.796576 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.688916 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/087e3f6e-992f-4784-a376-bd2af99f9851-host-slash\") pod \"iptables-alerter-hrrhk\" (UID: \"087e3f6e-992f-4784-a376-bd2af99f9851\") " pod="openshift-network-operator/iptables-alerter-hrrhk" Jul 12 19:22:17.796576 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.690573 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/8e016c2e-0c3a-49da-83d4-f526738c1cbd-node-log\") pod \"ovnkube-node-k5h9m\" (UID: \"8e016c2e-0c3a-49da-83d4-f526738c1cbd\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5h9m" Jul 12 19:22:17.796576 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.690606 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-csp9b\" (UniqueName: \"kubernetes.io/projected/8e016c2e-0c3a-49da-83d4-f526738c1cbd-kube-api-access-csp9b\") pod \"ovnkube-node-k5h9m\" (UID: \"8e016c2e-0c3a-49da-83d4-f526738c1cbd\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5h9m" Jul 12 19:22:17.796576 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.690635 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/51dee771-9663-4366-90ca-9b1c286e35f8-tuning-conf-dir\") pod \"multus-additional-cni-plugins-prfcd\" (UID: \"51dee771-9663-4366-90ca-9b1c286e35f8\") " pod="openshift-multus/multus-additional-cni-plugins-prfcd" Jul 12 19:22:17.796576 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.690661 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/23c32389-3933-4c29-a625-3e46ade08326-var-lib-kubelet\") pod \"tuned-vsllh\" (UID: \"23c32389-3933-4c29-a625-3e46ade08326\") " pod="openshift-cluster-node-tuning-operator/tuned-vsllh" Jul 12 19:22:17.796576 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.690685 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5h9cz\" (UniqueName: \"kubernetes.io/projected/23c32389-3933-4c29-a625-3e46ade08326-kube-api-access-5h9cz\") pod \"tuned-vsllh\" (UID: \"23c32389-3933-4c29-a625-3e46ade08326\") " pod="openshift-cluster-node-tuning-operator/tuned-vsllh" Jul 12 19:22:17.796576 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.690746 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/51dee771-9663-4366-90ca-9b1c286e35f8-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-prfcd\" (UID: \"51dee771-9663-4366-90ca-9b1c286e35f8\") " pod="openshift-multus/multus-additional-cni-plugins-prfcd" Jul 12 19:22:17.796576 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.690789 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/56c9f12c-6e22-414d-a8d7-b503ed1f38df-socket-dir\") pod \"aws-ebs-csi-driver-node-bwmdr\" (UID: \"56c9f12c-6e22-414d-a8d7-b503ed1f38df\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bwmdr" Jul 12 19:22:17.796576 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.690921 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 12 19:22:17.796576 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.690980 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 12 19:22:17.798344 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.690996 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/23c32389-3933-4c29-a625-3e46ade08326-host\") pod \"tuned-vsllh\" (UID: \"23c32389-3933-4c29-a625-3e46ade08326\") " pod="openshift-cluster-node-tuning-operator/tuned-vsllh" Jul 12 19:22:17.798344 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.691049 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/23c32389-3933-4c29-a625-3e46ade08326-sys\") pod \"tuned-vsllh\" (UID: \"23c32389-3933-4c29-a625-3e46ade08326\") " pod="openshift-cluster-node-tuning-operator/tuned-vsllh" Jul 12 19:22:17.798344 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.691081 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/23c32389-3933-4c29-a625-3e46ade08326-etc-tuned\") pod \"tuned-vsllh\" (UID: \"23c32389-3933-4c29-a625-3e46ade08326\") " pod="openshift-cluster-node-tuning-operator/tuned-vsllh" Jul 12 19:22:17.798344 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.691170 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/087e3f6e-992f-4784-a376-bd2af99f9851-iptables-alerter-script\") pod \"iptables-alerter-hrrhk\" (UID: \"087e3f6e-992f-4784-a376-bd2af99f9851\") " pod="openshift-network-operator/iptables-alerter-hrrhk" Jul 12 19:22:17.798344 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.691261 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/8e016c2e-0c3a-49da-83d4-f526738c1cbd-env-overrides\") pod \"ovnkube-node-k5h9m\" (UID: \"8e016c2e-0c3a-49da-83d4-f526738c1cbd\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5h9m" Jul 12 19:22:17.798344 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.691315 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/56c9f12c-6e22-414d-a8d7-b503ed1f38df-etc-selinux\") pod \"aws-ebs-csi-driver-node-bwmdr\" (UID: \"56c9f12c-6e22-414d-a8d7-b503ed1f38df\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bwmdr" Jul 12 19:22:17.798344 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.691399 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8e016c2e-0c3a-49da-83d4-f526738c1cbd-host-run-netns\") pod \"ovnkube-node-k5h9m\" (UID: \"8e016c2e-0c3a-49da-83d4-f526738c1cbd\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5h9m" Jul 12 19:22:17.798344 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.691429 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e7722ae2-e293-478e-9ac8-ea376f8d583f-metrics-certs\") pod \"network-metrics-daemon-2xmfq\" (UID: \"e7722ae2-e293-478e-9ac8-ea376f8d583f\") " pod="openshift-multus/network-metrics-daemon-2xmfq" Jul 12 19:22:17.798344 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.691454 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b8cca381-48d7-43ac-ae6a-15f96d99345d-host\") pod \"node-ca-c89jb\" (UID: \"b8cca381-48d7-43ac-ae6a-15f96d99345d\") " pod="openshift-image-registry/node-ca-c89jb" Jul 12 19:22:17.798344 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.691564 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/23c32389-3933-4c29-a625-3e46ade08326-run\") pod \"tuned-vsllh\" (UID: \"23c32389-3933-4c29-a625-3e46ade08326\") " pod="openshift-cluster-node-tuning-operator/tuned-vsllh" Jul 12 19:22:17.798344 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.691619 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8e016c2e-0c3a-49da-83d4-f526738c1cbd-host-slash\") pod \"ovnkube-node-k5h9m\" (UID: \"8e016c2e-0c3a-49da-83d4-f526738c1cbd\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5h9m" Jul 12 19:22:17.798344 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.691709 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8e016c2e-0c3a-49da-83d4-f526738c1cbd-host-cni-bin\") pod \"ovnkube-node-k5h9m\" (UID: \"8e016c2e-0c3a-49da-83d4-f526738c1cbd\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5h9m" Jul 12 19:22:17.798344 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.691775 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/51dee771-9663-4366-90ca-9b1c286e35f8-cni-binary-copy\") pod \"multus-additional-cni-plugins-prfcd\" (UID: \"51dee771-9663-4366-90ca-9b1c286e35f8\") " pod="openshift-multus/multus-additional-cni-plugins-prfcd" Jul 12 19:22:17.798344 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.691810 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/56c9f12c-6e22-414d-a8d7-b503ed1f38df-kubelet-dir\") pod \"aws-ebs-csi-driver-node-bwmdr\" (UID: \"56c9f12c-6e22-414d-a8d7-b503ed1f38df\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bwmdr" Jul 12 19:22:17.799376 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.691846 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/56c9f12c-6e22-414d-a8d7-b503ed1f38df-registration-dir\") pod \"aws-ebs-csi-driver-node-bwmdr\" (UID: \"56c9f12c-6e22-414d-a8d7-b503ed1f38df\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bwmdr" Jul 12 19:22:17.799376 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.691876 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/23c32389-3933-4c29-a625-3e46ade08326-lib-modules\") pod \"tuned-vsllh\" (UID: \"23c32389-3933-4c29-a625-3e46ade08326\") " pod="openshift-cluster-node-tuning-operator/tuned-vsllh" Jul 12 19:22:17.799376 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.692154 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-7vvbp" Jul 12 19:22:17.799376 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.691984 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/23c32389-3933-4c29-a625-3e46ade08326-tmp\") pod \"tuned-vsllh\" (UID: \"23c32389-3933-4c29-a625-3e46ade08326\") " pod="openshift-cluster-node-tuning-operator/tuned-vsllh" Jul 12 19:22:17.799376 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.692580 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxwbs\" (UniqueName: \"kubernetes.io/projected/51dee771-9663-4366-90ca-9b1c286e35f8-kube-api-access-rxwbs\") pod \"multus-additional-cni-plugins-prfcd\" (UID: \"51dee771-9663-4366-90ca-9b1c286e35f8\") " pod="openshift-multus/multus-additional-cni-plugins-prfcd" Jul 12 19:22:17.799376 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.692667 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-czqm5\" (UniqueName: \"kubernetes.io/projected/b8cca381-48d7-43ac-ae6a-15f96d99345d-kube-api-access-czqm5\") pod \"node-ca-c89jb\" (UID: \"b8cca381-48d7-43ac-ae6a-15f96d99345d\") " pod="openshift-image-registry/node-ca-c89jb" Jul 12 19:22:17.799376 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.692703 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/8e016c2e-0c3a-49da-83d4-f526738c1cbd-run-ovn\") pod \"ovnkube-node-k5h9m\" (UID: \"8e016c2e-0c3a-49da-83d4-f526738c1cbd\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5h9m" Jul 12 19:22:17.799376 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.692741 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nj2g8\" (UniqueName: \"kubernetes.io/projected/e7722ae2-e293-478e-9ac8-ea376f8d583f-kube-api-access-nj2g8\") pod \"network-metrics-daemon-2xmfq\" (UID: \"e7722ae2-e293-478e-9ac8-ea376f8d583f\") " pod="openshift-multus/network-metrics-daemon-2xmfq" Jul 12 19:22:17.799376 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.692816 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/51dee771-9663-4366-90ca-9b1c286e35f8-system-cni-dir\") pod \"multus-additional-cni-plugins-prfcd\" (UID: \"51dee771-9663-4366-90ca-9b1c286e35f8\") " pod="openshift-multus/multus-additional-cni-plugins-prfcd" Jul 12 19:22:17.799376 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.692966 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/23c32389-3933-4c29-a625-3e46ade08326-etc-sysconfig\") pod \"tuned-vsllh\" (UID: \"23c32389-3933-4c29-a625-3e46ade08326\") " pod="openshift-cluster-node-tuning-operator/tuned-vsllh" Jul 12 19:22:17.799376 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.693499 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8e016c2e-0c3a-49da-83d4-f526738c1cbd-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-k5h9m\" (UID: \"8e016c2e-0c3a-49da-83d4-f526738c1cbd\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5h9m" Jul 12 19:22:17.799376 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.693557 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/8e016c2e-0c3a-49da-83d4-f526738c1cbd-ovnkube-script-lib\") pod \"ovnkube-node-k5h9m\" (UID: \"8e016c2e-0c3a-49da-83d4-f526738c1cbd\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5h9m" Jul 12 19:22:17.799376 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.693580 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/51dee771-9663-4366-90ca-9b1c286e35f8-os-release\") pod \"multus-additional-cni-plugins-prfcd\" (UID: \"51dee771-9663-4366-90ca-9b1c286e35f8\") " pod="openshift-multus/multus-additional-cni-plugins-prfcd" Jul 12 19:22:17.799376 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.693602 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wftsz\" (UniqueName: \"kubernetes.io/projected/56c9f12c-6e22-414d-a8d7-b503ed1f38df-kube-api-access-wftsz\") pod \"aws-ebs-csi-driver-node-bwmdr\" (UID: \"56c9f12c-6e22-414d-a8d7-b503ed1f38df\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bwmdr" Jul 12 19:22:17.801596 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.693622 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/23c32389-3933-4c29-a625-3e46ade08326-etc-modprobe-d\") pod \"tuned-vsllh\" (UID: \"23c32389-3933-4c29-a625-3e46ade08326\") " pod="openshift-cluster-node-tuning-operator/tuned-vsllh" Jul 12 19:22:17.801596 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.693642 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8e016c2e-0c3a-49da-83d4-f526738c1cbd-etc-openvswitch\") pod \"ovnkube-node-k5h9m\" (UID: \"8e016c2e-0c3a-49da-83d4-f526738c1cbd\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5h9m" Jul 12 19:22:17.801596 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.693699 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/8e016c2e-0c3a-49da-83d4-f526738c1cbd-host-cni-netd\") pod \"ovnkube-node-k5h9m\" (UID: \"8e016c2e-0c3a-49da-83d4-f526738c1cbd\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5h9m" Jul 12 19:22:17.801596 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.693721 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/51dee771-9663-4366-90ca-9b1c286e35f8-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-prfcd\" (UID: \"51dee771-9663-4366-90ca-9b1c286e35f8\") " pod="openshift-multus/multus-additional-cni-plugins-prfcd" Jul 12 19:22:17.801596 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.693755 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b8cca381-48d7-43ac-ae6a-15f96d99345d-serviceca\") pod \"node-ca-c89jb\" (UID: \"b8cca381-48d7-43ac-ae6a-15f96d99345d\") " pod="openshift-image-registry/node-ca-c89jb" Jul 12 19:22:17.801596 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.693775 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/23c32389-3933-4c29-a625-3e46ade08326-etc-kubernetes\") pod \"tuned-vsllh\" (UID: \"23c32389-3933-4c29-a625-3e46ade08326\") " pod="openshift-cluster-node-tuning-operator/tuned-vsllh" Jul 12 19:22:17.801596 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.693794 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbxs5\" (UniqueName: \"kubernetes.io/projected/087e3f6e-992f-4784-a376-bd2af99f9851-kube-api-access-lbxs5\") pod \"iptables-alerter-hrrhk\" (UID: \"087e3f6e-992f-4784-a376-bd2af99f9851\") " pod="openshift-network-operator/iptables-alerter-hrrhk" Jul 12 19:22:17.801596 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.693813 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/23c32389-3933-4c29-a625-3e46ade08326-etc-sysctl-conf\") pod \"tuned-vsllh\" (UID: \"23c32389-3933-4c29-a625-3e46ade08326\") " pod="openshift-cluster-node-tuning-operator/tuned-vsllh" Jul 12 19:22:17.801596 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.693832 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8e016c2e-0c3a-49da-83d4-f526738c1cbd-var-lib-openvswitch\") pod \"ovnkube-node-k5h9m\" (UID: \"8e016c2e-0c3a-49da-83d4-f526738c1cbd\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5h9m" Jul 12 19:22:17.801596 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.693853 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/56c9f12c-6e22-414d-a8d7-b503ed1f38df-sys-fs\") pod \"aws-ebs-csi-driver-node-bwmdr\" (UID: \"56c9f12c-6e22-414d-a8d7-b503ed1f38df\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bwmdr" Jul 12 19:22:17.801596 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.693872 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/23c32389-3933-4c29-a625-3e46ade08326-etc-systemd\") pod \"tuned-vsllh\" (UID: \"23c32389-3933-4c29-a625-3e46ade08326\") " pod="openshift-cluster-node-tuning-operator/tuned-vsllh" Jul 12 19:22:17.801596 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.693906 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/8e016c2e-0c3a-49da-83d4-f526738c1cbd-host-kubelet\") pod \"ovnkube-node-k5h9m\" (UID: \"8e016c2e-0c3a-49da-83d4-f526738c1cbd\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5h9m" Jul 12 19:22:17.801596 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.693926 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/51dee771-9663-4366-90ca-9b1c286e35f8-cnibin\") pod \"multus-additional-cni-plugins-prfcd\" (UID: \"51dee771-9663-4366-90ca-9b1c286e35f8\") " pod="openshift-multus/multus-additional-cni-plugins-prfcd" Jul 12 19:22:17.802652 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.693949 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8e016c2e-0c3a-49da-83d4-f526738c1cbd-run-openvswitch\") pod \"ovnkube-node-k5h9m\" (UID: \"8e016c2e-0c3a-49da-83d4-f526738c1cbd\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5h9m" Jul 12 19:22:17.802652 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.693973 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8e016c2e-0c3a-49da-83d4-f526738c1cbd-host-run-ovn-kubernetes\") pod \"ovnkube-node-k5h9m\" (UID: \"8e016c2e-0c3a-49da-83d4-f526738c1cbd\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5h9m" Jul 12 19:22:17.802652 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.693998 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/23c32389-3933-4c29-a625-3e46ade08326-etc-sysctl-d\") pod \"tuned-vsllh\" (UID: \"23c32389-3933-4c29-a625-3e46ade08326\") " pod="openshift-cluster-node-tuning-operator/tuned-vsllh" Jul 12 19:22:17.802652 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.694034 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/8e016c2e-0c3a-49da-83d4-f526738c1cbd-systemd-units\") pod \"ovnkube-node-k5h9m\" (UID: \"8e016c2e-0c3a-49da-83d4-f526738c1cbd\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5h9m" Jul 12 19:22:17.802652 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.694062 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/8e016c2e-0c3a-49da-83d4-f526738c1cbd-ovn-node-metrics-cert\") pod \"ovnkube-node-k5h9m\" (UID: \"8e016c2e-0c3a-49da-83d4-f526738c1cbd\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5h9m" Jul 12 19:22:17.802652 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.694085 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/56c9f12c-6e22-414d-a8d7-b503ed1f38df-device-dir\") pod \"aws-ebs-csi-driver-node-bwmdr\" (UID: \"56c9f12c-6e22-414d-a8d7-b503ed1f38df\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bwmdr" Jul 12 19:22:17.802652 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.694191 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/8e016c2e-0c3a-49da-83d4-f526738c1cbd-run-systemd\") pod \"ovnkube-node-k5h9m\" (UID: \"8e016c2e-0c3a-49da-83d4-f526738c1cbd\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5h9m" Jul 12 19:22:17.802652 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.694368 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 12 19:22:17.802652 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.694666 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 12 19:22:17.802652 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.694939 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-4cprc\"" Jul 12 19:22:17.802652 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.696329 2565 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 12 19:22:17.802652 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.696652 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-zs4lw" Jul 12 19:22:17.802652 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.698441 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 12 19:22:17.802652 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.701468 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 12 19:22:17.802652 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.701482 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-sf9cw\"" Jul 12 19:22:17.802652 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.703893 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-9klzd" Jul 12 19:22:17.802652 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.705823 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 12 19:22:17.802652 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.706123 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-lv8p7\"" Jul 12 19:22:17.802652 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.706246 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-288bq" Jul 12 19:22:17.802652 ip-10-0-139-170 kubenswrapper[2565]: E0712 19:22:17.706322 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-288bq" podUID="a7a0abe6-09ba-429e-a7f2-8de5b3f5d620" Jul 12 19:22:17.802652 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.710019 2565 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-p9jps" Jul 12 19:22:17.802652 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.717030 2565 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-p9jps" Jul 12 19:22:17.804797 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.779454 2565 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 12 19:22:17.804797 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.795106 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/8e016c2e-0c3a-49da-83d4-f526738c1cbd-env-overrides\") pod \"ovnkube-node-k5h9m\" (UID: \"8e016c2e-0c3a-49da-83d4-f526738c1cbd\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5h9m" Jul 12 19:22:17.804797 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.795139 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/56c9f12c-6e22-414d-a8d7-b503ed1f38df-etc-selinux\") pod \"aws-ebs-csi-driver-node-bwmdr\" (UID: \"56c9f12c-6e22-414d-a8d7-b503ed1f38df\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bwmdr" Jul 12 19:22:17.804797 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.795170 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bdb521c9-c545-445b-aee4-872355e9eb78-system-cni-dir\") pod \"multus-9klzd\" (UID: \"bdb521c9-c545-445b-aee4-872355e9eb78\") " pod="openshift-multus/multus-9klzd" Jul 12 19:22:17.804797 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.795193 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8e016c2e-0c3a-49da-83d4-f526738c1cbd-host-run-netns\") pod \"ovnkube-node-k5h9m\" (UID: \"8e016c2e-0c3a-49da-83d4-f526738c1cbd\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5h9m" Jul 12 19:22:17.804797 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.795214 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e7722ae2-e293-478e-9ac8-ea376f8d583f-metrics-certs\") pod \"network-metrics-daemon-2xmfq\" (UID: \"e7722ae2-e293-478e-9ac8-ea376f8d583f\") " pod="openshift-multus/network-metrics-daemon-2xmfq" Jul 12 19:22:17.804797 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.795233 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b8cca381-48d7-43ac-ae6a-15f96d99345d-host\") pod \"node-ca-c89jb\" (UID: \"b8cca381-48d7-43ac-ae6a-15f96d99345d\") " pod="openshift-image-registry/node-ca-c89jb" Jul 12 19:22:17.804797 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.795254 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bp96h\" (UniqueName: \"kubernetes.io/projected/10c67d44-bdd0-4530-9f5d-fa10238ee66e-kube-api-access-bp96h\") pod \"network-check-target-76gfm\" (UID: \"10c67d44-bdd0-4530-9f5d-fa10238ee66e\") " pod="openshift-network-diagnostics/network-check-target-76gfm" Jul 12 19:22:17.804797 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.795275 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/bdb521c9-c545-445b-aee4-872355e9eb78-os-release\") pod \"multus-9klzd\" (UID: \"bdb521c9-c545-445b-aee4-872355e9eb78\") " pod="openshift-multus/multus-9klzd" Jul 12 19:22:17.804797 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.795306 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/bdb521c9-c545-445b-aee4-872355e9eb78-multus-socket-dir-parent\") pod \"multus-9klzd\" (UID: \"bdb521c9-c545-445b-aee4-872355e9eb78\") " pod="openshift-multus/multus-9klzd" Jul 12 19:22:17.804797 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.795327 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/bdb521c9-c545-445b-aee4-872355e9eb78-multus-daemon-config\") pod \"multus-9klzd\" (UID: \"bdb521c9-c545-445b-aee4-872355e9eb78\") " pod="openshift-multus/multus-9klzd" Jul 12 19:22:17.804797 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.795353 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/23c32389-3933-4c29-a625-3e46ade08326-run\") pod \"tuned-vsllh\" (UID: \"23c32389-3933-4c29-a625-3e46ade08326\") " pod="openshift-cluster-node-tuning-operator/tuned-vsllh" Jul 12 19:22:17.804797 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.795374 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8e016c2e-0c3a-49da-83d4-f526738c1cbd-host-slash\") pod \"ovnkube-node-k5h9m\" (UID: \"8e016c2e-0c3a-49da-83d4-f526738c1cbd\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5h9m" Jul 12 19:22:17.804797 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.795397 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8e016c2e-0c3a-49da-83d4-f526738c1cbd-host-cni-bin\") pod \"ovnkube-node-k5h9m\" (UID: \"8e016c2e-0c3a-49da-83d4-f526738c1cbd\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5h9m" Jul 12 19:22:17.804797 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.795434 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/51dee771-9663-4366-90ca-9b1c286e35f8-cni-binary-copy\") pod \"multus-additional-cni-plugins-prfcd\" (UID: \"51dee771-9663-4366-90ca-9b1c286e35f8\") " pod="openshift-multus/multus-additional-cni-plugins-prfcd" Jul 12 19:22:17.805834 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.795454 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/56c9f12c-6e22-414d-a8d7-b503ed1f38df-kubelet-dir\") pod \"aws-ebs-csi-driver-node-bwmdr\" (UID: \"56c9f12c-6e22-414d-a8d7-b503ed1f38df\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bwmdr" Jul 12 19:22:17.805834 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.795474 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/56c9f12c-6e22-414d-a8d7-b503ed1f38df-registration-dir\") pod \"aws-ebs-csi-driver-node-bwmdr\" (UID: \"56c9f12c-6e22-414d-a8d7-b503ed1f38df\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bwmdr" Jul 12 19:22:17.805834 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.795493 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bdb521c9-c545-445b-aee4-872355e9eb78-cni-binary-copy\") pod \"multus-9klzd\" (UID: \"bdb521c9-c545-445b-aee4-872355e9eb78\") " pod="openshift-multus/multus-9klzd" Jul 12 19:22:17.805834 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.795512 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/23c32389-3933-4c29-a625-3e46ade08326-lib-modules\") pod \"tuned-vsllh\" (UID: \"23c32389-3933-4c29-a625-3e46ade08326\") " pod="openshift-cluster-node-tuning-operator/tuned-vsllh" Jul 12 19:22:17.805834 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.795532 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/23c32389-3933-4c29-a625-3e46ade08326-tmp\") pod \"tuned-vsllh\" (UID: \"23c32389-3933-4c29-a625-3e46ade08326\") " pod="openshift-cluster-node-tuning-operator/tuned-vsllh" Jul 12 19:22:17.805834 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.795553 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxwbs\" (UniqueName: \"kubernetes.io/projected/51dee771-9663-4366-90ca-9b1c286e35f8-kube-api-access-rxwbs\") pod \"multus-additional-cni-plugins-prfcd\" (UID: \"51dee771-9663-4366-90ca-9b1c286e35f8\") " pod="openshift-multus/multus-additional-cni-plugins-prfcd" Jul 12 19:22:17.805834 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.795591 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czqm5\" (UniqueName: \"kubernetes.io/projected/b8cca381-48d7-43ac-ae6a-15f96d99345d-kube-api-access-czqm5\") pod \"node-ca-c89jb\" (UID: \"b8cca381-48d7-43ac-ae6a-15f96d99345d\") " pod="openshift-image-registry/node-ca-c89jb" Jul 12 19:22:17.805834 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.795612 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/bdb521c9-c545-445b-aee4-872355e9eb78-host-var-lib-cni-multus\") pod \"multus-9klzd\" (UID: \"bdb521c9-c545-445b-aee4-872355e9eb78\") " pod="openshift-multus/multus-9klzd" Jul 12 19:22:17.805834 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.795632 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/8e016c2e-0c3a-49da-83d4-f526738c1cbd-run-ovn\") pod \"ovnkube-node-k5h9m\" (UID: \"8e016c2e-0c3a-49da-83d4-f526738c1cbd\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5h9m" Jul 12 19:22:17.805834 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.795655 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nj2g8\" (UniqueName: \"kubernetes.io/projected/e7722ae2-e293-478e-9ac8-ea376f8d583f-kube-api-access-nj2g8\") pod \"network-metrics-daemon-2xmfq\" (UID: \"e7722ae2-e293-478e-9ac8-ea376f8d583f\") " pod="openshift-multus/network-metrics-daemon-2xmfq" Jul 12 19:22:17.805834 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.795690 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/51dee771-9663-4366-90ca-9b1c286e35f8-system-cni-dir\") pod \"multus-additional-cni-plugins-prfcd\" (UID: \"51dee771-9663-4366-90ca-9b1c286e35f8\") " pod="openshift-multus/multus-additional-cni-plugins-prfcd" Jul 12 19:22:17.805834 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.795715 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/23c32389-3933-4c29-a625-3e46ade08326-etc-sysconfig\") pod \"tuned-vsllh\" (UID: \"23c32389-3933-4c29-a625-3e46ade08326\") " pod="openshift-cluster-node-tuning-operator/tuned-vsllh" Jul 12 19:22:17.805834 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.795763 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8e016c2e-0c3a-49da-83d4-f526738c1cbd-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-k5h9m\" (UID: \"8e016c2e-0c3a-49da-83d4-f526738c1cbd\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5h9m" Jul 12 19:22:17.805834 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.795785 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/8e016c2e-0c3a-49da-83d4-f526738c1cbd-ovnkube-script-lib\") pod \"ovnkube-node-k5h9m\" (UID: \"8e016c2e-0c3a-49da-83d4-f526738c1cbd\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5h9m" Jul 12 19:22:17.812501 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.795805 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/51dee771-9663-4366-90ca-9b1c286e35f8-os-release\") pod \"multus-additional-cni-plugins-prfcd\" (UID: \"51dee771-9663-4366-90ca-9b1c286e35f8\") " pod="openshift-multus/multus-additional-cni-plugins-prfcd" Jul 12 19:22:17.812501 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.795835 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wftsz\" (UniqueName: \"kubernetes.io/projected/56c9f12c-6e22-414d-a8d7-b503ed1f38df-kube-api-access-wftsz\") pod \"aws-ebs-csi-driver-node-bwmdr\" (UID: \"56c9f12c-6e22-414d-a8d7-b503ed1f38df\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bwmdr" Jul 12 19:22:17.812501 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.795855 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a7a0abe6-09ba-429e-a7f2-8de5b3f5d620-dbus\") pod \"global-pull-secret-syncer-288bq\" (UID: \"a7a0abe6-09ba-429e-a7f2-8de5b3f5d620\") " pod="kube-system/global-pull-secret-syncer-288bq" Jul 12 19:22:17.812501 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.795875 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/23c32389-3933-4c29-a625-3e46ade08326-etc-modprobe-d\") pod \"tuned-vsllh\" (UID: \"23c32389-3933-4c29-a625-3e46ade08326\") " pod="openshift-cluster-node-tuning-operator/tuned-vsllh" Jul 12 19:22:17.812501 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.795925 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8e016c2e-0c3a-49da-83d4-f526738c1cbd-etc-openvswitch\") pod \"ovnkube-node-k5h9m\" (UID: \"8e016c2e-0c3a-49da-83d4-f526738c1cbd\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5h9m" Jul 12 19:22:17.812501 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.795962 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/8e016c2e-0c3a-49da-83d4-f526738c1cbd-host-cni-netd\") pod \"ovnkube-node-k5h9m\" (UID: \"8e016c2e-0c3a-49da-83d4-f526738c1cbd\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5h9m" Jul 12 19:22:17.812501 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.795992 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/51dee771-9663-4366-90ca-9b1c286e35f8-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-prfcd\" (UID: \"51dee771-9663-4366-90ca-9b1c286e35f8\") " pod="openshift-multus/multus-additional-cni-plugins-prfcd" Jul 12 19:22:17.812501 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.796014 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b8cca381-48d7-43ac-ae6a-15f96d99345d-serviceca\") pod \"node-ca-c89jb\" (UID: \"b8cca381-48d7-43ac-ae6a-15f96d99345d\") " pod="openshift-image-registry/node-ca-c89jb" Jul 12 19:22:17.812501 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.796033 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/bdb521c9-c545-445b-aee4-872355e9eb78-host-run-multus-certs\") pod \"multus-9klzd\" (UID: \"bdb521c9-c545-445b-aee4-872355e9eb78\") " pod="openshift-multus/multus-9klzd" Jul 12 19:22:17.812501 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.796053 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/23c32389-3933-4c29-a625-3e46ade08326-etc-kubernetes\") pod \"tuned-vsllh\" (UID: \"23c32389-3933-4c29-a625-3e46ade08326\") " pod="openshift-cluster-node-tuning-operator/tuned-vsllh" Jul 12 19:22:17.812501 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.796074 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbxs5\" (UniqueName: \"kubernetes.io/projected/087e3f6e-992f-4784-a376-bd2af99f9851-kube-api-access-lbxs5\") pod \"iptables-alerter-hrrhk\" (UID: \"087e3f6e-992f-4784-a376-bd2af99f9851\") " pod="openshift-network-operator/iptables-alerter-hrrhk" Jul 12 19:22:17.812501 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.796094 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bdb521c9-c545-445b-aee4-872355e9eb78-host-var-lib-kubelet\") pod \"multus-9klzd\" (UID: \"bdb521c9-c545-445b-aee4-872355e9eb78\") " pod="openshift-multus/multus-9klzd" Jul 12 19:22:17.812501 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.796121 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bdb521c9-c545-445b-aee4-872355e9eb78-multus-conf-dir\") pod \"multus-9klzd\" (UID: \"bdb521c9-c545-445b-aee4-872355e9eb78\") " pod="openshift-multus/multus-9klzd" Jul 12 19:22:17.812501 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.796145 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/23c32389-3933-4c29-a625-3e46ade08326-etc-sysctl-conf\") pod \"tuned-vsllh\" (UID: \"23c32389-3933-4c29-a625-3e46ade08326\") " pod="openshift-cluster-node-tuning-operator/tuned-vsllh" Jul 12 19:22:17.813310 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.796201 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8e016c2e-0c3a-49da-83d4-f526738c1cbd-var-lib-openvswitch\") pod \"ovnkube-node-k5h9m\" (UID: \"8e016c2e-0c3a-49da-83d4-f526738c1cbd\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5h9m" Jul 12 19:22:17.813310 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.796235 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/56c9f12c-6e22-414d-a8d7-b503ed1f38df-sys-fs\") pod \"aws-ebs-csi-driver-node-bwmdr\" (UID: \"56c9f12c-6e22-414d-a8d7-b503ed1f38df\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bwmdr" Jul 12 19:22:17.813310 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.796279 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bdb521c9-c545-445b-aee4-872355e9eb78-host-var-lib-cni-bin\") pod \"multus-9klzd\" (UID: \"bdb521c9-c545-445b-aee4-872355e9eb78\") " pod="openshift-multus/multus-9klzd" Jul 12 19:22:17.813310 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.796312 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/23c32389-3933-4c29-a625-3e46ade08326-etc-systemd\") pod \"tuned-vsllh\" (UID: \"23c32389-3933-4c29-a625-3e46ade08326\") " pod="openshift-cluster-node-tuning-operator/tuned-vsllh" Jul 12 19:22:17.813310 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.796333 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/8e016c2e-0c3a-49da-83d4-f526738c1cbd-host-kubelet\") pod \"ovnkube-node-k5h9m\" (UID: \"8e016c2e-0c3a-49da-83d4-f526738c1cbd\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5h9m" Jul 12 19:22:17.813310 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.796353 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/51dee771-9663-4366-90ca-9b1c286e35f8-cnibin\") pod \"multus-additional-cni-plugins-prfcd\" (UID: \"51dee771-9663-4366-90ca-9b1c286e35f8\") " pod="openshift-multus/multus-additional-cni-plugins-prfcd" Jul 12 19:22:17.813310 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.796373 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bdb521c9-c545-445b-aee4-872355e9eb78-etc-kubernetes\") pod \"multus-9klzd\" (UID: \"bdb521c9-c545-445b-aee4-872355e9eb78\") " pod="openshift-multus/multus-9klzd" Jul 12 19:22:17.813310 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.796394 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8e016c2e-0c3a-49da-83d4-f526738c1cbd-run-openvswitch\") pod \"ovnkube-node-k5h9m\" (UID: \"8e016c2e-0c3a-49da-83d4-f526738c1cbd\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5h9m" Jul 12 19:22:17.813310 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.796419 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8e016c2e-0c3a-49da-83d4-f526738c1cbd-host-run-ovn-kubernetes\") pod \"ovnkube-node-k5h9m\" (UID: \"8e016c2e-0c3a-49da-83d4-f526738c1cbd\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5h9m" Jul 12 19:22:17.813310 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.796439 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8e832009-b138-4442-b9db-10e73957481e-hosts-file\") pod \"node-resolver-zs4lw\" (UID: \"8e832009-b138-4442-b9db-10e73957481e\") " pod="openshift-dns/node-resolver-zs4lw" Jul 12 19:22:17.813310 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.796459 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2hmn\" (UniqueName: \"kubernetes.io/projected/8e832009-b138-4442-b9db-10e73957481e-kube-api-access-h2hmn\") pod \"node-resolver-zs4lw\" (UID: \"8e832009-b138-4442-b9db-10e73957481e\") " pod="openshift-dns/node-resolver-zs4lw" Jul 12 19:22:17.813310 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.796489 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/23c32389-3933-4c29-a625-3e46ade08326-etc-sysctl-d\") pod \"tuned-vsllh\" (UID: \"23c32389-3933-4c29-a625-3e46ade08326\") " pod="openshift-cluster-node-tuning-operator/tuned-vsllh" Jul 12 19:22:17.813310 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.796510 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/8e016c2e-0c3a-49da-83d4-f526738c1cbd-systemd-units\") pod \"ovnkube-node-k5h9m\" (UID: \"8e016c2e-0c3a-49da-83d4-f526738c1cbd\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5h9m" Jul 12 19:22:17.813310 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.796532 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/8e016c2e-0c3a-49da-83d4-f526738c1cbd-ovn-node-metrics-cert\") pod \"ovnkube-node-k5h9m\" (UID: \"8e016c2e-0c3a-49da-83d4-f526738c1cbd\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5h9m" Jul 12 19:22:17.814186 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.796560 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/56c9f12c-6e22-414d-a8d7-b503ed1f38df-device-dir\") pod \"aws-ebs-csi-driver-node-bwmdr\" (UID: \"56c9f12c-6e22-414d-a8d7-b503ed1f38df\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bwmdr" Jul 12 19:22:17.814186 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.796581 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/26ba017f-506c-4e00-9b24-4932c97ad489-agent-certs\") pod \"konnectivity-agent-7vvbp\" (UID: \"26ba017f-506c-4e00-9b24-4932c97ad489\") " pod="kube-system/konnectivity-agent-7vvbp" Jul 12 19:22:17.814186 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.796634 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bdb521c9-c545-445b-aee4-872355e9eb78-multus-cni-dir\") pod \"multus-9klzd\" (UID: \"bdb521c9-c545-445b-aee4-872355e9eb78\") " pod="openshift-multus/multus-9klzd" Jul 12 19:22:17.814186 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.796656 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/8e016c2e-0c3a-49da-83d4-f526738c1cbd-run-systemd\") pod \"ovnkube-node-k5h9m\" (UID: \"8e016c2e-0c3a-49da-83d4-f526738c1cbd\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5h9m" Jul 12 19:22:17.814186 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.796677 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/8e016c2e-0c3a-49da-83d4-f526738c1cbd-log-socket\") pod \"ovnkube-node-k5h9m\" (UID: \"8e016c2e-0c3a-49da-83d4-f526738c1cbd\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5h9m" Jul 12 19:22:17.814186 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.796703 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/8e016c2e-0c3a-49da-83d4-f526738c1cbd-ovnkube-config\") pod \"ovnkube-node-k5h9m\" (UID: \"8e016c2e-0c3a-49da-83d4-f526738c1cbd\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5h9m" Jul 12 19:22:17.814186 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.796732 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/087e3f6e-992f-4784-a376-bd2af99f9851-host-slash\") pod \"iptables-alerter-hrrhk\" (UID: \"087e3f6e-992f-4784-a376-bd2af99f9851\") " pod="openshift-network-operator/iptables-alerter-hrrhk" Jul 12 19:22:17.814186 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.796758 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/8e016c2e-0c3a-49da-83d4-f526738c1cbd-node-log\") pod \"ovnkube-node-k5h9m\" (UID: \"8e016c2e-0c3a-49da-83d4-f526738c1cbd\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5h9m" Jul 12 19:22:17.814186 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.796787 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-csp9b\" (UniqueName: \"kubernetes.io/projected/8e016c2e-0c3a-49da-83d4-f526738c1cbd-kube-api-access-csp9b\") pod \"ovnkube-node-k5h9m\" (UID: \"8e016c2e-0c3a-49da-83d4-f526738c1cbd\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5h9m" Jul 12 19:22:17.814186 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.796808 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/51dee771-9663-4366-90ca-9b1c286e35f8-tuning-conf-dir\") pod \"multus-additional-cni-plugins-prfcd\" (UID: \"51dee771-9663-4366-90ca-9b1c286e35f8\") " pod="openshift-multus/multus-additional-cni-plugins-prfcd" Jul 12 19:22:17.814186 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.796833 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e832009-b138-4442-b9db-10e73957481e-tmp-dir\") pod \"node-resolver-zs4lw\" (UID: \"8e832009-b138-4442-b9db-10e73957481e\") " pod="openshift-dns/node-resolver-zs4lw" Jul 12 19:22:17.814186 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.796856 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bdb521c9-c545-445b-aee4-872355e9eb78-host-run-netns\") pod \"multus-9klzd\" (UID: \"bdb521c9-c545-445b-aee4-872355e9eb78\") " pod="openshift-multus/multus-9klzd" Jul 12 19:22:17.814186 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.796912 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/23c32389-3933-4c29-a625-3e46ade08326-var-lib-kubelet\") pod \"tuned-vsllh\" (UID: \"23c32389-3933-4c29-a625-3e46ade08326\") " pod="openshift-cluster-node-tuning-operator/tuned-vsllh" Jul 12 19:22:17.814186 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.796934 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5h9cz\" (UniqueName: \"kubernetes.io/projected/23c32389-3933-4c29-a625-3e46ade08326-kube-api-access-5h9cz\") pod \"tuned-vsllh\" (UID: \"23c32389-3933-4c29-a625-3e46ade08326\") " pod="openshift-cluster-node-tuning-operator/tuned-vsllh" Jul 12 19:22:17.815144 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.796963 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/51dee771-9663-4366-90ca-9b1c286e35f8-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-prfcd\" (UID: \"51dee771-9663-4366-90ca-9b1c286e35f8\") " pod="openshift-multus/multus-additional-cni-plugins-prfcd" Jul 12 19:22:17.815144 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.796988 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/56c9f12c-6e22-414d-a8d7-b503ed1f38df-socket-dir\") pod \"aws-ebs-csi-driver-node-bwmdr\" (UID: \"56c9f12c-6e22-414d-a8d7-b503ed1f38df\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bwmdr" Jul 12 19:22:17.815144 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.797013 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/26ba017f-506c-4e00-9b24-4932c97ad489-konnectivity-ca\") pod \"konnectivity-agent-7vvbp\" (UID: \"26ba017f-506c-4e00-9b24-4932c97ad489\") " pod="kube-system/konnectivity-agent-7vvbp" Jul 12 19:22:17.815144 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.797033 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/bdb521c9-c545-445b-aee4-872355e9eb78-hostroot\") pod \"multus-9klzd\" (UID: \"bdb521c9-c545-445b-aee4-872355e9eb78\") " pod="openshift-multus/multus-9klzd" Jul 12 19:22:17.815144 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.797086 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58dpr\" (UniqueName: \"kubernetes.io/projected/bdb521c9-c545-445b-aee4-872355e9eb78-kube-api-access-58dpr\") pod \"multus-9klzd\" (UID: \"bdb521c9-c545-445b-aee4-872355e9eb78\") " pod="openshift-multus/multus-9klzd" Jul 12 19:22:17.815144 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.797106 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a7a0abe6-09ba-429e-a7f2-8de5b3f5d620-kubelet-config\") pod \"global-pull-secret-syncer-288bq\" (UID: \"a7a0abe6-09ba-429e-a7f2-8de5b3f5d620\") " pod="kube-system/global-pull-secret-syncer-288bq" Jul 12 19:22:17.815144 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.797143 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/23c32389-3933-4c29-a625-3e46ade08326-host\") pod \"tuned-vsllh\" (UID: \"23c32389-3933-4c29-a625-3e46ade08326\") " pod="openshift-cluster-node-tuning-operator/tuned-vsllh" Jul 12 19:22:17.815144 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.797167 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/23c32389-3933-4c29-a625-3e46ade08326-sys\") pod \"tuned-vsllh\" (UID: \"23c32389-3933-4c29-a625-3e46ade08326\") " pod="openshift-cluster-node-tuning-operator/tuned-vsllh" Jul 12 19:22:17.815144 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.797192 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/23c32389-3933-4c29-a625-3e46ade08326-etc-tuned\") pod \"tuned-vsllh\" (UID: \"23c32389-3933-4c29-a625-3e46ade08326\") " pod="openshift-cluster-node-tuning-operator/tuned-vsllh" Jul 12 19:22:17.815144 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.797221 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/087e3f6e-992f-4784-a376-bd2af99f9851-iptables-alerter-script\") pod \"iptables-alerter-hrrhk\" (UID: \"087e3f6e-992f-4784-a376-bd2af99f9851\") " pod="openshift-network-operator/iptables-alerter-hrrhk" Jul 12 19:22:17.815144 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.797258 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bdb521c9-c545-445b-aee4-872355e9eb78-cnibin\") pod \"multus-9klzd\" (UID: \"bdb521c9-c545-445b-aee4-872355e9eb78\") " pod="openshift-multus/multus-9klzd" Jul 12 19:22:17.815144 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.797279 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/bdb521c9-c545-445b-aee4-872355e9eb78-host-run-k8s-cni-cncf-io\") pod \"multus-9klzd\" (UID: \"bdb521c9-c545-445b-aee4-872355e9eb78\") " pod="openshift-multus/multus-9klzd" Jul 12 19:22:17.815144 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.797312 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/8e016c2e-0c3a-49da-83d4-f526738c1cbd-run-systemd\") pod \"ovnkube-node-k5h9m\" (UID: \"8e016c2e-0c3a-49da-83d4-f526738c1cbd\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5h9m" Jul 12 19:22:17.815144 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.797350 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/8e016c2e-0c3a-49da-83d4-f526738c1cbd-log-socket\") pod \"ovnkube-node-k5h9m\" (UID: \"8e016c2e-0c3a-49da-83d4-f526738c1cbd\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5h9m" Jul 12 19:22:17.816061 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.797480 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/23c32389-3933-4c29-a625-3e46ade08326-etc-sysctl-conf\") pod \"tuned-vsllh\" (UID: \"23c32389-3933-4c29-a625-3e46ade08326\") " pod="openshift-cluster-node-tuning-operator/tuned-vsllh" Jul 12 19:22:17.816061 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.797528 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8e016c2e-0c3a-49da-83d4-f526738c1cbd-var-lib-openvswitch\") pod \"ovnkube-node-k5h9m\" (UID: \"8e016c2e-0c3a-49da-83d4-f526738c1cbd\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5h9m" Jul 12 19:22:17.816061 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.797579 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/56c9f12c-6e22-414d-a8d7-b503ed1f38df-sys-fs\") pod \"aws-ebs-csi-driver-node-bwmdr\" (UID: \"56c9f12c-6e22-414d-a8d7-b503ed1f38df\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bwmdr" Jul 12 19:22:17.816061 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.797695 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/23c32389-3933-4c29-a625-3e46ade08326-etc-systemd\") pod \"tuned-vsllh\" (UID: \"23c32389-3933-4c29-a625-3e46ade08326\") " pod="openshift-cluster-node-tuning-operator/tuned-vsllh" Jul 12 19:22:17.816061 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.797713 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/23c32389-3933-4c29-a625-3e46ade08326-etc-sysctl-d\") pod \"tuned-vsllh\" (UID: \"23c32389-3933-4c29-a625-3e46ade08326\") " pod="openshift-cluster-node-tuning-operator/tuned-vsllh" Jul 12 19:22:17.816061 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.797750 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/8e016c2e-0c3a-49da-83d4-f526738c1cbd-host-kubelet\") pod \"ovnkube-node-k5h9m\" (UID: \"8e016c2e-0c3a-49da-83d4-f526738c1cbd\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5h9m" Jul 12 19:22:17.816061 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.797877 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/56c9f12c-6e22-414d-a8d7-b503ed1f38df-device-dir\") pod \"aws-ebs-csi-driver-node-bwmdr\" (UID: \"56c9f12c-6e22-414d-a8d7-b503ed1f38df\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bwmdr" Jul 12 19:22:17.816061 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.797956 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/8e016c2e-0c3a-49da-83d4-f526738c1cbd-systemd-units\") pod \"ovnkube-node-k5h9m\" (UID: \"8e016c2e-0c3a-49da-83d4-f526738c1cbd\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5h9m" Jul 12 19:22:17.816061 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.797958 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8e016c2e-0c3a-49da-83d4-f526738c1cbd-host-run-netns\") pod \"ovnkube-node-k5h9m\" (UID: \"8e016c2e-0c3a-49da-83d4-f526738c1cbd\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5h9m" Jul 12 19:22:17.816061 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.797959 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/8e016c2e-0c3a-49da-83d4-f526738c1cbd-ovnkube-config\") pod \"ovnkube-node-k5h9m\" (UID: \"8e016c2e-0c3a-49da-83d4-f526738c1cbd\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5h9m" Jul 12 19:22:17.816061 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.797991 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/51dee771-9663-4366-90ca-9b1c286e35f8-cnibin\") pod \"multus-additional-cni-plugins-prfcd\" (UID: \"51dee771-9663-4366-90ca-9b1c286e35f8\") " pod="openshift-multus/multus-additional-cni-plugins-prfcd" Jul 12 19:22:17.816061 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.798009 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/23c32389-3933-4c29-a625-3e46ade08326-sys\") pod \"tuned-vsllh\" (UID: \"23c32389-3933-4c29-a625-3e46ade08326\") " pod="openshift-cluster-node-tuning-operator/tuned-vsllh" Jul 12 19:22:17.816061 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.798092 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/56c9f12c-6e22-414d-a8d7-b503ed1f38df-socket-dir\") pod \"aws-ebs-csi-driver-node-bwmdr\" (UID: \"56c9f12c-6e22-414d-a8d7-b503ed1f38df\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bwmdr" Jul 12 19:22:17.816061 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.798142 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/087e3f6e-992f-4784-a376-bd2af99f9851-host-slash\") pod \"iptables-alerter-hrrhk\" (UID: \"087e3f6e-992f-4784-a376-bd2af99f9851\") " pod="openshift-network-operator/iptables-alerter-hrrhk" Jul 12 19:22:17.816061 ip-10-0-139-170 kubenswrapper[2565]: E0712 19:22:17.798157 2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 12 19:22:17.816868 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.798173 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8e016c2e-0c3a-49da-83d4-f526738c1cbd-run-openvswitch\") pod \"ovnkube-node-k5h9m\" (UID: \"8e016c2e-0c3a-49da-83d4-f526738c1cbd\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5h9m" Jul 12 19:22:17.816868 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.798208 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/8e016c2e-0c3a-49da-83d4-f526738c1cbd-node-log\") pod \"ovnkube-node-k5h9m\" (UID: \"8e016c2e-0c3a-49da-83d4-f526738c1cbd\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5h9m" Jul 12 19:22:17.816868 ip-10-0-139-170 kubenswrapper[2565]: E0712 19:22:17.798225 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e7722ae2-e293-478e-9ac8-ea376f8d583f-metrics-certs podName:e7722ae2-e293-478e-9ac8-ea376f8d583f nodeName:}" failed. No retries permitted until 2026-07-12 19:22:18.298205662 +0000 UTC m=+3.030778144 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e7722ae2-e293-478e-9ac8-ea376f8d583f-metrics-certs") pod "network-metrics-daemon-2xmfq" (UID: "e7722ae2-e293-478e-9ac8-ea376f8d583f") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 12 19:22:17.816868 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.798307 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/8e016c2e-0c3a-49da-83d4-f526738c1cbd-env-overrides\") pod \"ovnkube-node-k5h9m\" (UID: \"8e016c2e-0c3a-49da-83d4-f526738c1cbd\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5h9m" Jul 12 19:22:17.816868 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.798371 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a7a0abe6-09ba-429e-a7f2-8de5b3f5d620-original-pull-secret\") pod \"global-pull-secret-syncer-288bq\" (UID: \"a7a0abe6-09ba-429e-a7f2-8de5b3f5d620\") " pod="kube-system/global-pull-secret-syncer-288bq" Jul 12 19:22:17.816868 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.798416 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b8cca381-48d7-43ac-ae6a-15f96d99345d-host\") pod \"node-ca-c89jb\" (UID: \"b8cca381-48d7-43ac-ae6a-15f96d99345d\") " pod="openshift-image-registry/node-ca-c89jb" Jul 12 19:22:17.816868 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.798541 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/23c32389-3933-4c29-a625-3e46ade08326-run\") pod \"tuned-vsllh\" (UID: \"23c32389-3933-4c29-a625-3e46ade08326\") " pod="openshift-cluster-node-tuning-operator/tuned-vsllh" Jul 12 19:22:17.816868 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.798601 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8e016c2e-0c3a-49da-83d4-f526738c1cbd-host-slash\") pod \"ovnkube-node-k5h9m\" (UID: \"8e016c2e-0c3a-49da-83d4-f526738c1cbd\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5h9m" Jul 12 19:22:17.816868 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.798623 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/8e016c2e-0c3a-49da-83d4-f526738c1cbd-run-ovn\") pod \"ovnkube-node-k5h9m\" (UID: \"8e016c2e-0c3a-49da-83d4-f526738c1cbd\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5h9m" Jul 12 19:22:17.816868 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.798636 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/23c32389-3933-4c29-a625-3e46ade08326-etc-modprobe-d\") pod \"tuned-vsllh\" (UID: \"23c32389-3933-4c29-a625-3e46ade08326\") " pod="openshift-cluster-node-tuning-operator/tuned-vsllh" Jul 12 19:22:17.816868 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.798640 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8e016c2e-0c3a-49da-83d4-f526738c1cbd-host-cni-bin\") pod \"ovnkube-node-k5h9m\" (UID: \"8e016c2e-0c3a-49da-83d4-f526738c1cbd\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5h9m" Jul 12 19:22:17.816868 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.798690 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/8e016c2e-0c3a-49da-83d4-f526738c1cbd-host-cni-netd\") pod \"ovnkube-node-k5h9m\" (UID: \"8e016c2e-0c3a-49da-83d4-f526738c1cbd\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5h9m" Jul 12 19:22:17.816868 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.798777 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/56c9f12c-6e22-414d-a8d7-b503ed1f38df-registration-dir\") pod \"aws-ebs-csi-driver-node-bwmdr\" (UID: \"56c9f12c-6e22-414d-a8d7-b503ed1f38df\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bwmdr" Jul 12 19:22:17.816868 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.798774 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/51dee771-9663-4366-90ca-9b1c286e35f8-tuning-conf-dir\") pod \"multus-additional-cni-plugins-prfcd\" (UID: \"51dee771-9663-4366-90ca-9b1c286e35f8\") " pod="openshift-multus/multus-additional-cni-plugins-prfcd" Jul 12 19:22:17.817829 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.798834 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/23c32389-3933-4c29-a625-3e46ade08326-var-lib-kubelet\") pod \"tuned-vsllh\" (UID: \"23c32389-3933-4c29-a625-3e46ade08326\") " pod="openshift-cluster-node-tuning-operator/tuned-vsllh" Jul 12 19:22:17.817829 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.798897 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/23c32389-3933-4c29-a625-3e46ade08326-lib-modules\") pod \"tuned-vsllh\" (UID: \"23c32389-3933-4c29-a625-3e46ade08326\") " pod="openshift-cluster-node-tuning-operator/tuned-vsllh" Jul 12 19:22:17.817829 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.798966 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8e016c2e-0c3a-49da-83d4-f526738c1cbd-host-run-ovn-kubernetes\") pod \"ovnkube-node-k5h9m\" (UID: \"8e016c2e-0c3a-49da-83d4-f526738c1cbd\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5h9m" Jul 12 19:22:17.817829 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.799020 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/51dee771-9663-4366-90ca-9b1c286e35f8-system-cni-dir\") pod \"multus-additional-cni-plugins-prfcd\" (UID: \"51dee771-9663-4366-90ca-9b1c286e35f8\") " pod="openshift-multus/multus-additional-cni-plugins-prfcd" Jul 12 19:22:17.817829 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.799132 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/23c32389-3933-4c29-a625-3e46ade08326-etc-sysconfig\") pod \"tuned-vsllh\" (UID: \"23c32389-3933-4c29-a625-3e46ade08326\") " pod="openshift-cluster-node-tuning-operator/tuned-vsllh" Jul 12 19:22:17.817829 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.799227 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8e016c2e-0c3a-49da-83d4-f526738c1cbd-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-k5h9m\" (UID: \"8e016c2e-0c3a-49da-83d4-f526738c1cbd\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5h9m" Jul 12 19:22:17.817829 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.799230 2565 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 12 19:22:17.817829 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.799406 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/51dee771-9663-4366-90ca-9b1c286e35f8-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-prfcd\" (UID: \"51dee771-9663-4366-90ca-9b1c286e35f8\") " pod="openshift-multus/multus-additional-cni-plugins-prfcd" Jul 12 19:22:17.817829 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.799780 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/087e3f6e-992f-4784-a376-bd2af99f9851-iptables-alerter-script\") pod \"iptables-alerter-hrrhk\" (UID: \"087e3f6e-992f-4784-a376-bd2af99f9851\") " pod="openshift-network-operator/iptables-alerter-hrrhk" Jul 12 19:22:17.817829 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.799911 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/56c9f12c-6e22-414d-a8d7-b503ed1f38df-etc-selinux\") pod \"aws-ebs-csi-driver-node-bwmdr\" (UID: \"56c9f12c-6e22-414d-a8d7-b503ed1f38df\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bwmdr" Jul 12 19:22:17.817829 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.800032 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/23c32389-3933-4c29-a625-3e46ade08326-etc-kubernetes\") pod \"tuned-vsllh\" (UID: \"23c32389-3933-4c29-a625-3e46ade08326\") " pod="openshift-cluster-node-tuning-operator/tuned-vsllh" Jul 12 19:22:17.817829 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.800321 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/51dee771-9663-4366-90ca-9b1c286e35f8-os-release\") pod \"multus-additional-cni-plugins-prfcd\" (UID: \"51dee771-9663-4366-90ca-9b1c286e35f8\") " pod="openshift-multus/multus-additional-cni-plugins-prfcd" Jul 12 19:22:17.817829 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.800366 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8e016c2e-0c3a-49da-83d4-f526738c1cbd-etc-openvswitch\") pod \"ovnkube-node-k5h9m\" (UID: \"8e016c2e-0c3a-49da-83d4-f526738c1cbd\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5h9m" Jul 12 19:22:17.817829 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.800449 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/8e016c2e-0c3a-49da-83d4-f526738c1cbd-ovnkube-script-lib\") pod \"ovnkube-node-k5h9m\" (UID: \"8e016c2e-0c3a-49da-83d4-f526738c1cbd\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5h9m" Jul 12 19:22:17.818627 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.800048 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/56c9f12c-6e22-414d-a8d7-b503ed1f38df-kubelet-dir\") pod \"aws-ebs-csi-driver-node-bwmdr\" (UID: \"56c9f12c-6e22-414d-a8d7-b503ed1f38df\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bwmdr" Jul 12 19:22:17.818627 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.800643 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/23c32389-3933-4c29-a625-3e46ade08326-host\") pod \"tuned-vsllh\" (UID: \"23c32389-3933-4c29-a625-3e46ade08326\") " pod="openshift-cluster-node-tuning-operator/tuned-vsllh" Jul 12 19:22:17.818627 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.812708 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b8cca381-48d7-43ac-ae6a-15f96d99345d-serviceca\") pod \"node-ca-c89jb\" (UID: \"b8cca381-48d7-43ac-ae6a-15f96d99345d\") " pod="openshift-image-registry/node-ca-c89jb" Jul 12 19:22:17.818627 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.812850 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/51dee771-9663-4366-90ca-9b1c286e35f8-cni-binary-copy\") pod \"multus-additional-cni-plugins-prfcd\" (UID: \"51dee771-9663-4366-90ca-9b1c286e35f8\") " pod="openshift-multus/multus-additional-cni-plugins-prfcd" Jul 12 19:22:17.818627 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.813039 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/23c32389-3933-4c29-a625-3e46ade08326-etc-tuned\") pod \"tuned-vsllh\" (UID: \"23c32389-3933-4c29-a625-3e46ade08326\") " pod="openshift-cluster-node-tuning-operator/tuned-vsllh" Jul 12 19:22:17.818627 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.813130 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/51dee771-9663-4366-90ca-9b1c286e35f8-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-prfcd\" (UID: \"51dee771-9663-4366-90ca-9b1c286e35f8\") " pod="openshift-multus/multus-additional-cni-plugins-prfcd" Jul 12 19:22:17.818627 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.813244 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/23c32389-3933-4c29-a625-3e46ade08326-tmp\") pod \"tuned-vsllh\" (UID: \"23c32389-3933-4c29-a625-3e46ade08326\") " pod="openshift-cluster-node-tuning-operator/tuned-vsllh" Jul 12 19:22:17.818627 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.815251 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/8e016c2e-0c3a-49da-83d4-f526738c1cbd-ovn-node-metrics-cert\") pod \"ovnkube-node-k5h9m\" (UID: \"8e016c2e-0c3a-49da-83d4-f526738c1cbd\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5h9m" Jul 12 19:22:17.818627 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.817102 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5h9cz\" (UniqueName: \"kubernetes.io/projected/23c32389-3933-4c29-a625-3e46ade08326-kube-api-access-5h9cz\") pod \"tuned-vsllh\" (UID: \"23c32389-3933-4c29-a625-3e46ade08326\") " pod="openshift-cluster-node-tuning-operator/tuned-vsllh" Jul 12 19:22:17.988055 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.820211 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-csp9b\" (UniqueName: \"kubernetes.io/projected/8e016c2e-0c3a-49da-83d4-f526738c1cbd-kube-api-access-csp9b\") pod \"ovnkube-node-k5h9m\" (UID: \"8e016c2e-0c3a-49da-83d4-f526738c1cbd\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5h9m" Jul 12 19:22:17.988055 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.820664 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbxs5\" (UniqueName: \"kubernetes.io/projected/087e3f6e-992f-4784-a376-bd2af99f9851-kube-api-access-lbxs5\") pod \"iptables-alerter-hrrhk\" (UID: \"087e3f6e-992f-4784-a376-bd2af99f9851\") " pod="openshift-network-operator/iptables-alerter-hrrhk" Jul 12 19:22:17.988055 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.820798 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-czqm5\" (UniqueName: \"kubernetes.io/projected/b8cca381-48d7-43ac-ae6a-15f96d99345d-kube-api-access-czqm5\") pod \"node-ca-c89jb\" (UID: \"b8cca381-48d7-43ac-ae6a-15f96d99345d\") " pod="openshift-image-registry/node-ca-c89jb" Jul 12 19:22:17.988055 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.821699 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nj2g8\" (UniqueName: \"kubernetes.io/projected/e7722ae2-e293-478e-9ac8-ea376f8d583f-kube-api-access-nj2g8\") pod \"network-metrics-daemon-2xmfq\" (UID: \"e7722ae2-e293-478e-9ac8-ea376f8d583f\") " pod="openshift-multus/network-metrics-daemon-2xmfq" Jul 12 19:22:17.988055 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.821963 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wftsz\" (UniqueName: \"kubernetes.io/projected/56c9f12c-6e22-414d-a8d7-b503ed1f38df-kube-api-access-wftsz\") pod \"aws-ebs-csi-driver-node-bwmdr\" (UID: \"56c9f12c-6e22-414d-a8d7-b503ed1f38df\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bwmdr" Jul 12 19:22:17.988055 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.822591 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxwbs\" (UniqueName: \"kubernetes.io/projected/51dee771-9663-4366-90ca-9b1c286e35f8-kube-api-access-rxwbs\") pod \"multus-additional-cni-plugins-prfcd\" (UID: \"51dee771-9663-4366-90ca-9b1c286e35f8\") " pod="openshift-multus/multus-additional-cni-plugins-prfcd" Jul 12 19:22:17.988055 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.861558 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-170.ec2.internal" event={"ID":"2a4584a3f0966be0c3f50f97c2a5e31a","Type":"ContainerStarted","Data":"5268b43e5f11755e1f696a7ca89028441497336049f4d47bd07a61d5e9beb792"} Jul 12 19:22:17.988055 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.862647 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-170.ec2.internal" event={"ID":"83a48f1c2ae61195e40b72789388b7f8","Type":"ContainerStarted","Data":"47ba7d14b0c53865b1ee26bd94d25a5c5589ab7b75a2ed97ea256e7fbf78bd5e"} Jul 12 19:22:17.988055 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.899030 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/26ba017f-506c-4e00-9b24-4932c97ad489-agent-certs\") pod \"konnectivity-agent-7vvbp\" (UID: \"26ba017f-506c-4e00-9b24-4932c97ad489\") " pod="kube-system/konnectivity-agent-7vvbp" Jul 12 19:22:17.988055 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.899074 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bdb521c9-c545-445b-aee4-872355e9eb78-multus-cni-dir\") pod \"multus-9klzd\" (UID: \"bdb521c9-c545-445b-aee4-872355e9eb78\") " pod="openshift-multus/multus-9klzd" Jul 12 19:22:17.988055 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.899206 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e832009-b138-4442-b9db-10e73957481e-tmp-dir\") pod \"node-resolver-zs4lw\" (UID: \"8e832009-b138-4442-b9db-10e73957481e\") " pod="openshift-dns/node-resolver-zs4lw" Jul 12 19:22:17.988055 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.899245 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bdb521c9-c545-445b-aee4-872355e9eb78-host-run-netns\") pod \"multus-9klzd\" (UID: \"bdb521c9-c545-445b-aee4-872355e9eb78\") " pod="openshift-multus/multus-9klzd" Jul 12 19:22:17.988055 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.899250 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bdb521c9-c545-445b-aee4-872355e9eb78-multus-cni-dir\") pod \"multus-9klzd\" (UID: \"bdb521c9-c545-445b-aee4-872355e9eb78\") " pod="openshift-multus/multus-9klzd" Jul 12 19:22:17.988055 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.899270 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/26ba017f-506c-4e00-9b24-4932c97ad489-konnectivity-ca\") pod \"konnectivity-agent-7vvbp\" (UID: \"26ba017f-506c-4e00-9b24-4932c97ad489\") " pod="kube-system/konnectivity-agent-7vvbp" Jul 12 19:22:17.988055 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.899293 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/bdb521c9-c545-445b-aee4-872355e9eb78-hostroot\") pod \"multus-9klzd\" (UID: \"bdb521c9-c545-445b-aee4-872355e9eb78\") " pod="openshift-multus/multus-9klzd" Jul 12 19:22:17.990596 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.899305 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bdb521c9-c545-445b-aee4-872355e9eb78-host-run-netns\") pod \"multus-9klzd\" (UID: \"bdb521c9-c545-445b-aee4-872355e9eb78\") " pod="openshift-multus/multus-9klzd" Jul 12 19:22:17.990596 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.899316 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58dpr\" (UniqueName: \"kubernetes.io/projected/bdb521c9-c545-445b-aee4-872355e9eb78-kube-api-access-58dpr\") pod \"multus-9klzd\" (UID: \"bdb521c9-c545-445b-aee4-872355e9eb78\") " pod="openshift-multus/multus-9klzd" Jul 12 19:22:17.990596 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.899342 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a7a0abe6-09ba-429e-a7f2-8de5b3f5d620-kubelet-config\") pod \"global-pull-secret-syncer-288bq\" (UID: \"a7a0abe6-09ba-429e-a7f2-8de5b3f5d620\") " pod="kube-system/global-pull-secret-syncer-288bq" Jul 12 19:22:17.990596 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.899368 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bdb521c9-c545-445b-aee4-872355e9eb78-cnibin\") pod \"multus-9klzd\" (UID: \"bdb521c9-c545-445b-aee4-872355e9eb78\") " pod="openshift-multus/multus-9klzd" Jul 12 19:22:17.990596 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.899391 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/bdb521c9-c545-445b-aee4-872355e9eb78-host-run-k8s-cni-cncf-io\") pod \"multus-9klzd\" (UID: \"bdb521c9-c545-445b-aee4-872355e9eb78\") " pod="openshift-multus/multus-9klzd" Jul 12 19:22:17.990596 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.899405 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/bdb521c9-c545-445b-aee4-872355e9eb78-hostroot\") pod \"multus-9klzd\" (UID: \"bdb521c9-c545-445b-aee4-872355e9eb78\") " pod="openshift-multus/multus-9klzd" Jul 12 19:22:17.990596 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.899415 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a7a0abe6-09ba-429e-a7f2-8de5b3f5d620-original-pull-secret\") pod \"global-pull-secret-syncer-288bq\" (UID: \"a7a0abe6-09ba-429e-a7f2-8de5b3f5d620\") " pod="kube-system/global-pull-secret-syncer-288bq" Jul 12 19:22:17.990596 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.899441 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a7a0abe6-09ba-429e-a7f2-8de5b3f5d620-kubelet-config\") pod \"global-pull-secret-syncer-288bq\" (UID: \"a7a0abe6-09ba-429e-a7f2-8de5b3f5d620\") " pod="kube-system/global-pull-secret-syncer-288bq" Jul 12 19:22:17.990596 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.899457 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/bdb521c9-c545-445b-aee4-872355e9eb78-host-run-k8s-cni-cncf-io\") pod \"multus-9klzd\" (UID: \"bdb521c9-c545-445b-aee4-872355e9eb78\") " pod="openshift-multus/multus-9klzd" Jul 12 19:22:17.990596 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.899465 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bdb521c9-c545-445b-aee4-872355e9eb78-cnibin\") pod \"multus-9klzd\" (UID: \"bdb521c9-c545-445b-aee4-872355e9eb78\") " pod="openshift-multus/multus-9klzd" Jul 12 19:22:17.990596 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.899516 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bdb521c9-c545-445b-aee4-872355e9eb78-system-cni-dir\") pod \"multus-9klzd\" (UID: \"bdb521c9-c545-445b-aee4-872355e9eb78\") " pod="openshift-multus/multus-9klzd" Jul 12 19:22:17.990596 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.899538 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e832009-b138-4442-b9db-10e73957481e-tmp-dir\") pod \"node-resolver-zs4lw\" (UID: \"8e832009-b138-4442-b9db-10e73957481e\") " pod="openshift-dns/node-resolver-zs4lw" Jul 12 19:22:17.990596 ip-10-0-139-170 kubenswrapper[2565]: E0712 19:22:17.899522 2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 12 19:22:17.990596 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.899569 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bp96h\" (UniqueName: \"kubernetes.io/projected/10c67d44-bdd0-4530-9f5d-fa10238ee66e-kube-api-access-bp96h\") pod \"network-check-target-76gfm\" (UID: \"10c67d44-bdd0-4530-9f5d-fa10238ee66e\") " pod="openshift-network-diagnostics/network-check-target-76gfm" Jul 12 19:22:17.990596 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.899577 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bdb521c9-c545-445b-aee4-872355e9eb78-system-cni-dir\") pod \"multus-9klzd\" (UID: \"bdb521c9-c545-445b-aee4-872355e9eb78\") " pod="openshift-multus/multus-9klzd" Jul 12 19:22:17.990596 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.899593 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/bdb521c9-c545-445b-aee4-872355e9eb78-os-release\") pod \"multus-9klzd\" (UID: \"bdb521c9-c545-445b-aee4-872355e9eb78\") " pod="openshift-multus/multus-9klzd" Jul 12 19:22:17.991231 ip-10-0-139-170 kubenswrapper[2565]: E0712 19:22:17.899630 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a7a0abe6-09ba-429e-a7f2-8de5b3f5d620-original-pull-secret podName:a7a0abe6-09ba-429e-a7f2-8de5b3f5d620 nodeName:}" failed. No retries permitted until 2026-07-12 19:22:18.399611429 +0000 UTC m=+3.132183909 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a7a0abe6-09ba-429e-a7f2-8de5b3f5d620-original-pull-secret") pod "global-pull-secret-syncer-288bq" (UID: "a7a0abe6-09ba-429e-a7f2-8de5b3f5d620") : object "kube-system"/"original-pull-secret" not registered Jul 12 19:22:17.991231 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.899649 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/bdb521c9-c545-445b-aee4-872355e9eb78-multus-socket-dir-parent\") pod \"multus-9klzd\" (UID: \"bdb521c9-c545-445b-aee4-872355e9eb78\") " pod="openshift-multus/multus-9klzd" Jul 12 19:22:17.991231 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.899662 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/bdb521c9-c545-445b-aee4-872355e9eb78-os-release\") pod \"multus-9klzd\" (UID: \"bdb521c9-c545-445b-aee4-872355e9eb78\") " pod="openshift-multus/multus-9klzd" Jul 12 19:22:17.991231 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.899678 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/bdb521c9-c545-445b-aee4-872355e9eb78-multus-daemon-config\") pod \"multus-9klzd\" (UID: \"bdb521c9-c545-445b-aee4-872355e9eb78\") " pod="openshift-multus/multus-9klzd" Jul 12 19:22:17.991231 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.899708 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bdb521c9-c545-445b-aee4-872355e9eb78-cni-binary-copy\") pod \"multus-9klzd\" (UID: \"bdb521c9-c545-445b-aee4-872355e9eb78\") " pod="openshift-multus/multus-9klzd" Jul 12 19:22:17.991231 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.899714 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/bdb521c9-c545-445b-aee4-872355e9eb78-multus-socket-dir-parent\") pod \"multus-9klzd\" (UID: \"bdb521c9-c545-445b-aee4-872355e9eb78\") " pod="openshift-multus/multus-9klzd" Jul 12 19:22:17.991231 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.899735 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/bdb521c9-c545-445b-aee4-872355e9eb78-host-var-lib-cni-multus\") pod \"multus-9klzd\" (UID: \"bdb521c9-c545-445b-aee4-872355e9eb78\") " pod="openshift-multus/multus-9klzd" Jul 12 19:22:17.991231 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.899762 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a7a0abe6-09ba-429e-a7f2-8de5b3f5d620-dbus\") pod \"global-pull-secret-syncer-288bq\" (UID: \"a7a0abe6-09ba-429e-a7f2-8de5b3f5d620\") " pod="kube-system/global-pull-secret-syncer-288bq" Jul 12 19:22:17.991231 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.899782 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/bdb521c9-c545-445b-aee4-872355e9eb78-host-run-multus-certs\") pod \"multus-9klzd\" (UID: \"bdb521c9-c545-445b-aee4-872355e9eb78\") " pod="openshift-multus/multus-9klzd" Jul 12 19:22:17.991231 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.899797 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bdb521c9-c545-445b-aee4-872355e9eb78-host-var-lib-kubelet\") pod \"multus-9klzd\" (UID: \"bdb521c9-c545-445b-aee4-872355e9eb78\") " pod="openshift-multus/multus-9klzd" Jul 12 19:22:17.991231 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.899816 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bdb521c9-c545-445b-aee4-872355e9eb78-multus-conf-dir\") pod \"multus-9klzd\" (UID: \"bdb521c9-c545-445b-aee4-872355e9eb78\") " pod="openshift-multus/multus-9klzd" Jul 12 19:22:17.991231 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.899837 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a7a0abe6-09ba-429e-a7f2-8de5b3f5d620-dbus\") pod \"global-pull-secret-syncer-288bq\" (UID: \"a7a0abe6-09ba-429e-a7f2-8de5b3f5d620\") " pod="kube-system/global-pull-secret-syncer-288bq" Jul 12 19:22:17.991231 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.899840 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/bdb521c9-c545-445b-aee4-872355e9eb78-host-run-multus-certs\") pod \"multus-9klzd\" (UID: \"bdb521c9-c545-445b-aee4-872355e9eb78\") " pod="openshift-multus/multus-9klzd" Jul 12 19:22:17.991231 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.899841 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bdb521c9-c545-445b-aee4-872355e9eb78-host-var-lib-cni-bin\") pod \"multus-9klzd\" (UID: \"bdb521c9-c545-445b-aee4-872355e9eb78\") " pod="openshift-multus/multus-9klzd" Jul 12 19:22:17.992702 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.899839 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/26ba017f-506c-4e00-9b24-4932c97ad489-konnectivity-ca\") pod \"konnectivity-agent-7vvbp\" (UID: \"26ba017f-506c-4e00-9b24-4932c97ad489\") " pod="kube-system/konnectivity-agent-7vvbp" Jul 12 19:22:17.992702 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.899875 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bdb521c9-c545-445b-aee4-872355e9eb78-host-var-lib-cni-bin\") pod \"multus-9klzd\" (UID: \"bdb521c9-c545-445b-aee4-872355e9eb78\") " pod="openshift-multus/multus-9klzd" Jul 12 19:22:17.992702 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.899900 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bdb521c9-c545-445b-aee4-872355e9eb78-etc-kubernetes\") pod \"multus-9klzd\" (UID: \"bdb521c9-c545-445b-aee4-872355e9eb78\") " pod="openshift-multus/multus-9klzd" Jul 12 19:22:17.992702 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.899910 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bdb521c9-c545-445b-aee4-872355e9eb78-host-var-lib-kubelet\") pod \"multus-9klzd\" (UID: \"bdb521c9-c545-445b-aee4-872355e9eb78\") " pod="openshift-multus/multus-9klzd" Jul 12 19:22:17.992702 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.899926 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8e832009-b138-4442-b9db-10e73957481e-hosts-file\") pod \"node-resolver-zs4lw\" (UID: \"8e832009-b138-4442-b9db-10e73957481e\") " pod="openshift-dns/node-resolver-zs4lw" Jul 12 19:22:17.992702 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.899931 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/bdb521c9-c545-445b-aee4-872355e9eb78-host-var-lib-cni-multus\") pod \"multus-9klzd\" (UID: \"bdb521c9-c545-445b-aee4-872355e9eb78\") " pod="openshift-multus/multus-9klzd" Jul 12 19:22:17.992702 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.899949 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bdb521c9-c545-445b-aee4-872355e9eb78-multus-conf-dir\") pod \"multus-9klzd\" (UID: \"bdb521c9-c545-445b-aee4-872355e9eb78\") " pod="openshift-multus/multus-9klzd" Jul 12 19:22:17.992702 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.899958 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2hmn\" (UniqueName: \"kubernetes.io/projected/8e832009-b138-4442-b9db-10e73957481e-kube-api-access-h2hmn\") pod \"node-resolver-zs4lw\" (UID: \"8e832009-b138-4442-b9db-10e73957481e\") " pod="openshift-dns/node-resolver-zs4lw" Jul 12 19:22:17.992702 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.899979 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8e832009-b138-4442-b9db-10e73957481e-hosts-file\") pod \"node-resolver-zs4lw\" (UID: \"8e832009-b138-4442-b9db-10e73957481e\") " pod="openshift-dns/node-resolver-zs4lw" Jul 12 19:22:17.992702 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.899985 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bdb521c9-c545-445b-aee4-872355e9eb78-etc-kubernetes\") pod \"multus-9klzd\" (UID: \"bdb521c9-c545-445b-aee4-872355e9eb78\") " pod="openshift-multus/multus-9klzd" Jul 12 19:22:17.992702 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.900280 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/bdb521c9-c545-445b-aee4-872355e9eb78-multus-daemon-config\") pod \"multus-9klzd\" (UID: \"bdb521c9-c545-445b-aee4-872355e9eb78\") " pod="openshift-multus/multus-9klzd" Jul 12 19:22:17.992702 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.900315 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bdb521c9-c545-445b-aee4-872355e9eb78-cni-binary-copy\") pod \"multus-9klzd\" (UID: \"bdb521c9-c545-445b-aee4-872355e9eb78\") " pod="openshift-multus/multus-9klzd" Jul 12 19:22:17.992702 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.901831 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/26ba017f-506c-4e00-9b24-4932c97ad489-agent-certs\") pod \"konnectivity-agent-7vvbp\" (UID: \"26ba017f-506c-4e00-9b24-4932c97ad489\") " pod="kube-system/konnectivity-agent-7vvbp" Jul 12 19:22:17.992702 ip-10-0-139-170 kubenswrapper[2565]: E0712 19:22:17.905299 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 12 19:22:17.992702 ip-10-0-139-170 kubenswrapper[2565]: E0712 19:22:17.905320 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 12 19:22:17.992702 ip-10-0-139-170 kubenswrapper[2565]: E0712 19:22:17.905340 2565 projected.go:196] Error preparing data for projected volume kube-api-access-bp96h for pod openshift-network-diagnostics/network-check-target-76gfm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 12 19:22:17.993756 ip-10-0-139-170 kubenswrapper[2565]: E0712 19:22:17.905428 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/10c67d44-bdd0-4530-9f5d-fa10238ee66e-kube-api-access-bp96h podName:10c67d44-bdd0-4530-9f5d-fa10238ee66e nodeName:}" failed. No retries permitted until 2026-07-12 19:22:18.405410535 +0000 UTC m=+3.137983022 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-bp96h" (UniqueName: "kubernetes.io/projected/10c67d44-bdd0-4530-9f5d-fa10238ee66e-kube-api-access-bp96h") pod "network-check-target-76gfm" (UID: "10c67d44-bdd0-4530-9f5d-fa10238ee66e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 12 19:22:17.993756 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.907459 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-58dpr\" (UniqueName: \"kubernetes.io/projected/bdb521c9-c545-445b-aee4-872355e9eb78-kube-api-access-58dpr\") pod \"multus-9klzd\" (UID: \"bdb521c9-c545-445b-aee4-872355e9eb78\") " pod="openshift-multus/multus-9klzd" Jul 12 19:22:17.993756 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.907531 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2hmn\" (UniqueName: \"kubernetes.io/projected/8e832009-b138-4442-b9db-10e73957481e-kube-api-access-h2hmn\") pod \"node-resolver-zs4lw\" (UID: \"8e832009-b138-4442-b9db-10e73957481e\") " pod="openshift-dns/node-resolver-zs4lw" Jul 12 19:22:17.993756 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:17.988183 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-prfcd" Jul 12 19:22:18.120181 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:18.002409 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-hrrhk" Jul 12 19:22:18.120181 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:18.010185 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bwmdr" Jul 12 19:22:18.120181 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:18.015834 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-vsllh" Jul 12 19:22:18.120181 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:18.024348 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-c89jb" Jul 12 19:22:18.120181 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:18.030970 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-k5h9m" Jul 12 19:22:18.120181 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:18.037278 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-7vvbp" Jul 12 19:22:18.120181 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:18.043876 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-zs4lw" Jul 12 19:22:18.120181 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:18.048546 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-9klzd" Jul 12 19:22:18.391377 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:18.302361 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e7722ae2-e293-478e-9ac8-ea376f8d583f-metrics-certs\") pod \"network-metrics-daemon-2xmfq\" (UID: \"e7722ae2-e293-478e-9ac8-ea376f8d583f\") " pod="openshift-multus/network-metrics-daemon-2xmfq" Jul 12 19:22:18.391377 ip-10-0-139-170 kubenswrapper[2565]: E0712 19:22:18.302495 2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 12 19:22:18.391377 ip-10-0-139-170 kubenswrapper[2565]: E0712 19:22:18.302557 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e7722ae2-e293-478e-9ac8-ea376f8d583f-metrics-certs podName:e7722ae2-e293-478e-9ac8-ea376f8d583f nodeName:}" failed. No retries permitted until 2026-07-12 19:22:19.302537569 +0000 UTC m=+4.035110037 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e7722ae2-e293-478e-9ac8-ea376f8d583f-metrics-certs") pod "network-metrics-daemon-2xmfq" (UID: "e7722ae2-e293-478e-9ac8-ea376f8d583f") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 12 19:22:18.516217 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:18.403284 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a7a0abe6-09ba-429e-a7f2-8de5b3f5d620-original-pull-secret\") pod \"global-pull-secret-syncer-288bq\" (UID: \"a7a0abe6-09ba-429e-a7f2-8de5b3f5d620\") " pod="kube-system/global-pull-secret-syncer-288bq" Jul 12 19:22:18.516217 ip-10-0-139-170 kubenswrapper[2565]: E0712 19:22:18.403451 2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 12 19:22:18.516217 ip-10-0-139-170 kubenswrapper[2565]: E0712 19:22:18.403531 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a7a0abe6-09ba-429e-a7f2-8de5b3f5d620-original-pull-secret podName:a7a0abe6-09ba-429e-a7f2-8de5b3f5d620 nodeName:}" failed. No retries permitted until 2026-07-12 19:22:19.403508253 +0000 UTC m=+4.136080739 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a7a0abe6-09ba-429e-a7f2-8de5b3f5d620-original-pull-secret") pod "global-pull-secret-syncer-288bq" (UID: "a7a0abe6-09ba-429e-a7f2-8de5b3f5d620") : object "kube-system"/"original-pull-secret" not registered Jul 12 19:22:18.516217 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:18.503745 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bp96h\" (UniqueName: \"kubernetes.io/projected/10c67d44-bdd0-4530-9f5d-fa10238ee66e-kube-api-access-bp96h\") pod \"network-check-target-76gfm\" (UID: \"10c67d44-bdd0-4530-9f5d-fa10238ee66e\") " pod="openshift-network-diagnostics/network-check-target-76gfm" Jul 12 19:22:18.516217 ip-10-0-139-170 kubenswrapper[2565]: E0712 19:22:18.503901 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 12 19:22:18.516217 ip-10-0-139-170 kubenswrapper[2565]: E0712 19:22:18.503920 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 12 19:22:18.516217 ip-10-0-139-170 kubenswrapper[2565]: E0712 19:22:18.503929 2565 projected.go:196] Error preparing data for projected volume kube-api-access-bp96h for pod openshift-network-diagnostics/network-check-target-76gfm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 12 19:22:18.516217 ip-10-0-139-170 kubenswrapper[2565]: E0712 19:22:18.503980 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/10c67d44-bdd0-4530-9f5d-fa10238ee66e-kube-api-access-bp96h podName:10c67d44-bdd0-4530-9f5d-fa10238ee66e nodeName:}" failed. No retries permitted until 2026-07-12 19:22:19.503964544 +0000 UTC m=+4.236537015 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-bp96h" (UniqueName: "kubernetes.io/projected/10c67d44-bdd0-4530-9f5d-fa10238ee66e-kube-api-access-bp96h") pod "network-check-target-76gfm" (UID: "10c67d44-bdd0-4530-9f5d-fa10238ee66e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 12 19:22:18.720289 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:18.720248 2565 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-11 19:17:17 +0000 UTC" deadline="2028-01-26 06:42:22.289389189 +0000 UTC" Jul 12 19:22:18.720289 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:18.720282 2565 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13499h20m3.569111818s" Jul 12 19:22:18.887922 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:18.887896 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23c32389_3933_4c29_a625_3e46ade08326.slice/crio-b6d250a52c889e1ef7e673e29a0c9840846698730478f5feea32da641b94db69 WatchSource:0}: Error finding container b6d250a52c889e1ef7e673e29a0c9840846698730478f5feea32da641b94db69: Status 404 returned error can't find the container with id b6d250a52c889e1ef7e673e29a0c9840846698730478f5feea32da641b94db69 Jul 12 19:22:18.888729 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:18.888704 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e016c2e_0c3a_49da_83d4_f526738c1cbd.slice/crio-d33e3c06e8177095daf411ca685cec17ef320a83656f65989139cfe109b3f95f WatchSource:0}: Error finding container d33e3c06e8177095daf411ca685cec17ef320a83656f65989139cfe109b3f95f: Status 404 returned error can't find the container with id d33e3c06e8177095daf411ca685cec17ef320a83656f65989139cfe109b3f95f Jul 12 19:22:18.890806 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:18.890599 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8cca381_48d7_43ac_ae6a_15f96d99345d.slice/crio-fa930fb6fedc64a2026ecccefa7035e55d35c8ae5a55fbc5d1104c5bc6590d4f WatchSource:0}: Error finding container fa930fb6fedc64a2026ecccefa7035e55d35c8ae5a55fbc5d1104c5bc6590d4f: Status 404 returned error can't find the container with id fa930fb6fedc64a2026ecccefa7035e55d35c8ae5a55fbc5d1104c5bc6590d4f Jul 12 19:22:18.893244 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:18.893017 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51dee771_9663_4366_90ca_9b1c286e35f8.slice/crio-5bbc8abcb0b3d88e778fac3adad7624b39c1e616120acce8e61960221c452a2e WatchSource:0}: Error finding container 5bbc8abcb0b3d88e778fac3adad7624b39c1e616120acce8e61960221c452a2e: Status 404 returned error can't find the container with id 5bbc8abcb0b3d88e778fac3adad7624b39c1e616120acce8e61960221c452a2e Jul 12 19:22:18.896225 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:18.896208 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod087e3f6e_992f_4784_a376_bd2af99f9851.slice/crio-9b68f7b6b9db6a0c4a0bbc31f25c163aaeea90ad18b9aeed4c2092853ebba655 WatchSource:0}: Error finding container 9b68f7b6b9db6a0c4a0bbc31f25c163aaeea90ad18b9aeed4c2092853ebba655: Status 404 returned error can't find the container with id 9b68f7b6b9db6a0c4a0bbc31f25c163aaeea90ad18b9aeed4c2092853ebba655 Jul 12 19:22:18.896766 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:22:18.896692 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26ba017f_506c_4e00_9b24_4932c97ad489.slice/crio-63d708193b53292231eae92fd0d79efbbc73f421b387c9a08bcd691d49156b2c WatchSource:0}: Error finding container 63d708193b53292231eae92fd0d79efbbc73f421b387c9a08bcd691d49156b2c: Status 404 returned error can't find the container with id 63d708193b53292231eae92fd0d79efbbc73f421b387c9a08bcd691d49156b2c Jul 12 19:22:19.309487 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:19.309222 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e7722ae2-e293-478e-9ac8-ea376f8d583f-metrics-certs\") pod \"network-metrics-daemon-2xmfq\" (UID: \"e7722ae2-e293-478e-9ac8-ea376f8d583f\") " pod="openshift-multus/network-metrics-daemon-2xmfq" Jul 12 19:22:19.309649 ip-10-0-139-170 kubenswrapper[2565]: E0712 19:22:19.309469 2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 12 19:22:19.309649 ip-10-0-139-170 kubenswrapper[2565]: E0712 19:22:19.309600 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e7722ae2-e293-478e-9ac8-ea376f8d583f-metrics-certs podName:e7722ae2-e293-478e-9ac8-ea376f8d583f nodeName:}" failed. No retries permitted until 2026-07-12 19:22:21.309580522 +0000 UTC m=+6.042152991 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e7722ae2-e293-478e-9ac8-ea376f8d583f-metrics-certs") pod "network-metrics-daemon-2xmfq" (UID: "e7722ae2-e293-478e-9ac8-ea376f8d583f") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 12 19:22:19.410733 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:19.410045 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a7a0abe6-09ba-429e-a7f2-8de5b3f5d620-original-pull-secret\") pod \"global-pull-secret-syncer-288bq\" (UID: \"a7a0abe6-09ba-429e-a7f2-8de5b3f5d620\") " pod="kube-system/global-pull-secret-syncer-288bq" Jul 12 19:22:19.410733 ip-10-0-139-170 kubenswrapper[2565]: E0712 19:22:19.410235 2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 12 19:22:19.410733 ip-10-0-139-170 kubenswrapper[2565]: E0712 19:22:19.410295 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a7a0abe6-09ba-429e-a7f2-8de5b3f5d620-original-pull-secret podName:a7a0abe6-09ba-429e-a7f2-8de5b3f5d620 nodeName:}" failed. No retries permitted until 2026-07-12 19:22:21.41027714 +0000 UTC m=+6.142849610 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a7a0abe6-09ba-429e-a7f2-8de5b3f5d620-original-pull-secret") pod "global-pull-secret-syncer-288bq" (UID: "a7a0abe6-09ba-429e-a7f2-8de5b3f5d620") : object "kube-system"/"original-pull-secret" not registered Jul 12 19:22:19.511463 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:19.511425 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bp96h\" (UniqueName: \"kubernetes.io/projected/10c67d44-bdd0-4530-9f5d-fa10238ee66e-kube-api-access-bp96h\") pod \"network-check-target-76gfm\" (UID: \"10c67d44-bdd0-4530-9f5d-fa10238ee66e\") " pod="openshift-network-diagnostics/network-check-target-76gfm" Jul 12 19:22:19.511653 ip-10-0-139-170 kubenswrapper[2565]: E0712 19:22:19.511606 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 12 19:22:19.511653 ip-10-0-139-170 kubenswrapper[2565]: E0712 19:22:19.511630 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 12 19:22:19.511653 ip-10-0-139-170 kubenswrapper[2565]: E0712 19:22:19.511642 2565 projected.go:196] Error preparing data for projected volume kube-api-access-bp96h for pod openshift-network-diagnostics/network-check-target-76gfm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 12 19:22:19.511825 ip-10-0-139-170 kubenswrapper[2565]: E0712 19:22:19.511710 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/10c67d44-bdd0-4530-9f5d-fa10238ee66e-kube-api-access-bp96h podName:10c67d44-bdd0-4530-9f5d-fa10238ee66e nodeName:}" failed. No retries permitted until 2026-07-12 19:22:21.511692567 +0000 UTC m=+6.244265038 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-bp96h" (UniqueName: "kubernetes.io/projected/10c67d44-bdd0-4530-9f5d-fa10238ee66e-kube-api-access-bp96h") pod "network-check-target-76gfm" (UID: "10c67d44-bdd0-4530-9f5d-fa10238ee66e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 12 19:22:19.720559 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:19.720428 2565 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-11 19:17:17 +0000 UTC" deadline="2028-02-13 21:39:44.459979223 +0000 UTC" Jul 12 19:22:19.720559 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:19.720496 2565 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13946h17m24.739487697s" Jul 12 19:22:19.853044 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:19.853016 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-76gfm" Jul 12 19:22:19.853208 ip-10-0-139-170 kubenswrapper[2565]: E0712 19:22:19.853141 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-76gfm" podUID="10c67d44-bdd0-4530-9f5d-fa10238ee66e" Jul 12 19:22:19.854086 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:19.853535 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-288bq" Jul 12 19:22:19.854086 ip-10-0-139-170 kubenswrapper[2565]: E0712 19:22:19.853621 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-288bq" podUID="a7a0abe6-09ba-429e-a7f2-8de5b3f5d620" Jul 12 19:22:19.854086 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:19.853905 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2xmfq" Jul 12 19:22:19.854086 ip-10-0-139-170 kubenswrapper[2565]: E0712 19:22:19.853999 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2xmfq" podUID="e7722ae2-e293-478e-9ac8-ea376f8d583f" Jul 12 19:22:19.884771 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:19.884432 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-prfcd" event={"ID":"51dee771-9663-4366-90ca-9b1c286e35f8","Type":"ContainerStarted","Data":"5bbc8abcb0b3d88e778fac3adad7624b39c1e616120acce8e61960221c452a2e"} Jul 12 19:22:19.892190 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:19.891971 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k5h9m" event={"ID":"8e016c2e-0c3a-49da-83d4-f526738c1cbd","Type":"ContainerStarted","Data":"d33e3c06e8177095daf411ca685cec17ef320a83656f65989139cfe109b3f95f"} Jul 12 19:22:19.894567 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:19.894533 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-vsllh" event={"ID":"23c32389-3933-4c29-a625-3e46ade08326","Type":"ContainerStarted","Data":"b6d250a52c889e1ef7e673e29a0c9840846698730478f5feea32da641b94db69"} Jul 12 19:22:19.905852 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:19.905366 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-170.ec2.internal" event={"ID":"2a4584a3f0966be0c3f50f97c2a5e31a","Type":"ContainerStarted","Data":"a61ef609efcff097efe6157a34b55978343e6d89fc150319a26a69823110a354"} Jul 12 19:22:19.932329 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:19.932267 2565 generic.go:356] "Generic (PLEG): container finished" podID="83a48f1c2ae61195e40b72789388b7f8" containerID="84d612890ded98b79f17fab8b411f0acad94c56414a1afc99d02e13f3bf3a7b8" exitCode=0 Jul 12 19:22:19.932491 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:19.932362 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-170.ec2.internal" event={"ID":"83a48f1c2ae61195e40b72789388b7f8","Type":"ContainerDied","Data":"84d612890ded98b79f17fab8b411f0acad94c56414a1afc99d02e13f3bf3a7b8"} Jul 12 19:22:19.942866 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:19.942829 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-7vvbp" event={"ID":"26ba017f-506c-4e00-9b24-4932c97ad489","Type":"ContainerStarted","Data":"63d708193b53292231eae92fd0d79efbbc73f421b387c9a08bcd691d49156b2c"} Jul 12 19:22:19.945900 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:19.945833 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-170.ec2.internal" podStartSLOduration=3.9458130049999998 podStartE2EDuration="3.945813005s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-12 19:22:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 19:22:19.921200774 +0000 UTC m=+4.653773266" watchObservedRunningTime="2026-07-12 19:22:19.945813005 +0000 UTC m=+4.678385496" Jul 12 19:22:19.964631 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:19.964595 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-c89jb" event={"ID":"b8cca381-48d7-43ac-ae6a-15f96d99345d","Type":"ContainerStarted","Data":"fa930fb6fedc64a2026ecccefa7035e55d35c8ae5a55fbc5d1104c5bc6590d4f"} Jul 12 19:22:19.978062 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:19.977996 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-9klzd" event={"ID":"bdb521c9-c545-445b-aee4-872355e9eb78","Type":"ContainerStarted","Data":"76b6977651b09d427f0e3bf27cde15b22496b7342018b6e59a93285744799df8"} Jul 12 19:22:19.987973 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:19.987937 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-zs4lw" event={"ID":"8e832009-b138-4442-b9db-10e73957481e","Type":"ContainerStarted","Data":"3f86be18937c895761ba7ba96fbaf05dbe4b41177a7d63b150e70a770602894f"} Jul 12 19:22:20.010249 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:20.010211 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bwmdr" event={"ID":"56c9f12c-6e22-414d-a8d7-b503ed1f38df","Type":"ContainerStarted","Data":"dbf1648c5a490b40e5400542c268a8bb024f3648c63a1f4b4c3df5df80ef0e15"} Jul 12 19:22:20.023430 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:20.023357 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-hrrhk" event={"ID":"087e3f6e-992f-4784-a376-bd2af99f9851","Type":"ContainerStarted","Data":"9b68f7b6b9db6a0c4a0bbc31f25c163aaeea90ad18b9aeed4c2092853ebba655"} Jul 12 19:22:21.047283 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:21.047205 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-170.ec2.internal" event={"ID":"83a48f1c2ae61195e40b72789388b7f8","Type":"ContainerStarted","Data":"a6548f4c96bdc77ca6724a39ea87518718a35a0f604db91e335025dceed91063"} Jul 12 19:22:21.327324 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:21.327232 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e7722ae2-e293-478e-9ac8-ea376f8d583f-metrics-certs\") pod \"network-metrics-daemon-2xmfq\" (UID: \"e7722ae2-e293-478e-9ac8-ea376f8d583f\") " pod="openshift-multus/network-metrics-daemon-2xmfq" Jul 12 19:22:21.327535 ip-10-0-139-170 kubenswrapper[2565]: E0712 19:22:21.327396 2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 12 19:22:21.327535 ip-10-0-139-170 kubenswrapper[2565]: E0712 19:22:21.327468 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e7722ae2-e293-478e-9ac8-ea376f8d583f-metrics-certs podName:e7722ae2-e293-478e-9ac8-ea376f8d583f nodeName:}" failed. No retries permitted until 2026-07-12 19:22:25.32745008 +0000 UTC m=+10.060022551 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e7722ae2-e293-478e-9ac8-ea376f8d583f-metrics-certs") pod "network-metrics-daemon-2xmfq" (UID: "e7722ae2-e293-478e-9ac8-ea376f8d583f") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 12 19:22:21.427873 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:21.427832 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a7a0abe6-09ba-429e-a7f2-8de5b3f5d620-original-pull-secret\") pod \"global-pull-secret-syncer-288bq\" (UID: \"a7a0abe6-09ba-429e-a7f2-8de5b3f5d620\") " pod="kube-system/global-pull-secret-syncer-288bq" Jul 12 19:22:21.428093 ip-10-0-139-170 kubenswrapper[2565]: E0712 19:22:21.428065 2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 12 19:22:21.428155 ip-10-0-139-170 kubenswrapper[2565]: E0712 19:22:21.428131 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a7a0abe6-09ba-429e-a7f2-8de5b3f5d620-original-pull-secret podName:a7a0abe6-09ba-429e-a7f2-8de5b3f5d620 nodeName:}" failed. No retries permitted until 2026-07-12 19:22:25.42811337 +0000 UTC m=+10.160685841 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a7a0abe6-09ba-429e-a7f2-8de5b3f5d620-original-pull-secret") pod "global-pull-secret-syncer-288bq" (UID: "a7a0abe6-09ba-429e-a7f2-8de5b3f5d620") : object "kube-system"/"original-pull-secret" not registered Jul 12 19:22:21.528394 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:21.528361 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bp96h\" (UniqueName: \"kubernetes.io/projected/10c67d44-bdd0-4530-9f5d-fa10238ee66e-kube-api-access-bp96h\") pod \"network-check-target-76gfm\" (UID: \"10c67d44-bdd0-4530-9f5d-fa10238ee66e\") " pod="openshift-network-diagnostics/network-check-target-76gfm" Jul 12 19:22:21.528558 ip-10-0-139-170 kubenswrapper[2565]: E0712 19:22:21.528513 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 12 19:22:21.528558 ip-10-0-139-170 kubenswrapper[2565]: E0712 19:22:21.528530 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 12 19:22:21.528558 ip-10-0-139-170 kubenswrapper[2565]: E0712 19:22:21.528541 2565 projected.go:196] Error preparing data for projected volume kube-api-access-bp96h for pod openshift-network-diagnostics/network-check-target-76gfm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 12 19:22:21.528718 ip-10-0-139-170 kubenswrapper[2565]: E0712 19:22:21.528604 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/10c67d44-bdd0-4530-9f5d-fa10238ee66e-kube-api-access-bp96h podName:10c67d44-bdd0-4530-9f5d-fa10238ee66e nodeName:}" failed. No retries permitted until 2026-07-12 19:22:25.528585878 +0000 UTC m=+10.261158355 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-bp96h" (UniqueName: "kubernetes.io/projected/10c67d44-bdd0-4530-9f5d-fa10238ee66e-kube-api-access-bp96h") pod "network-check-target-76gfm" (UID: "10c67d44-bdd0-4530-9f5d-fa10238ee66e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 12 19:22:21.853213 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:21.852515 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-76gfm" Jul 12 19:22:21.853213 ip-10-0-139-170 kubenswrapper[2565]: E0712 19:22:21.852654 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-76gfm" podUID="10c67d44-bdd0-4530-9f5d-fa10238ee66e" Jul 12 19:22:21.853213 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:21.853079 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-288bq" Jul 12 19:22:21.853213 ip-10-0-139-170 kubenswrapper[2565]: E0712 19:22:21.853165 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-288bq" podUID="a7a0abe6-09ba-429e-a7f2-8de5b3f5d620" Jul 12 19:22:21.853766 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:21.853591 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2xmfq" Jul 12 19:22:21.853766 ip-10-0-139-170 kubenswrapper[2565]: E0712 19:22:21.853691 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2xmfq" podUID="e7722ae2-e293-478e-9ac8-ea376f8d583f" Jul 12 19:22:23.852742 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:23.852706 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-76gfm" Jul 12 19:22:23.852742 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:23.852734 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-288bq" Jul 12 19:22:23.853314 ip-10-0-139-170 kubenswrapper[2565]: E0712 19:22:23.852833 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-76gfm" podUID="10c67d44-bdd0-4530-9f5d-fa10238ee66e" Jul 12 19:22:23.853314 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:23.852706 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2xmfq" Jul 12 19:22:23.853314 ip-10-0-139-170 kubenswrapper[2565]: E0712 19:22:23.852991 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-288bq" podUID="a7a0abe6-09ba-429e-a7f2-8de5b3f5d620" Jul 12 19:22:23.853314 ip-10-0-139-170 kubenswrapper[2565]: E0712 19:22:23.853100 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2xmfq" podUID="e7722ae2-e293-478e-9ac8-ea376f8d583f" Jul 12 19:22:25.361817 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:25.361773 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e7722ae2-e293-478e-9ac8-ea376f8d583f-metrics-certs\") pod \"network-metrics-daemon-2xmfq\" (UID: \"e7722ae2-e293-478e-9ac8-ea376f8d583f\") " pod="openshift-multus/network-metrics-daemon-2xmfq" Jul 12 19:22:25.362306 ip-10-0-139-170 kubenswrapper[2565]: E0712 19:22:25.361947 2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 12 19:22:25.362306 ip-10-0-139-170 kubenswrapper[2565]: E0712 19:22:25.362014 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e7722ae2-e293-478e-9ac8-ea376f8d583f-metrics-certs podName:e7722ae2-e293-478e-9ac8-ea376f8d583f nodeName:}" failed. No retries permitted until 2026-07-12 19:22:33.361994821 +0000 UTC m=+18.094567291 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e7722ae2-e293-478e-9ac8-ea376f8d583f-metrics-certs") pod "network-metrics-daemon-2xmfq" (UID: "e7722ae2-e293-478e-9ac8-ea376f8d583f") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 12 19:22:25.462901 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:25.462853 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a7a0abe6-09ba-429e-a7f2-8de5b3f5d620-original-pull-secret\") pod \"global-pull-secret-syncer-288bq\" (UID: \"a7a0abe6-09ba-429e-a7f2-8de5b3f5d620\") " pod="kube-system/global-pull-secret-syncer-288bq" Jul 12 19:22:25.463143 ip-10-0-139-170 kubenswrapper[2565]: E0712 19:22:25.463021 2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 12 19:22:25.463143 ip-10-0-139-170 kubenswrapper[2565]: E0712 19:22:25.463105 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a7a0abe6-09ba-429e-a7f2-8de5b3f5d620-original-pull-secret podName:a7a0abe6-09ba-429e-a7f2-8de5b3f5d620 nodeName:}" failed. No retries permitted until 2026-07-12 19:22:33.463085613 +0000 UTC m=+18.195658087 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a7a0abe6-09ba-429e-a7f2-8de5b3f5d620-original-pull-secret") pod "global-pull-secret-syncer-288bq" (UID: "a7a0abe6-09ba-429e-a7f2-8de5b3f5d620") : object "kube-system"/"original-pull-secret" not registered Jul 12 19:22:25.563642 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:25.563551 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bp96h\" (UniqueName: \"kubernetes.io/projected/10c67d44-bdd0-4530-9f5d-fa10238ee66e-kube-api-access-bp96h\") pod \"network-check-target-76gfm\" (UID: \"10c67d44-bdd0-4530-9f5d-fa10238ee66e\") " pod="openshift-network-diagnostics/network-check-target-76gfm" Jul 12 19:22:25.563831 ip-10-0-139-170 kubenswrapper[2565]: E0712 19:22:25.563721 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 12 19:22:25.563831 ip-10-0-139-170 kubenswrapper[2565]: E0712 19:22:25.563745 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 12 19:22:25.563831 ip-10-0-139-170 kubenswrapper[2565]: E0712 19:22:25.563760 2565 projected.go:196] Error preparing data for projected volume kube-api-access-bp96h for pod openshift-network-diagnostics/network-check-target-76gfm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 12 19:22:25.563831 ip-10-0-139-170 kubenswrapper[2565]: E0712 19:22:25.563827 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/10c67d44-bdd0-4530-9f5d-fa10238ee66e-kube-api-access-bp96h podName:10c67d44-bdd0-4530-9f5d-fa10238ee66e nodeName:}" failed. No retries permitted until 2026-07-12 19:22:33.563807803 +0000 UTC m=+18.296380280 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-bp96h" (UniqueName: "kubernetes.io/projected/10c67d44-bdd0-4530-9f5d-fa10238ee66e-kube-api-access-bp96h") pod "network-check-target-76gfm" (UID: "10c67d44-bdd0-4530-9f5d-fa10238ee66e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 12 19:22:25.852979 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:25.852717 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-76gfm" Jul 12 19:22:25.852979 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:25.852749 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-288bq" Jul 12 19:22:25.852979 ip-10-0-139-170 kubenswrapper[2565]: E0712 19:22:25.852838 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-76gfm" podUID="10c67d44-bdd0-4530-9f5d-fa10238ee66e" Jul 12 19:22:25.853287 ip-10-0-139-170 kubenswrapper[2565]: E0712 19:22:25.853255 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-288bq" podUID="a7a0abe6-09ba-429e-a7f2-8de5b3f5d620" Jul 12 19:22:25.853416 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:25.853327 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2xmfq" Jul 12 19:22:25.853480 ip-10-0-139-170 kubenswrapper[2565]: E0712 19:22:25.853411 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2xmfq" podUID="e7722ae2-e293-478e-9ac8-ea376f8d583f" Jul 12 19:22:27.852446 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:27.852408 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2xmfq" Jul 12 19:22:27.852898 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:27.852408 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-288bq" Jul 12 19:22:27.852898 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:27.852409 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-76gfm" Jul 12 19:22:27.852898 ip-10-0-139-170 kubenswrapper[2565]: E0712 19:22:27.852568 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2xmfq" podUID="e7722ae2-e293-478e-9ac8-ea376f8d583f" Jul 12 19:22:27.852898 ip-10-0-139-170 kubenswrapper[2565]: E0712 19:22:27.852691 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-76gfm" podUID="10c67d44-bdd0-4530-9f5d-fa10238ee66e" Jul 12 19:22:27.852898 ip-10-0-139-170 kubenswrapper[2565]: E0712 19:22:27.852769 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-288bq" podUID="a7a0abe6-09ba-429e-a7f2-8de5b3f5d620" Jul 12 19:22:29.851976 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:29.851934 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2xmfq" Jul 12 19:22:29.851976 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:29.851976 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-288bq" Jul 12 19:22:29.852503 ip-10-0-139-170 kubenswrapper[2565]: E0712 19:22:29.852090 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2xmfq" podUID="e7722ae2-e293-478e-9ac8-ea376f8d583f" Jul 12 19:22:29.852503 ip-10-0-139-170 kubenswrapper[2565]: E0712 19:22:29.852178 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-288bq" podUID="a7a0abe6-09ba-429e-a7f2-8de5b3f5d620" Jul 12 19:22:29.852503 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:29.852203 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-76gfm" Jul 12 19:22:29.852503 ip-10-0-139-170 kubenswrapper[2565]: E0712 19:22:29.852302 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-76gfm" podUID="10c67d44-bdd0-4530-9f5d-fa10238ee66e" Jul 12 19:22:31.852296 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:31.852259 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-288bq" Jul 12 19:22:31.852296 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:31.852294 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-76gfm" Jul 12 19:22:31.852909 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:31.852262 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2xmfq" Jul 12 19:22:31.852909 ip-10-0-139-170 kubenswrapper[2565]: E0712 19:22:31.852401 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-288bq" podUID="a7a0abe6-09ba-429e-a7f2-8de5b3f5d620" Jul 12 19:22:31.852909 ip-10-0-139-170 kubenswrapper[2565]: E0712 19:22:31.852479 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2xmfq" podUID="e7722ae2-e293-478e-9ac8-ea376f8d583f" Jul 12 19:22:31.852909 ip-10-0-139-170 kubenswrapper[2565]: E0712 19:22:31.852572 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-76gfm" podUID="10c67d44-bdd0-4530-9f5d-fa10238ee66e" Jul 12 19:22:33.419815 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:33.419775 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e7722ae2-e293-478e-9ac8-ea376f8d583f-metrics-certs\") pod \"network-metrics-daemon-2xmfq\" (UID: \"e7722ae2-e293-478e-9ac8-ea376f8d583f\") " pod="openshift-multus/network-metrics-daemon-2xmfq" Jul 12 19:22:33.420349 ip-10-0-139-170 kubenswrapper[2565]: E0712 19:22:33.419954 2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 12 19:22:33.420349 ip-10-0-139-170 kubenswrapper[2565]: E0712 19:22:33.420027 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e7722ae2-e293-478e-9ac8-ea376f8d583f-metrics-certs podName:e7722ae2-e293-478e-9ac8-ea376f8d583f nodeName:}" failed. No retries permitted until 2026-07-12 19:22:49.420004385 +0000 UTC m=+34.152576853 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e7722ae2-e293-478e-9ac8-ea376f8d583f-metrics-certs") pod "network-metrics-daemon-2xmfq" (UID: "e7722ae2-e293-478e-9ac8-ea376f8d583f") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 12 19:22:33.521056 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:33.521022 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a7a0abe6-09ba-429e-a7f2-8de5b3f5d620-original-pull-secret\") pod \"global-pull-secret-syncer-288bq\" (UID: \"a7a0abe6-09ba-429e-a7f2-8de5b3f5d620\") " pod="kube-system/global-pull-secret-syncer-288bq" Jul 12 19:22:33.521244 ip-10-0-139-170 kubenswrapper[2565]: E0712 19:22:33.521184 2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 12 19:22:33.521304 ip-10-0-139-170 kubenswrapper[2565]: E0712 19:22:33.521258 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a7a0abe6-09ba-429e-a7f2-8de5b3f5d620-original-pull-secret podName:a7a0abe6-09ba-429e-a7f2-8de5b3f5d620 nodeName:}" failed. No retries permitted until 2026-07-12 19:22:49.521242274 +0000 UTC m=+34.253814742 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a7a0abe6-09ba-429e-a7f2-8de5b3f5d620-original-pull-secret") pod "global-pull-secret-syncer-288bq" (UID: "a7a0abe6-09ba-429e-a7f2-8de5b3f5d620") : object "kube-system"/"original-pull-secret" not registered Jul 12 19:22:33.622241 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:33.622196 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bp96h\" (UniqueName: \"kubernetes.io/projected/10c67d44-bdd0-4530-9f5d-fa10238ee66e-kube-api-access-bp96h\") pod \"network-check-target-76gfm\" (UID: \"10c67d44-bdd0-4530-9f5d-fa10238ee66e\") " pod="openshift-network-diagnostics/network-check-target-76gfm" Jul 12 19:22:33.622422 ip-10-0-139-170 kubenswrapper[2565]: E0712 19:22:33.622349 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 12 19:22:33.622422 ip-10-0-139-170 kubenswrapper[2565]: E0712 19:22:33.622377 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 12 19:22:33.622422 ip-10-0-139-170 kubenswrapper[2565]: E0712 19:22:33.622388 2565 projected.go:196] Error preparing data for projected volume kube-api-access-bp96h for pod openshift-network-diagnostics/network-check-target-76gfm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 12 19:22:33.622565 ip-10-0-139-170 kubenswrapper[2565]: E0712 19:22:33.622449 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/10c67d44-bdd0-4530-9f5d-fa10238ee66e-kube-api-access-bp96h podName:10c67d44-bdd0-4530-9f5d-fa10238ee66e nodeName:}" failed. No retries permitted until 2026-07-12 19:22:49.622429518 +0000 UTC m=+34.355002003 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-bp96h" (UniqueName: "kubernetes.io/projected/10c67d44-bdd0-4530-9f5d-fa10238ee66e-kube-api-access-bp96h") pod "network-check-target-76gfm" (UID: "10c67d44-bdd0-4530-9f5d-fa10238ee66e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 12 19:22:33.851969 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:33.851877 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-76gfm" Jul 12 19:22:33.852132 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:33.851877 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-288bq" Jul 12 19:22:33.852132 ip-10-0-139-170 kubenswrapper[2565]: E0712 19:22:33.852011 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-76gfm" podUID="10c67d44-bdd0-4530-9f5d-fa10238ee66e" Jul 12 19:22:33.852132 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:33.852035 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2xmfq" Jul 12 19:22:33.852287 ip-10-0-139-170 kubenswrapper[2565]: E0712 19:22:33.852136 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-288bq" podUID="a7a0abe6-09ba-429e-a7f2-8de5b3f5d620" Jul 12 19:22:33.852287 ip-10-0-139-170 kubenswrapper[2565]: E0712 19:22:33.852224 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2xmfq" podUID="e7722ae2-e293-478e-9ac8-ea376f8d583f" Jul 12 19:22:35.853502 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:35.853375 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-288bq" Jul 12 19:22:35.853502 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:35.853418 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2xmfq" Jul 12 19:22:35.853502 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:35.853475 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-76gfm" Jul 12 19:22:35.853502 ip-10-0-139-170 kubenswrapper[2565]: E0712 19:22:35.853498 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-288bq" podUID="a7a0abe6-09ba-429e-a7f2-8de5b3f5d620" Jul 12 19:22:35.854163 ip-10-0-139-170 kubenswrapper[2565]: E0712 19:22:35.853565 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2xmfq" podUID="e7722ae2-e293-478e-9ac8-ea376f8d583f" Jul 12 19:22:35.854163 ip-10-0-139-170 kubenswrapper[2565]: E0712 19:22:35.853630 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-76gfm" podUID="10c67d44-bdd0-4530-9f5d-fa10238ee66e" Jul 12 19:22:37.852346 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:37.852029 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2xmfq" Jul 12 19:22:37.853213 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:37.852030 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-76gfm" Jul 12 19:22:37.853213 ip-10-0-139-170 kubenswrapper[2565]: E0712 19:22:37.852434 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2xmfq" podUID="e7722ae2-e293-478e-9ac8-ea376f8d583f" Jul 12 19:22:37.853213 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:37.852030 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-288bq" Jul 12 19:22:37.853213 ip-10-0-139-170 kubenswrapper[2565]: E0712 19:22:37.852498 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-76gfm" podUID="10c67d44-bdd0-4530-9f5d-fa10238ee66e" Jul 12 19:22:37.853213 ip-10-0-139-170 kubenswrapper[2565]: E0712 19:22:37.852581 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-288bq" podUID="a7a0abe6-09ba-429e-a7f2-8de5b3f5d620" Jul 12 19:22:38.081732 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:38.081641 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-9klzd" event={"ID":"bdb521c9-c545-445b-aee4-872355e9eb78","Type":"ContainerStarted","Data":"5b174eb7d23189108025a5642779fe0940862b4ccd8a0f91066ea42274d8cb69"} Jul 12 19:22:38.083065 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:38.083039 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-zs4lw" event={"ID":"8e832009-b138-4442-b9db-10e73957481e","Type":"ContainerStarted","Data":"9cc168b0345d8960d4f82b349281286711007e9fd754725228ffffb36fa0d9bc"} Jul 12 19:22:38.084811 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:38.084590 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bwmdr" event={"ID":"56c9f12c-6e22-414d-a8d7-b503ed1f38df","Type":"ContainerStarted","Data":"63af17f6025058c4b24dc14d0e7e2e6ba9243094898d589b9019ad9b34cef110"} Jul 12 19:22:38.088862 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:38.088837 2565 generic.go:356] "Generic (PLEG): container finished" podID="51dee771-9663-4366-90ca-9b1c286e35f8" containerID="e303d7653ad22dd446089c3b40e6609c9d9d1cf5351e26c63f7e6a92a77c81d2" exitCode=0 Jul 12 19:22:38.089009 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:38.088930 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-prfcd" event={"ID":"51dee771-9663-4366-90ca-9b1c286e35f8","Type":"ContainerDied","Data":"e303d7653ad22dd446089c3b40e6609c9d9d1cf5351e26c63f7e6a92a77c81d2"} Jul 12 19:22:38.091165 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:38.091147 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k5h9m_8e016c2e-0c3a-49da-83d4-f526738c1cbd/ovn-acl-logging/0.log" Jul 12 19:22:38.091511 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:38.091486 2565 generic.go:356] "Generic (PLEG): container finished" podID="8e016c2e-0c3a-49da-83d4-f526738c1cbd" containerID="05f2c00691adb37392b6ecfcf4d5a31a8fed8e65e58f9a145ed612f19da7ec00" exitCode=1 Jul 12 19:22:38.091595 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:38.091551 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k5h9m" event={"ID":"8e016c2e-0c3a-49da-83d4-f526738c1cbd","Type":"ContainerStarted","Data":"9511f8adfabbba558e73f0d0d0e434dbb65c971c4fa01f7a61a7cd1837c92786"} Jul 12 19:22:38.091641 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:38.091612 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k5h9m" event={"ID":"8e016c2e-0c3a-49da-83d4-f526738c1cbd","Type":"ContainerStarted","Data":"d1d14dbb7841c9db9f69e568354b6b7b4ac4a8c79359d876b80a1bf365f6de43"} Jul 12 19:22:38.091688 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:38.091658 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k5h9m" event={"ID":"8e016c2e-0c3a-49da-83d4-f526738c1cbd","Type":"ContainerStarted","Data":"0b90fbf2d0d77b849c8acaa8041f5978b57ef02d2bb2605615866fcbdebed1e0"} Jul 12 19:22:38.091736 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:38.091712 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k5h9m" event={"ID":"8e016c2e-0c3a-49da-83d4-f526738c1cbd","Type":"ContainerDied","Data":"05f2c00691adb37392b6ecfcf4d5a31a8fed8e65e58f9a145ed612f19da7ec00"} Jul 12 19:22:38.091774 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:38.091744 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k5h9m" event={"ID":"8e016c2e-0c3a-49da-83d4-f526738c1cbd","Type":"ContainerStarted","Data":"8262545279a95f6d6c0ba4db28380244be97a6b520e6b97acaba5d83b5411a04"} Jul 12 19:22:38.094924 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:38.094877 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-vsllh" event={"ID":"23c32389-3933-4c29-a625-3e46ade08326","Type":"ContainerStarted","Data":"44d1010dfa68b4888c4d5ce9b63159dcd28a4d35fca9d702c45a9e869274ca05"} Jul 12 19:22:38.096265 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:38.096242 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-7vvbp" event={"ID":"26ba017f-506c-4e00-9b24-4932c97ad489","Type":"ContainerStarted","Data":"da7db76297fe665b5c521f78e45623554a0eb793347e34d200015f3fb7d0fbe8"} Jul 12 19:22:38.097594 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:38.097573 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-c89jb" event={"ID":"b8cca381-48d7-43ac-ae6a-15f96d99345d","Type":"ContainerStarted","Data":"7be17512ccbce375b84fb53cfb3701389bcd102b74c00a5db70eda69e0f968f2"} Jul 12 19:22:38.099681 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:38.099647 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-170.ec2.internal" podStartSLOduration=22.099635976 podStartE2EDuration="22.099635976s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-12 19:22:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 19:22:21.0609996 +0000 UTC m=+5.793572092" watchObservedRunningTime="2026-07-12 19:22:38.099635976 +0000 UTC m=+22.832208465" Jul 12 19:22:38.099855 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:38.099837 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-9klzd" podStartSLOduration=4.992597268 podStartE2EDuration="23.099833746s" totalImagesPullingTime="18.107236478s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-12 19:22:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 19:22:38.098834896 +0000 UTC m=+22.831407389" watchObservedRunningTime="2026-07-12 19:22:38.099833746 +0000 UTC m=+22.832406235" Jul 12 19:22:38.115943 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:38.115879 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-zs4lw" podStartSLOduration=5.019591689 podStartE2EDuration="23.115863564s" totalImagesPullingTime="18.096271875s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-12 19:22:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 19:22:38.115805832 +0000 UTC m=+22.848378322" watchObservedRunningTime="2026-07-12 19:22:38.115863564 +0000 UTC m=+22.848436057" Jul 12 19:22:38.148064 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:38.148020 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-vsllh" podStartSLOduration=5.03768289 podStartE2EDuration="23.148005956s" totalImagesPullingTime="18.110323066s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-12 19:22:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 19:22:38.14780825 +0000 UTC m=+22.880380742" watchObservedRunningTime="2026-07-12 19:22:38.148005956 +0000 UTC m=+22.880578446" Jul 12 19:22:38.163872 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:38.163830 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-7vvbp" podStartSLOduration=13.145010754 podStartE2EDuration="23.163815047s" totalImagesPullingTime="10.018804293s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 19:22:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 19:22:38.163312031 +0000 UTC m=+22.895884523" watchObservedRunningTime="2026-07-12 19:22:38.163815047 +0000 UTC m=+22.896387536" Jul 12 19:22:38.176186 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:38.176140 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-c89jb" podStartSLOduration=5.068680866 podStartE2EDuration="23.176122472s" totalImagesPullingTime="18.107441606s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-12 19:22:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 19:22:38.176072926 +0000 UTC m=+22.908645420" watchObservedRunningTime="2026-07-12 19:22:38.176122472 +0000 UTC m=+22.908694961" Jul 12 19:22:38.247170 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:38.247131 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:22:38.324319 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:38.324289 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:22:38.548322 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:38.548297 2565 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 12 19:22:38.785204 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:38.785030 2565 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-12T19:22:38.548314334Z","UUID":"1b2af078-4f7b-4071-a2a3-1b260c190abf","Handler":null,"Name":"","Endpoint":""} Jul 12 19:22:38.786610 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:38.786584 2565 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 12 19:22:38.786757 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:38.786628 2565 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 12 19:22:39.103155 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:39.103063 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bwmdr" event={"ID":"56c9f12c-6e22-414d-a8d7-b503ed1f38df","Type":"ContainerStarted","Data":"e7cba005bd0e0af10dd44e507e46e4a9b2156d4999ed777879d5f2d397a878d4"} Jul 12 19:22:39.104863 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:39.104834 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-hrrhk" event={"ID":"087e3f6e-992f-4784-a376-bd2af99f9851","Type":"ContainerStarted","Data":"2f5591e7e29014806062bf8f4874399437fe4a6c52e58808612539af3db8e672"} Jul 12 19:22:39.108816 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:39.108794 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k5h9m_8e016c2e-0c3a-49da-83d4-f526738c1cbd/ovn-acl-logging/0.log" Jul 12 19:22:39.109322 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:39.109294 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k5h9m" event={"ID":"8e016c2e-0c3a-49da-83d4-f526738c1cbd","Type":"ContainerStarted","Data":"4ba3f62720227259393ce9a2cb5de82b50acfe51d8983de77dc465acd1462073"} Jul 12 19:22:39.121180 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:39.121134 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-hrrhk" podStartSLOduration=6.020164509 podStartE2EDuration="24.121119563s" totalImagesPullingTime="18.100955054s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-12 19:22:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 19:22:39.12074945 +0000 UTC m=+23.853321940" watchObservedRunningTime="2026-07-12 19:22:39.121119563 +0000 UTC m=+23.853692086" Jul 12 19:22:39.178811 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:39.178781 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:22:39.279317 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:39.279278 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:22:39.350373 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:39.350341 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:22:39.563062 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:39.563027 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:22:39.662547 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:39.662518 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:22:39.817235 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:39.817158 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:22:39.852603 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:39.852580 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2xmfq" Jul 12 19:22:39.852603 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:39.852595 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-76gfm" Jul 12 19:22:39.852762 ip-10-0-139-170 kubenswrapper[2565]: E0712 19:22:39.852702 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2xmfq" podUID="e7722ae2-e293-478e-9ac8-ea376f8d583f" Jul 12 19:22:39.852762 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:39.852738 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-288bq" Jul 12 19:22:39.852854 ip-10-0-139-170 kubenswrapper[2565]: E0712 19:22:39.852827 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-288bq" podUID="a7a0abe6-09ba-429e-a7f2-8de5b3f5d620" Jul 12 19:22:39.852933 ip-10-0-139-170 kubenswrapper[2565]: E0712 19:22:39.852909 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-76gfm" podUID="10c67d44-bdd0-4530-9f5d-fa10238ee66e" Jul 12 19:22:39.907725 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:39.907693 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:22:39.995832 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:39.995791 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:22:40.114276 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:40.114196 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bwmdr" event={"ID":"56c9f12c-6e22-414d-a8d7-b503ed1f38df","Type":"ContainerStarted","Data":"67ddea7252adc72c4c867928bbe7377031d7d90dba564f9d4b0753a5556cdc75"} Jul 12 19:22:40.121044 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:40.121017 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:22:40.131272 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:40.131234 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bwmdr" podStartSLOduration=4.663375565 podStartE2EDuration="25.131220394s" totalImagesPullingTime="20.467844829s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-12 19:22:15 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 19:22:40.130557311 +0000 UTC m=+24.863129800" watchObservedRunningTime="2026-07-12 19:22:40.131220394 +0000 UTC m=+24.863792883" Jul 12 19:22:40.222698 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:40.222667 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:22:40.546755 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:40.546568 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:22:40.594973 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:40.594934 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:22:40.819628 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:40.819594 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:22:40.868552 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:40.868512 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:22:41.120034 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:41.119968 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k5h9m_8e016c2e-0c3a-49da-83d4-f526738c1cbd/ovn-acl-logging/0.log" Jul 12 19:22:41.120476 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:41.120452 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k5h9m" event={"ID":"8e016c2e-0c3a-49da-83d4-f526738c1cbd","Type":"ContainerStarted","Data":"fd387ef54dfbd7d1c0db0df69ab1f9e6e21ec0ec0813803c6dc65624539c364e"} Jul 12 19:22:41.851758 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:41.851723 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-288bq" Jul 12 19:22:41.851758 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:41.851762 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-76gfm" Jul 12 19:22:41.852008 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:41.851723 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2xmfq" Jul 12 19:22:41.852008 ip-10-0-139-170 kubenswrapper[2565]: E0712 19:22:41.851851 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-288bq" podUID="a7a0abe6-09ba-429e-a7f2-8de5b3f5d620" Jul 12 19:22:41.852008 ip-10-0-139-170 kubenswrapper[2565]: E0712 19:22:41.851929 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2xmfq" podUID="e7722ae2-e293-478e-9ac8-ea376f8d583f" Jul 12 19:22:41.852008 ip-10-0-139-170 kubenswrapper[2565]: E0712 19:22:41.851997 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-76gfm" podUID="10c67d44-bdd0-4530-9f5d-fa10238ee66e" Jul 12 19:22:42.134305 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:42.134238 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-zs4lw_8e832009-b138-4442-b9db-10e73957481e/dns-node-resolver/0.log" Jul 12 19:22:42.268968 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:42.268945 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-7vvbp" Jul 12 19:22:42.269434 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:42.269419 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-7vvbp" Jul 12 19:22:42.448304 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:42.448278 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:22:42.534908 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:42.534862 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-c89jb_b8cca381-48d7-43ac-ae6a-15f96d99345d/node-ca/0.log" Jul 12 19:22:42.945325 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:42.945140 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:22:43.044945 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:43.044914 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:22:43.126545 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:43.126512 2565 generic.go:356] "Generic (PLEG): container finished" podID="51dee771-9663-4366-90ca-9b1c286e35f8" containerID="7e0d3e0d1ecba6469b6afceb4a758f0393f17e9358d87e2983f46a13cdc3358e" exitCode=0 Jul 12 19:22:43.126715 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:43.126555 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-prfcd" event={"ID":"51dee771-9663-4366-90ca-9b1c286e35f8","Type":"ContainerDied","Data":"7e0d3e0d1ecba6469b6afceb4a758f0393f17e9358d87e2983f46a13cdc3358e"} Jul 12 19:22:43.130093 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:43.130075 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k5h9m_8e016c2e-0c3a-49da-83d4-f526738c1cbd/ovn-acl-logging/0.log" Jul 12 19:22:43.130610 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:43.130582 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k5h9m" event={"ID":"8e016c2e-0c3a-49da-83d4-f526738c1cbd","Type":"ContainerStarted","Data":"626ae1df78227fac24aba51050ecfdd41f387d33777b3220cddff1ec46af93c8"} Jul 12 19:22:43.130980 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:43.130960 2565 scope.go:117] "RemoveContainer" containerID="05f2c00691adb37392b6ecfcf4d5a31a8fed8e65e58f9a145ed612f19da7ec00" Jul 12 19:22:43.269646 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:43.269617 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:22:43.545603 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:43.545522 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-7vvbp" Jul 12 19:22:43.546254 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:43.546234 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-7vvbp" Jul 12 19:22:43.852094 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:43.852026 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2xmfq" Jul 12 19:22:43.852094 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:43.852061 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-288bq" Jul 12 19:22:43.852268 ip-10-0-139-170 kubenswrapper[2565]: E0712 19:22:43.852152 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2xmfq" podUID="e7722ae2-e293-478e-9ac8-ea376f8d583f" Jul 12 19:22:43.852309 ip-10-0-139-170 kubenswrapper[2565]: E0712 19:22:43.852253 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-288bq" podUID="a7a0abe6-09ba-429e-a7f2-8de5b3f5d620" Jul 12 19:22:43.852345 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:43.852325 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-76gfm" Jul 12 19:22:43.852424 ip-10-0-139-170 kubenswrapper[2565]: E0712 19:22:43.852409 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-76gfm" podUID="10c67d44-bdd0-4530-9f5d-fa10238ee66e" Jul 12 19:22:44.136834 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:44.136770 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k5h9m_8e016c2e-0c3a-49da-83d4-f526738c1cbd/ovn-acl-logging/0.log" Jul 12 19:22:44.137198 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:44.137177 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k5h9m" event={"ID":"8e016c2e-0c3a-49da-83d4-f526738c1cbd","Type":"ContainerStarted","Data":"37f14a58f4b1512c3342e31df08d967909a45ded1560371b005f892d0cbbba81"} Jul 12 19:22:44.137447 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:44.137420 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-k5h9m" Jul 12 19:22:44.137555 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:44.137502 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-k5h9m" Jul 12 19:22:44.137555 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:44.137553 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-k5h9m" Jul 12 19:22:44.153539 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:44.153506 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-k5h9m" Jul 12 19:22:44.153858 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:44.153842 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-k5h9m" Jul 12 19:22:44.168347 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:44.168302 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-k5h9m" podStartSLOduration=10.779451547 podStartE2EDuration="29.168284617s" totalImagesPullingTime="18.38883307s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-12 19:22:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 19:22:44.163605715 +0000 UTC m=+28.896178206" watchObservedRunningTime="2026-07-12 19:22:44.168284617 +0000 UTC m=+28.900857133" Jul 12 19:22:44.354479 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:44.354286 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:22:44.510794 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:44.510760 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-288bq"] Jul 12 19:22:44.510956 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:44.510919 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-288bq" Jul 12 19:22:44.511025 ip-10-0-139-170 kubenswrapper[2565]: E0712 19:22:44.511006 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-288bq" podUID="a7a0abe6-09ba-429e-a7f2-8de5b3f5d620" Jul 12 19:22:44.514619 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:44.514592 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-2xmfq"] Jul 12 19:22:44.514747 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:44.514711 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2xmfq" Jul 12 19:22:44.514845 ip-10-0-139-170 kubenswrapper[2565]: E0712 19:22:44.514823 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2xmfq" podUID="e7722ae2-e293-478e-9ac8-ea376f8d583f" Jul 12 19:22:44.515267 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:44.515239 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-76gfm"] Jul 12 19:22:44.515354 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:44.515344 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-76gfm" Jul 12 19:22:44.515437 ip-10-0-139-170 kubenswrapper[2565]: E0712 19:22:44.515414 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-76gfm" podUID="10c67d44-bdd0-4530-9f5d-fa10238ee66e" Jul 12 19:22:45.141324 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:45.141292 2565 generic.go:356] "Generic (PLEG): container finished" podID="51dee771-9663-4366-90ca-9b1c286e35f8" containerID="0b02931f3cc35169e2cf5aa458b2a39b4a7612bda6683dc5eb3c4a4de27e13b9" exitCode=0 Jul 12 19:22:45.141554 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:45.141376 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-prfcd" event={"ID":"51dee771-9663-4366-90ca-9b1c286e35f8","Type":"ContainerDied","Data":"0b02931f3cc35169e2cf5aa458b2a39b4a7612bda6683dc5eb3c4a4de27e13b9"} Jul 12 19:22:45.241608 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:45.241574 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:22:45.575702 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:45.575625 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:22:45.796850 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:45.796810 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:22:45.853581 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:45.853510 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-288bq" Jul 12 19:22:45.853702 ip-10-0-139-170 kubenswrapper[2565]: E0712 19:22:45.853627 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-288bq" podUID="a7a0abe6-09ba-429e-a7f2-8de5b3f5d620" Jul 12 19:22:45.853702 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:45.853672 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2xmfq" Jul 12 19:22:45.853784 ip-10-0-139-170 kubenswrapper[2565]: E0712 19:22:45.853766 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2xmfq" podUID="e7722ae2-e293-478e-9ac8-ea376f8d583f" Jul 12 19:22:45.884813 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:45.884784 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:22:45.985343 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:45.985315 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:22:46.086097 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:46.086060 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:22:46.282193 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:46.282159 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:22:46.852166 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:46.852095 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-76gfm" Jul 12 19:22:46.852767 ip-10-0-139-170 kubenswrapper[2565]: E0712 19:22:46.852219 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-76gfm" podUID="10c67d44-bdd0-4530-9f5d-fa10238ee66e" Jul 12 19:22:47.083621 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:47.083543 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:22:47.148108 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:47.148074 2565 generic.go:356] "Generic (PLEG): container finished" podID="51dee771-9663-4366-90ca-9b1c286e35f8" containerID="470cb9e7d2f89804fd67160279728d08eda58defb87fb3c11c01941dd5366d6c" exitCode=0 Jul 12 19:22:47.148243 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:47.148148 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-prfcd" event={"ID":"51dee771-9663-4366-90ca-9b1c286e35f8","Type":"ContainerDied","Data":"470cb9e7d2f89804fd67160279728d08eda58defb87fb3c11c01941dd5366d6c"} Jul 12 19:22:47.680269 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:47.680236 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:22:47.851593 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:47.851555 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2xmfq" Jul 12 19:22:47.851593 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:47.851596 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-288bq" Jul 12 19:22:47.851833 ip-10-0-139-170 kubenswrapper[2565]: E0712 19:22:47.851699 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2xmfq" podUID="e7722ae2-e293-478e-9ac8-ea376f8d583f" Jul 12 19:22:47.851833 ip-10-0-139-170 kubenswrapper[2565]: E0712 19:22:47.851815 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-288bq" podUID="a7a0abe6-09ba-429e-a7f2-8de5b3f5d620" Jul 12 19:22:47.882990 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:47.882953 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:22:48.079210 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:48.079127 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:22:48.411346 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:48.411312 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:22:48.513320 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:48.513277 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:22:48.559710 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:48.559678 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:22:48.852533 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:48.852445 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-76gfm" Jul 12 19:22:48.852697 ip-10-0-139-170 kubenswrapper[2565]: E0712 19:22:48.852570 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-76gfm" podUID="10c67d44-bdd0-4530-9f5d-fa10238ee66e" Jul 12 19:22:49.077617 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:49.077585 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:22:49.276296 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:49.276265 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:22:49.380521 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:49.380485 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:22:49.442799 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:49.442769 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e7722ae2-e293-478e-9ac8-ea376f8d583f-metrics-certs\") pod \"network-metrics-daemon-2xmfq\" (UID: \"e7722ae2-e293-478e-9ac8-ea376f8d583f\") " pod="openshift-multus/network-metrics-daemon-2xmfq" Jul 12 19:22:49.442982 ip-10-0-139-170 kubenswrapper[2565]: E0712 19:22:49.442913 2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 12 19:22:49.442982 ip-10-0-139-170 kubenswrapper[2565]: E0712 19:22:49.442979 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e7722ae2-e293-478e-9ac8-ea376f8d583f-metrics-certs podName:e7722ae2-e293-478e-9ac8-ea376f8d583f nodeName:}" failed. No retries permitted until 2026-07-12 19:23:21.442962363 +0000 UTC m=+66.175534834 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e7722ae2-e293-478e-9ac8-ea376f8d583f-metrics-certs") pod "network-metrics-daemon-2xmfq" (UID: "e7722ae2-e293-478e-9ac8-ea376f8d583f") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 12 19:22:49.480312 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:49.480282 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:22:49.543685 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:49.543600 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a7a0abe6-09ba-429e-a7f2-8de5b3f5d620-original-pull-secret\") pod \"global-pull-secret-syncer-288bq\" (UID: \"a7a0abe6-09ba-429e-a7f2-8de5b3f5d620\") " pod="kube-system/global-pull-secret-syncer-288bq" Jul 12 19:22:49.543838 ip-10-0-139-170 kubenswrapper[2565]: E0712 19:22:49.543740 2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 12 19:22:49.543838 ip-10-0-139-170 kubenswrapper[2565]: E0712 19:22:49.543806 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a7a0abe6-09ba-429e-a7f2-8de5b3f5d620-original-pull-secret podName:a7a0abe6-09ba-429e-a7f2-8de5b3f5d620 nodeName:}" failed. No retries permitted until 2026-07-12 19:23:21.543787983 +0000 UTC m=+66.276360464 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a7a0abe6-09ba-429e-a7f2-8de5b3f5d620-original-pull-secret") pod "global-pull-secret-syncer-288bq" (UID: "a7a0abe6-09ba-429e-a7f2-8de5b3f5d620") : object "kube-system"/"original-pull-secret" not registered Jul 12 19:22:49.644493 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:49.644459 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bp96h\" (UniqueName: \"kubernetes.io/projected/10c67d44-bdd0-4530-9f5d-fa10238ee66e-kube-api-access-bp96h\") pod \"network-check-target-76gfm\" (UID: \"10c67d44-bdd0-4530-9f5d-fa10238ee66e\") " pod="openshift-network-diagnostics/network-check-target-76gfm" Jul 12 19:22:49.644663 ip-10-0-139-170 kubenswrapper[2565]: E0712 19:22:49.644618 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 12 19:22:49.644741 ip-10-0-139-170 kubenswrapper[2565]: E0712 19:22:49.644638 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 12 19:22:49.644741 ip-10-0-139-170 kubenswrapper[2565]: E0712 19:22:49.644679 2565 projected.go:196] Error preparing data for projected volume kube-api-access-bp96h for pod openshift-network-diagnostics/network-check-target-76gfm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 12 19:22:49.644834 ip-10-0-139-170 kubenswrapper[2565]: E0712 19:22:49.644743 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/10c67d44-bdd0-4530-9f5d-fa10238ee66e-kube-api-access-bp96h podName:10c67d44-bdd0-4530-9f5d-fa10238ee66e nodeName:}" failed. No retries permitted until 2026-07-12 19:23:21.644723311 +0000 UTC m=+66.377295795 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-bp96h" (UniqueName: "kubernetes.io/projected/10c67d44-bdd0-4530-9f5d-fa10238ee66e-kube-api-access-bp96h") pod "network-check-target-76gfm" (UID: "10c67d44-bdd0-4530-9f5d-fa10238ee66e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 12 19:22:49.851983 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:49.851873 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-288bq" Jul 12 19:22:49.852144 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:49.851900 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2xmfq" Jul 12 19:22:49.852144 ip-10-0-139-170 kubenswrapper[2565]: E0712 19:22:49.852023 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-288bq" podUID="a7a0abe6-09ba-429e-a7f2-8de5b3f5d620" Jul 12 19:22:49.852144 ip-10-0-139-170 kubenswrapper[2565]: E0712 19:22:49.852108 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2xmfq" podUID="e7722ae2-e293-478e-9ac8-ea376f8d583f" Jul 12 19:22:49.878441 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:49.878402 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:22:50.278482 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:50.278440 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:22:50.328769 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:50.328740 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:22:50.475670 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:50.475633 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:22:50.851789 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:50.851760 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-76gfm" Jul 12 19:22:50.851986 ip-10-0-139-170 kubenswrapper[2565]: E0712 19:22:50.851879 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-76gfm" podUID="10c67d44-bdd0-4530-9f5d-fa10238ee66e" Jul 12 19:22:51.076172 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:51.075954 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:22:51.479315 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:51.479204 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:22:51.655793 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:51.655762 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:22:51.704387 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:51.704353 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:22:51.852517 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:51.852426 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-288bq" Jul 12 19:22:51.852664 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:51.852439 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2xmfq" Jul 12 19:22:51.852664 ip-10-0-139-170 kubenswrapper[2565]: E0712 19:22:51.852533 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-288bq" podUID="a7a0abe6-09ba-429e-a7f2-8de5b3f5d620" Jul 12 19:22:51.852664 ip-10-0-139-170 kubenswrapper[2565]: E0712 19:22:51.852653 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2xmfq" podUID="e7722ae2-e293-478e-9ac8-ea376f8d583f" Jul 12 19:22:51.887593 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:51.887545 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:22:52.162318 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:52.162280 2565 generic.go:356] "Generic (PLEG): container finished" podID="51dee771-9663-4366-90ca-9b1c286e35f8" containerID="0301f3940d2297fea32ab78a4aa69e1438df2bea8192c29c4947cba3959e5af6" exitCode=0 Jul 12 19:22:52.162474 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:52.162336 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-prfcd" event={"ID":"51dee771-9663-4366-90ca-9b1c286e35f8","Type":"ContainerDied","Data":"0301f3940d2297fea32ab78a4aa69e1438df2bea8192c29c4947cba3959e5af6"} Jul 12 19:22:52.277902 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:52.277861 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:22:52.677479 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:52.677444 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:22:52.852158 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:52.852128 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-76gfm" Jul 12 19:22:52.852320 ip-10-0-139-170 kubenswrapper[2565]: E0712 19:22:52.852226 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-76gfm" podUID="10c67d44-bdd0-4530-9f5d-fa10238ee66e" Jul 12 19:22:52.875234 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:52.875203 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:22:53.168007 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:53.167972 2565 generic.go:356] "Generic (PLEG): container finished" podID="51dee771-9663-4366-90ca-9b1c286e35f8" containerID="a2937561476e77520347ef1351a16ed68a5fcfc304ca86438c13f5d18f4cbe87" exitCode=0 Jul 12 19:22:53.168167 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:53.168025 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-prfcd" event={"ID":"51dee771-9663-4366-90ca-9b1c286e35f8","Type":"ContainerDied","Data":"a2937561476e77520347ef1351a16ed68a5fcfc304ca86438c13f5d18f4cbe87"} Jul 12 19:22:53.479013 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:53.478924 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:22:53.852062 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:53.851992 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-288bq" Jul 12 19:22:53.852435 ip-10-0-139-170 kubenswrapper[2565]: E0712 19:22:53.852098 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-288bq" podUID="a7a0abe6-09ba-429e-a7f2-8de5b3f5d620" Jul 12 19:22:53.852435 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:53.852178 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2xmfq" Jul 12 19:22:53.852435 ip-10-0-139-170 kubenswrapper[2565]: E0712 19:22:53.852267 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2xmfq" podUID="e7722ae2-e293-478e-9ac8-ea376f8d583f" Jul 12 19:22:54.083798 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:54.083767 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:22:54.173866 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:54.173837 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-prfcd" event={"ID":"51dee771-9663-4366-90ca-9b1c286e35f8","Type":"ContainerStarted","Data":"e5d4ddf9c6237cb78b90fbc408ba65fa4cfdfb299a990d69592d9ac92d436672"} Jul 12 19:22:54.196069 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:54.196023 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-prfcd" podStartSLOduration=10.883637221 podStartE2EDuration="39.196008983s" totalImagesPullingTime="28.312371762s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-12 19:22:15 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 19:22:54.194783572 +0000 UTC m=+38.927356061" watchObservedRunningTime="2026-07-12 19:22:54.196008983 +0000 UTC m=+38.928581473" Jul 12 19:22:54.280170 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:54.280135 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:22:54.482964 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:54.482806 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:22:54.679504 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:54.679474 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:22:54.732198 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:54.732161 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:22:54.852351 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:54.852274 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-76gfm" Jul 12 19:22:54.852692 ip-10-0-139-170 kubenswrapper[2565]: E0712 19:22:54.852372 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-76gfm" podUID="10c67d44-bdd0-4530-9f5d-fa10238ee66e" Jul 12 19:22:55.084440 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:55.084404 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:22:55.141380 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:55.141354 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:22:55.683245 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:55.683218 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:22:55.853350 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:55.853315 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-288bq" Jul 12 19:22:55.853711 ip-10-0-139-170 kubenswrapper[2565]: E0712 19:22:55.853423 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-288bq" podUID="a7a0abe6-09ba-429e-a7f2-8de5b3f5d620" Jul 12 19:22:55.853711 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:55.853432 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2xmfq" Jul 12 19:22:55.853711 ip-10-0-139-170 kubenswrapper[2565]: E0712 19:22:55.853495 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2xmfq" podUID="e7722ae2-e293-478e-9ac8-ea376f8d583f" Jul 12 19:22:55.880304 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:55.880278 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:22:56.079237 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:56.079156 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:22:56.281444 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:56.281409 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:22:56.851991 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:56.851956 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-76gfm" Jul 12 19:22:56.852143 ip-10-0-139-170 kubenswrapper[2565]: E0712 19:22:56.852054 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-76gfm" podUID="10c67d44-bdd0-4530-9f5d-fa10238ee66e" Jul 12 19:22:57.852489 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:57.852461 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-288bq" Jul 12 19:22:57.852945 ip-10-0-139-170 kubenswrapper[2565]: E0712 19:22:57.852575 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-288bq" podUID="a7a0abe6-09ba-429e-a7f2-8de5b3f5d620" Jul 12 19:22:57.852945 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:57.852630 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2xmfq" Jul 12 19:22:57.852945 ip-10-0-139-170 kubenswrapper[2565]: E0712 19:22:57.852724 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2xmfq" podUID="e7722ae2-e293-478e-9ac8-ea376f8d583f" Jul 12 19:22:58.851452 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:58.851419 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-76gfm" Jul 12 19:22:58.851630 ip-10-0-139-170 kubenswrapper[2565]: E0712 19:22:58.851530 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-76gfm" podUID="10c67d44-bdd0-4530-9f5d-fa10238ee66e" Jul 12 19:22:59.854458 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:59.854430 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-288bq" Jul 12 19:22:59.854826 ip-10-0-139-170 kubenswrapper[2565]: E0712 19:22:59.854526 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-288bq" podUID="a7a0abe6-09ba-429e-a7f2-8de5b3f5d620" Jul 12 19:22:59.854826 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:22:59.854573 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2xmfq" Jul 12 19:22:59.854826 ip-10-0-139-170 kubenswrapper[2565]: E0712 19:22:59.854656 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2xmfq" podUID="e7722ae2-e293-478e-9ac8-ea376f8d583f" Jul 12 19:23:00.574130 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:00.574098 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:23:00.674671 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:00.674635 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:23:00.852116 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:00.852044 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-76gfm" Jul 12 19:23:00.852300 ip-10-0-139-170 kubenswrapper[2565]: E0712 19:23:00.852138 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-76gfm" podUID="10c67d44-bdd0-4530-9f5d-fa10238ee66e" Jul 12 19:23:01.043121 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:01.043084 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:23:01.249869 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:01.249840 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:23:01.345420 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:01.345384 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:23:01.496919 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:01.496870 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:23:01.611002 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:01.610971 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:23:01.709149 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:01.709118 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:23:01.764605 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:01.764577 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:23:01.852134 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:01.852053 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2xmfq" Jul 12 19:23:01.852291 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:01.852053 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-288bq" Jul 12 19:23:01.852291 ip-10-0-139-170 kubenswrapper[2565]: E0712 19:23:01.852193 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2xmfq" podUID="e7722ae2-e293-478e-9ac8-ea376f8d583f" Jul 12 19:23:01.852291 ip-10-0-139-170 kubenswrapper[2565]: E0712 19:23:01.852252 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-288bq" podUID="a7a0abe6-09ba-429e-a7f2-8de5b3f5d620" Jul 12 19:23:02.054392 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:02.054362 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:23:02.106070 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:02.105996 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-170.ec2.internal" event="NodeReady" Jul 12 19:23:02.106211 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:02.106122 2565 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 12 19:23:02.146604 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:02.146581 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-zn4n8"] Jul 12 19:23:02.177834 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:02.177808 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-vcdcf"] Jul 12 19:23:02.177980 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:02.177939 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-zn4n8" Jul 12 19:23:02.180298 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:02.180271 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-kfwdz\"" Jul 12 19:23:02.180298 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:02.180286 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 12 19:23:02.180480 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:02.180407 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 12 19:23:02.191826 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:02.191804 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-zn4n8"] Jul 12 19:23:02.191970 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:02.191843 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-vcdcf"] Jul 12 19:23:02.191970 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:02.191862 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-qhx89"] Jul 12 19:23:02.191970 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:02.191918 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-vcdcf" Jul 12 19:23:02.194421 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:02.194399 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 12 19:23:02.194693 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:02.194678 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 12 19:23:02.194775 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:02.194759 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-vfn6k\"" Jul 12 19:23:02.194835 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:02.194779 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 12 19:23:02.213904 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:02.213867 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-qhx89"] Jul 12 19:23:02.214004 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:02.213983 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-qhx89" Jul 12 19:23:02.216308 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:02.216291 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 12 19:23:02.216406 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:02.216310 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-gnkpn\"" Jul 12 19:23:02.216540 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:02.216528 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 12 19:23:02.216586 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:02.216540 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 12 19:23:02.217092 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:02.217073 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 12 19:23:02.242860 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:02.242839 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:23:02.337836 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:02.337811 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a7e56501-c8eb-4615-8bf8-1598f8f9fe39-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qhx89\" (UID: \"a7e56501-c8eb-4615-8bf8-1598f8f9fe39\") " pod="openshift-insights/insights-runtime-extractor-qhx89" Jul 12 19:23:02.337961 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:02.337842 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a7e56501-c8eb-4615-8bf8-1598f8f9fe39-data-volume\") pod \"insights-runtime-extractor-qhx89\" (UID: \"a7e56501-c8eb-4615-8bf8-1598f8f9fe39\") " pod="openshift-insights/insights-runtime-extractor-qhx89" Jul 12 19:23:02.337961 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:02.337860 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/53b79340-2472-4ee4-ab93-4eecee72fbfb-config-volume\") pod \"dns-default-zn4n8\" (UID: \"53b79340-2472-4ee4-ab93-4eecee72fbfb\") " pod="openshift-dns/dns-default-zn4n8" Jul 12 19:23:02.337961 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:02.337873 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/53b79340-2472-4ee4-ab93-4eecee72fbfb-metrics-tls\") pod \"dns-default-zn4n8\" (UID: \"53b79340-2472-4ee4-ab93-4eecee72fbfb\") " pod="openshift-dns/dns-default-zn4n8" Jul 12 19:23:02.337961 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:02.337913 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a7e56501-c8eb-4615-8bf8-1598f8f9fe39-crio-socket\") pod \"insights-runtime-extractor-qhx89\" (UID: \"a7e56501-c8eb-4615-8bf8-1598f8f9fe39\") " pod="openshift-insights/insights-runtime-extractor-qhx89" Jul 12 19:23:02.337961 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:02.337930 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mc2mr\" (UniqueName: \"kubernetes.io/projected/ea94aee7-5bcb-41be-9513-c66834d390e3-kube-api-access-mc2mr\") pod \"ingress-canary-vcdcf\" (UID: \"ea94aee7-5bcb-41be-9513-c66834d390e3\") " pod="openshift-ingress-canary/ingress-canary-vcdcf" Jul 12 19:23:02.338194 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:02.337972 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7smv6\" (UniqueName: \"kubernetes.io/projected/a7e56501-c8eb-4615-8bf8-1598f8f9fe39-kube-api-access-7smv6\") pod \"insights-runtime-extractor-qhx89\" (UID: \"a7e56501-c8eb-4615-8bf8-1598f8f9fe39\") " pod="openshift-insights/insights-runtime-extractor-qhx89" Jul 12 19:23:02.338194 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:02.338006 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ea94aee7-5bcb-41be-9513-c66834d390e3-cert\") pod \"ingress-canary-vcdcf\" (UID: \"ea94aee7-5bcb-41be-9513-c66834d390e3\") " pod="openshift-ingress-canary/ingress-canary-vcdcf" Jul 12 19:23:02.338194 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:02.338046 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/53b79340-2472-4ee4-ab93-4eecee72fbfb-tmp-dir\") pod \"dns-default-zn4n8\" (UID: \"53b79340-2472-4ee4-ab93-4eecee72fbfb\") " pod="openshift-dns/dns-default-zn4n8" Jul 12 19:23:02.338194 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:02.338075 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a7e56501-c8eb-4615-8bf8-1598f8f9fe39-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qhx89\" (UID: \"a7e56501-c8eb-4615-8bf8-1598f8f9fe39\") " pod="openshift-insights/insights-runtime-extractor-qhx89" Jul 12 19:23:02.338194 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:02.338134 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-822nq\" (UniqueName: \"kubernetes.io/projected/53b79340-2472-4ee4-ab93-4eecee72fbfb-kube-api-access-822nq\") pod \"dns-default-zn4n8\" (UID: \"53b79340-2472-4ee4-ab93-4eecee72fbfb\") " pod="openshift-dns/dns-default-zn4n8" Jul 12 19:23:02.342210 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:02.342191 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:23:02.438966 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:02.438931 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7smv6\" (UniqueName: \"kubernetes.io/projected/a7e56501-c8eb-4615-8bf8-1598f8f9fe39-kube-api-access-7smv6\") pod \"insights-runtime-extractor-qhx89\" (UID: \"a7e56501-c8eb-4615-8bf8-1598f8f9fe39\") " pod="openshift-insights/insights-runtime-extractor-qhx89" Jul 12 19:23:02.438966 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:02.438971 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ea94aee7-5bcb-41be-9513-c66834d390e3-cert\") pod \"ingress-canary-vcdcf\" (UID: \"ea94aee7-5bcb-41be-9513-c66834d390e3\") " pod="openshift-ingress-canary/ingress-canary-vcdcf" Jul 12 19:23:02.439198 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:02.439004 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/53b79340-2472-4ee4-ab93-4eecee72fbfb-tmp-dir\") pod \"dns-default-zn4n8\" (UID: \"53b79340-2472-4ee4-ab93-4eecee72fbfb\") " pod="openshift-dns/dns-default-zn4n8" Jul 12 19:23:02.439198 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:02.439029 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a7e56501-c8eb-4615-8bf8-1598f8f9fe39-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qhx89\" (UID: \"a7e56501-c8eb-4615-8bf8-1598f8f9fe39\") " pod="openshift-insights/insights-runtime-extractor-qhx89" Jul 12 19:23:02.439198 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:02.439073 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-822nq\" (UniqueName: \"kubernetes.io/projected/53b79340-2472-4ee4-ab93-4eecee72fbfb-kube-api-access-822nq\") pod \"dns-default-zn4n8\" (UID: \"53b79340-2472-4ee4-ab93-4eecee72fbfb\") " pod="openshift-dns/dns-default-zn4n8" Jul 12 19:23:02.439198 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:02.439122 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a7e56501-c8eb-4615-8bf8-1598f8f9fe39-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qhx89\" (UID: \"a7e56501-c8eb-4615-8bf8-1598f8f9fe39\") " pod="openshift-insights/insights-runtime-extractor-qhx89" Jul 12 19:23:02.439198 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:02.439156 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a7e56501-c8eb-4615-8bf8-1598f8f9fe39-data-volume\") pod \"insights-runtime-extractor-qhx89\" (UID: \"a7e56501-c8eb-4615-8bf8-1598f8f9fe39\") " pod="openshift-insights/insights-runtime-extractor-qhx89" Jul 12 19:23:02.439198 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:02.439183 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/53b79340-2472-4ee4-ab93-4eecee72fbfb-config-volume\") pod \"dns-default-zn4n8\" (UID: \"53b79340-2472-4ee4-ab93-4eecee72fbfb\") " pod="openshift-dns/dns-default-zn4n8" Jul 12 19:23:02.439493 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:02.439204 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/53b79340-2472-4ee4-ab93-4eecee72fbfb-metrics-tls\") pod \"dns-default-zn4n8\" (UID: \"53b79340-2472-4ee4-ab93-4eecee72fbfb\") " pod="openshift-dns/dns-default-zn4n8" Jul 12 19:23:02.439493 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:02.439239 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a7e56501-c8eb-4615-8bf8-1598f8f9fe39-crio-socket\") pod \"insights-runtime-extractor-qhx89\" (UID: \"a7e56501-c8eb-4615-8bf8-1598f8f9fe39\") " pod="openshift-insights/insights-runtime-extractor-qhx89" Jul 12 19:23:02.439493 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:02.439262 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mc2mr\" (UniqueName: \"kubernetes.io/projected/ea94aee7-5bcb-41be-9513-c66834d390e3-kube-api-access-mc2mr\") pod \"ingress-canary-vcdcf\" (UID: \"ea94aee7-5bcb-41be-9513-c66834d390e3\") " pod="openshift-ingress-canary/ingress-canary-vcdcf" Jul 12 19:23:02.439493 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:02.439410 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/53b79340-2472-4ee4-ab93-4eecee72fbfb-tmp-dir\") pod \"dns-default-zn4n8\" (UID: \"53b79340-2472-4ee4-ab93-4eecee72fbfb\") " pod="openshift-dns/dns-default-zn4n8" Jul 12 19:23:02.439493 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:02.439469 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a7e56501-c8eb-4615-8bf8-1598f8f9fe39-data-volume\") pod \"insights-runtime-extractor-qhx89\" (UID: \"a7e56501-c8eb-4615-8bf8-1598f8f9fe39\") " pod="openshift-insights/insights-runtime-extractor-qhx89" Jul 12 19:23:02.439705 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:02.439626 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a7e56501-c8eb-4615-8bf8-1598f8f9fe39-crio-socket\") pod \"insights-runtime-extractor-qhx89\" (UID: \"a7e56501-c8eb-4615-8bf8-1598f8f9fe39\") " pod="openshift-insights/insights-runtime-extractor-qhx89" Jul 12 19:23:02.439768 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:02.439732 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/53b79340-2472-4ee4-ab93-4eecee72fbfb-config-volume\") pod \"dns-default-zn4n8\" (UID: \"53b79340-2472-4ee4-ab93-4eecee72fbfb\") " pod="openshift-dns/dns-default-zn4n8" Jul 12 19:23:02.439768 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:02.439732 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a7e56501-c8eb-4615-8bf8-1598f8f9fe39-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qhx89\" (UID: \"a7e56501-c8eb-4615-8bf8-1598f8f9fe39\") " pod="openshift-insights/insights-runtime-extractor-qhx89" Jul 12 19:23:02.443233 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:02.443212 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:23:02.443348 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:02.443248 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a7e56501-c8eb-4615-8bf8-1598f8f9fe39-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qhx89\" (UID: \"a7e56501-c8eb-4615-8bf8-1598f8f9fe39\") " pod="openshift-insights/insights-runtime-extractor-qhx89" Jul 12 19:23:02.443348 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:02.443251 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/53b79340-2472-4ee4-ab93-4eecee72fbfb-metrics-tls\") pod \"dns-default-zn4n8\" (UID: \"53b79340-2472-4ee4-ab93-4eecee72fbfb\") " pod="openshift-dns/dns-default-zn4n8" Jul 12 19:23:02.443348 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:02.443284 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ea94aee7-5bcb-41be-9513-c66834d390e3-cert\") pod \"ingress-canary-vcdcf\" (UID: \"ea94aee7-5bcb-41be-9513-c66834d390e3\") " pod="openshift-ingress-canary/ingress-canary-vcdcf" Jul 12 19:23:02.446058 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:02.446035 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7smv6\" (UniqueName: \"kubernetes.io/projected/a7e56501-c8eb-4615-8bf8-1598f8f9fe39-kube-api-access-7smv6\") pod \"insights-runtime-extractor-qhx89\" (UID: \"a7e56501-c8eb-4615-8bf8-1598f8f9fe39\") " pod="openshift-insights/insights-runtime-extractor-qhx89" Jul 12 19:23:02.446177 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:02.446160 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mc2mr\" (UniqueName: \"kubernetes.io/projected/ea94aee7-5bcb-41be-9513-c66834d390e3-kube-api-access-mc2mr\") pod \"ingress-canary-vcdcf\" (UID: \"ea94aee7-5bcb-41be-9513-c66834d390e3\") " pod="openshift-ingress-canary/ingress-canary-vcdcf" Jul 12 19:23:02.446327 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:02.446309 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-822nq\" (UniqueName: \"kubernetes.io/projected/53b79340-2472-4ee4-ab93-4eecee72fbfb-kube-api-access-822nq\") pod \"dns-default-zn4n8\" (UID: \"53b79340-2472-4ee4-ab93-4eecee72fbfb\") " pod="openshift-dns/dns-default-zn4n8" Jul 12 19:23:02.487332 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:02.487310 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-zn4n8" Jul 12 19:23:02.500421 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:02.500404 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-vcdcf" Jul 12 19:23:02.521495 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:02.521479 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-qhx89" Jul 12 19:23:02.638266 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:02.638236 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:23:02.723498 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:02.723473 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-zn4n8"] Jul 12 19:23:02.725948 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:23:02.725923 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53b79340_2472_4ee4_ab93_4eecee72fbfb.slice/crio-49283e6c2f0f1037fc7732a55f2083f5195c4c18118ff8a685e83412c2e8d3a4 WatchSource:0}: Error finding container 49283e6c2f0f1037fc7732a55f2083f5195c4c18118ff8a685e83412c2e8d3a4: Status 404 returned error can't find the container with id 49283e6c2f0f1037fc7732a55f2083f5195c4c18118ff8a685e83412c2e8d3a4 Jul 12 19:23:02.852343 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:02.852313 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-76gfm" Jul 12 19:23:02.854615 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:02.854593 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 12 19:23:02.854722 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:02.854668 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-cl9vq\"" Jul 12 19:23:02.854775 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:02.854756 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 12 19:23:02.942907 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:02.942813 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-vcdcf"] Jul 12 19:23:02.947163 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:02.946120 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-qhx89"] Jul 12 19:23:02.947459 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:23:02.947435 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea94aee7_5bcb_41be_9513_c66834d390e3.slice/crio-d5a62a48dfecf9bfd3fb769ac2cc9cf92816eb85ea377d7df1ef33d0cb7bca6a WatchSource:0}: Error finding container d5a62a48dfecf9bfd3fb769ac2cc9cf92816eb85ea377d7df1ef33d0cb7bca6a: Status 404 returned error can't find the container with id d5a62a48dfecf9bfd3fb769ac2cc9cf92816eb85ea377d7df1ef33d0cb7bca6a Jul 12 19:23:02.951047 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:23:02.951023 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7e56501_c8eb_4615_8bf8_1598f8f9fe39.slice/crio-8e612cd981e3e179390d737de84631595f781ed48c55c6c66838a94028ba298b WatchSource:0}: Error finding container 8e612cd981e3e179390d737de84631595f781ed48c55c6c66838a94028ba298b: Status 404 returned error can't find the container with id 8e612cd981e3e179390d737de84631595f781ed48c55c6c66838a94028ba298b Jul 12 19:23:03.123040 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:03.123005 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:23:03.174224 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:03.174195 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:23:03.195436 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:03.195366 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-vcdcf" event={"ID":"ea94aee7-5bcb-41be-9513-c66834d390e3","Type":"ContainerStarted","Data":"d5a62a48dfecf9bfd3fb769ac2cc9cf92816eb85ea377d7df1ef33d0cb7bca6a"} Jul 12 19:23:03.196726 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:03.196691 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-qhx89" event={"ID":"a7e56501-c8eb-4615-8bf8-1598f8f9fe39","Type":"ContainerStarted","Data":"8e612cd981e3e179390d737de84631595f781ed48c55c6c66838a94028ba298b"} Jul 12 19:23:03.197899 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:03.197870 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-zn4n8" event={"ID":"53b79340-2472-4ee4-ab93-4eecee72fbfb","Type":"ContainerStarted","Data":"49283e6c2f0f1037fc7732a55f2083f5195c4c18118ff8a685e83412c2e8d3a4"} Jul 12 19:23:03.794785 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:03.794736 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:23:03.852035 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:03.852004 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-288bq" Jul 12 19:23:03.852223 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:03.852011 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2xmfq" Jul 12 19:23:03.857830 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:03.856926 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-dcr6j\"" Jul 12 19:23:03.857830 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:03.857181 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 12 19:23:03.857830 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:03.857399 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 12 19:23:06.650442 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:06.650409 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:23:06.704743 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:06.704712 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:23:09.214946 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:09.214907 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-vcdcf" event={"ID":"ea94aee7-5bcb-41be-9513-c66834d390e3","Type":"ContainerStarted","Data":"a8cbea66c8c70598faeede546e84deb7ff9e8bd08185c6c0a48a2a2a141d9277"} Jul 12 19:23:09.216437 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:09.216412 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-qhx89" event={"ID":"a7e56501-c8eb-4615-8bf8-1598f8f9fe39","Type":"ContainerStarted","Data":"c7d4a443cfdd3f36f7aabbe9e7c4c78fddb934bd4b4194219e2760ffc0d76eb0"} Jul 12 19:23:09.218043 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:09.218016 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-zn4n8" event={"ID":"53b79340-2472-4ee4-ab93-4eecee72fbfb","Type":"ContainerStarted","Data":"db6fcc574377cd8c4f70510a48fd272749d3a6284ec1da2066e57656d3023ded"} Jul 12 19:23:09.218159 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:09.218063 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-zn4n8" event={"ID":"53b79340-2472-4ee4-ab93-4eecee72fbfb","Type":"ContainerStarted","Data":"311c5580655c34f2a234956c39c02b21974ad0b65a903a038da62f5167ea4b56"} Jul 12 19:23:09.218159 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:09.218101 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-zn4n8" Jul 12 19:23:09.229230 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:09.229183 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-vcdcf" podStartSLOduration=1.768977596 podStartE2EDuration="7.229168245s" totalImagesPullingTime="5.460190649s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 19:23:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 19:23:09.228079857 +0000 UTC m=+53.960652348" watchObservedRunningTime="2026-07-12 19:23:09.229168245 +0000 UTC m=+53.961740735" Jul 12 19:23:09.243395 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:09.243347 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-zn4n8" podStartSLOduration=1.5631131919999999 podStartE2EDuration="7.243329748s" totalImagesPullingTime="5.680216556s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 19:23:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 19:23:09.242269367 +0000 UTC m=+53.974841857" watchObservedRunningTime="2026-07-12 19:23:09.243329748 +0000 UTC m=+53.975902236" Jul 12 19:23:09.482992 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:09.482908 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:23:09.540500 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:09.540454 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:23:10.222364 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:10.222274 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-qhx89" event={"ID":"a7e56501-c8eb-4615-8bf8-1598f8f9fe39","Type":"ContainerStarted","Data":"55a12a63f0b9b5017a450b4c445ebae1250b8e65b3ae8032c9283bbbf6a23f7b"} Jul 12 19:23:12.251496 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:12.251460 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-b6d9ddcc-2fvdr"] Jul 12 19:23:12.353454 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:12.353412 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-b6d9ddcc-2fvdr"] Jul 12 19:23:12.353628 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:12.353514 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-b6d9ddcc-2fvdr" Jul 12 19:23:12.356478 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:12.356450 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 12 19:23:12.356618 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:12.356476 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-9d9l9\"" Jul 12 19:23:12.356618 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:12.356494 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 12 19:23:12.357461 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:12.357282 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 12 19:23:12.357461 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:12.357330 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 12 19:23:12.357461 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:12.357355 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 12 19:23:12.357461 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:12.357372 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 12 19:23:12.357461 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:12.357388 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 12 19:23:12.420232 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:12.420199 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/dc6274fb-accb-4f81-ae6e-8133c6ed4132-console-serving-cert\") pod \"console-b6d9ddcc-2fvdr\" (UID: \"dc6274fb-accb-4f81-ae6e-8133c6ed4132\") " pod="openshift-console/console-b6d9ddcc-2fvdr" Jul 12 19:23:12.420428 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:12.420242 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdkff\" (UniqueName: \"kubernetes.io/projected/dc6274fb-accb-4f81-ae6e-8133c6ed4132-kube-api-access-sdkff\") pod \"console-b6d9ddcc-2fvdr\" (UID: \"dc6274fb-accb-4f81-ae6e-8133c6ed4132\") " pod="openshift-console/console-b6d9ddcc-2fvdr" Jul 12 19:23:12.420428 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:12.420328 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/dc6274fb-accb-4f81-ae6e-8133c6ed4132-console-oauth-config\") pod \"console-b6d9ddcc-2fvdr\" (UID: \"dc6274fb-accb-4f81-ae6e-8133c6ed4132\") " pod="openshift-console/console-b6d9ddcc-2fvdr" Jul 12 19:23:12.420428 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:12.420358 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/dc6274fb-accb-4f81-ae6e-8133c6ed4132-console-config\") pod \"console-b6d9ddcc-2fvdr\" (UID: \"dc6274fb-accb-4f81-ae6e-8133c6ed4132\") " pod="openshift-console/console-b6d9ddcc-2fvdr" Jul 12 19:23:12.420591 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:12.420438 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/dc6274fb-accb-4f81-ae6e-8133c6ed4132-oauth-serving-cert\") pod \"console-b6d9ddcc-2fvdr\" (UID: \"dc6274fb-accb-4f81-ae6e-8133c6ed4132\") " pod="openshift-console/console-b6d9ddcc-2fvdr" Jul 12 19:23:12.420591 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:12.420485 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/dc6274fb-accb-4f81-ae6e-8133c6ed4132-service-ca\") pod \"console-b6d9ddcc-2fvdr\" (UID: \"dc6274fb-accb-4f81-ae6e-8133c6ed4132\") " pod="openshift-console/console-b6d9ddcc-2fvdr" Jul 12 19:23:12.521395 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:12.521311 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/dc6274fb-accb-4f81-ae6e-8133c6ed4132-oauth-serving-cert\") pod \"console-b6d9ddcc-2fvdr\" (UID: \"dc6274fb-accb-4f81-ae6e-8133c6ed4132\") " pod="openshift-console/console-b6d9ddcc-2fvdr" Jul 12 19:23:12.521395 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:12.521353 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/dc6274fb-accb-4f81-ae6e-8133c6ed4132-service-ca\") pod \"console-b6d9ddcc-2fvdr\" (UID: \"dc6274fb-accb-4f81-ae6e-8133c6ed4132\") " pod="openshift-console/console-b6d9ddcc-2fvdr" Jul 12 19:23:12.521395 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:12.521377 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/dc6274fb-accb-4f81-ae6e-8133c6ed4132-console-serving-cert\") pod \"console-b6d9ddcc-2fvdr\" (UID: \"dc6274fb-accb-4f81-ae6e-8133c6ed4132\") " pod="openshift-console/console-b6d9ddcc-2fvdr" Jul 12 19:23:12.521395 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:12.521397 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdkff\" (UniqueName: \"kubernetes.io/projected/dc6274fb-accb-4f81-ae6e-8133c6ed4132-kube-api-access-sdkff\") pod \"console-b6d9ddcc-2fvdr\" (UID: \"dc6274fb-accb-4f81-ae6e-8133c6ed4132\") " pod="openshift-console/console-b6d9ddcc-2fvdr" Jul 12 19:23:12.521718 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:12.521431 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/dc6274fb-accb-4f81-ae6e-8133c6ed4132-console-oauth-config\") pod \"console-b6d9ddcc-2fvdr\" (UID: \"dc6274fb-accb-4f81-ae6e-8133c6ed4132\") " pod="openshift-console/console-b6d9ddcc-2fvdr" Jul 12 19:23:12.521718 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:12.521462 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/dc6274fb-accb-4f81-ae6e-8133c6ed4132-console-config\") pod \"console-b6d9ddcc-2fvdr\" (UID: \"dc6274fb-accb-4f81-ae6e-8133c6ed4132\") " pod="openshift-console/console-b6d9ddcc-2fvdr" Jul 12 19:23:12.522171 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:12.522146 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/dc6274fb-accb-4f81-ae6e-8133c6ed4132-console-config\") pod \"console-b6d9ddcc-2fvdr\" (UID: \"dc6274fb-accb-4f81-ae6e-8133c6ed4132\") " pod="openshift-console/console-b6d9ddcc-2fvdr" Jul 12 19:23:12.522171 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:12.522156 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/dc6274fb-accb-4f81-ae6e-8133c6ed4132-service-ca\") pod \"console-b6d9ddcc-2fvdr\" (UID: \"dc6274fb-accb-4f81-ae6e-8133c6ed4132\") " pod="openshift-console/console-b6d9ddcc-2fvdr" Jul 12 19:23:12.522360 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:12.522156 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/dc6274fb-accb-4f81-ae6e-8133c6ed4132-oauth-serving-cert\") pod \"console-b6d9ddcc-2fvdr\" (UID: \"dc6274fb-accb-4f81-ae6e-8133c6ed4132\") " pod="openshift-console/console-b6d9ddcc-2fvdr" Jul 12 19:23:12.523826 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:12.523802 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/dc6274fb-accb-4f81-ae6e-8133c6ed4132-console-oauth-config\") pod \"console-b6d9ddcc-2fvdr\" (UID: \"dc6274fb-accb-4f81-ae6e-8133c6ed4132\") " pod="openshift-console/console-b6d9ddcc-2fvdr" Jul 12 19:23:12.523938 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:12.523833 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/dc6274fb-accb-4f81-ae6e-8133c6ed4132-console-serving-cert\") pod \"console-b6d9ddcc-2fvdr\" (UID: \"dc6274fb-accb-4f81-ae6e-8133c6ed4132\") " pod="openshift-console/console-b6d9ddcc-2fvdr" Jul 12 19:23:12.528822 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:12.528797 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdkff\" (UniqueName: \"kubernetes.io/projected/dc6274fb-accb-4f81-ae6e-8133c6ed4132-kube-api-access-sdkff\") pod \"console-b6d9ddcc-2fvdr\" (UID: \"dc6274fb-accb-4f81-ae6e-8133c6ed4132\") " pod="openshift-console/console-b6d9ddcc-2fvdr" Jul 12 19:23:12.664972 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:12.664941 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-b6d9ddcc-2fvdr" Jul 12 19:23:12.801660 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:12.801589 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-b6d9ddcc-2fvdr"] Jul 12 19:23:12.847619 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:12.847591 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:23:12.893830 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:12.893799 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:23:12.957711 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:23:12.957685 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc6274fb_accb_4f81_ae6e_8133c6ed4132.slice/crio-0cf3f0d43f68b6e6e436ee375388a6f4a6fcd23780b9ec4585f4d387e7b6c983 WatchSource:0}: Error finding container 0cf3f0d43f68b6e6e436ee375388a6f4a6fcd23780b9ec4585f4d387e7b6c983: Status 404 returned error can't find the container with id 0cf3f0d43f68b6e6e436ee375388a6f4a6fcd23780b9ec4585f4d387e7b6c983 Jul 12 19:23:13.235725 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:13.235685 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-b6d9ddcc-2fvdr" event={"ID":"dc6274fb-accb-4f81-ae6e-8133c6ed4132","Type":"ContainerStarted","Data":"0cf3f0d43f68b6e6e436ee375388a6f4a6fcd23780b9ec4585f4d387e7b6c983"} Jul 12 19:23:13.237990 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:13.237964 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-qhx89" event={"ID":"a7e56501-c8eb-4615-8bf8-1598f8f9fe39","Type":"ContainerStarted","Data":"bb70cc82e1ad2d2e08bf785128a8a4b53a1c19de518fa13e834ab3ee9cc61756"} Jul 12 19:23:13.254203 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:13.254156 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-qhx89" podStartSLOduration=1.447157198 podStartE2EDuration="11.254143353s" totalImagesPullingTime="9.806986155s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-12 19:23:02 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 19:23:13.253203476 +0000 UTC m=+57.985775967" watchObservedRunningTime="2026-07-12 19:23:13.254143353 +0000 UTC m=+57.986715883" Jul 12 19:23:14.513632 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:14.513588 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:23:14.611387 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:14.611350 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:23:14.683639 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:14.683603 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:23:15.145850 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:15.145812 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:23:15.645753 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:15.645720 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:23:15.703318 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:15.703279 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:23:16.154055 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:16.154029 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-k5h9m" Jul 12 19:23:16.249314 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:16.249273 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-b6d9ddcc-2fvdr" event={"ID":"dc6274fb-accb-4f81-ae6e-8133c6ed4132","Type":"ContainerStarted","Data":"01938b77bfc3c666db2904818b37798c50537c4f9353b74b6bc6c71d5a1d00d7"} Jul 12 19:23:16.259826 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:16.259801 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:23:16.269416 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:16.269373 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-b6d9ddcc-2fvdr" podStartSLOduration=1.493628639 podStartE2EDuration="4.269356092s" totalImagesPullingTime="2.775727453s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 19:23:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 19:23:16.268992391 +0000 UTC m=+61.001564881" watchObservedRunningTime="2026-07-12 19:23:16.269356092 +0000 UTC m=+61.001928581" Jul 12 19:23:16.360126 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:16.360095 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:23:16.410242 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:16.410168 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:23:16.548096 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:16.548062 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:23:16.648984 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:16.648950 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:23:16.699315 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:16.699240 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:23:18.737875 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:18.737844 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:23:18.838675 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:18.838643 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:23:18.890143 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:18.890119 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:23:19.225108 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:19.225079 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-zn4n8" Jul 12 19:23:19.692473 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:19.692433 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:23:19.722626 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:19.722590 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-64785b6675-vcm7j"] Jul 12 19:23:19.725090 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:19.725070 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-64785b6675-vcm7j" Jul 12 19:23:19.727136 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:19.727112 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-7phxv\"" Jul 12 19:23:19.727281 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:19.727259 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Jul 12 19:23:19.728209 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:19.728188 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 12 19:23:19.728299 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:19.728211 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Jul 12 19:23:19.728299 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:19.728265 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 12 19:23:19.728462 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:19.728445 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 12 19:23:19.735689 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:19.735662 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-64785b6675-vcm7j"] Jul 12 19:23:19.755101 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:19.755068 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:23:19.778414 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:19.778379 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c0d417af-5977-4ea9-b7d7-53532cf3f29c-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-64785b6675-vcm7j\" (UID: \"c0d417af-5977-4ea9-b7d7-53532cf3f29c\") " pod="openshift-monitoring/prometheus-operator-64785b6675-vcm7j" Jul 12 19:23:19.778539 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:19.778451 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mtfg6\" (UniqueName: \"kubernetes.io/projected/c0d417af-5977-4ea9-b7d7-53532cf3f29c-kube-api-access-mtfg6\") pod \"prometheus-operator-64785b6675-vcm7j\" (UID: \"c0d417af-5977-4ea9-b7d7-53532cf3f29c\") " pod="openshift-monitoring/prometheus-operator-64785b6675-vcm7j" Jul 12 19:23:19.778539 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:19.778479 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c0d417af-5977-4ea9-b7d7-53532cf3f29c-metrics-client-ca\") pod \"prometheus-operator-64785b6675-vcm7j\" (UID: \"c0d417af-5977-4ea9-b7d7-53532cf3f29c\") " pod="openshift-monitoring/prometheus-operator-64785b6675-vcm7j" Jul 12 19:23:19.778539 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:19.778498 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/c0d417af-5977-4ea9-b7d7-53532cf3f29c-prometheus-operator-tls\") pod \"prometheus-operator-64785b6675-vcm7j\" (UID: \"c0d417af-5977-4ea9-b7d7-53532cf3f29c\") " pod="openshift-monitoring/prometheus-operator-64785b6675-vcm7j" Jul 12 19:23:19.879607 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:19.879572 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/c0d417af-5977-4ea9-b7d7-53532cf3f29c-prometheus-operator-tls\") pod \"prometheus-operator-64785b6675-vcm7j\" (UID: \"c0d417af-5977-4ea9-b7d7-53532cf3f29c\") " pod="openshift-monitoring/prometheus-operator-64785b6675-vcm7j" Jul 12 19:23:19.879804 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:19.879651 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c0d417af-5977-4ea9-b7d7-53532cf3f29c-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-64785b6675-vcm7j\" (UID: \"c0d417af-5977-4ea9-b7d7-53532cf3f29c\") " pod="openshift-monitoring/prometheus-operator-64785b6675-vcm7j" Jul 12 19:23:19.879804 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:19.879706 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtfg6\" (UniqueName: \"kubernetes.io/projected/c0d417af-5977-4ea9-b7d7-53532cf3f29c-kube-api-access-mtfg6\") pod \"prometheus-operator-64785b6675-vcm7j\" (UID: \"c0d417af-5977-4ea9-b7d7-53532cf3f29c\") " pod="openshift-monitoring/prometheus-operator-64785b6675-vcm7j" Jul 12 19:23:19.879804 ip-10-0-139-170 kubenswrapper[2565]: E0712 19:23:19.879714 2565 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found Jul 12 19:23:19.879804 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:19.879733 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c0d417af-5977-4ea9-b7d7-53532cf3f29c-metrics-client-ca\") pod \"prometheus-operator-64785b6675-vcm7j\" (UID: \"c0d417af-5977-4ea9-b7d7-53532cf3f29c\") " pod="openshift-monitoring/prometheus-operator-64785b6675-vcm7j" Jul 12 19:23:19.879804 ip-10-0-139-170 kubenswrapper[2565]: E0712 19:23:19.879775 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c0d417af-5977-4ea9-b7d7-53532cf3f29c-prometheus-operator-tls podName:c0d417af-5977-4ea9-b7d7-53532cf3f29c nodeName:}" failed. No retries permitted until 2026-07-12 19:23:20.379759313 +0000 UTC m=+65.112331781 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/c0d417af-5977-4ea9-b7d7-53532cf3f29c-prometheus-operator-tls") pod "prometheus-operator-64785b6675-vcm7j" (UID: "c0d417af-5977-4ea9-b7d7-53532cf3f29c") : secret "prometheus-operator-tls" not found Jul 12 19:23:19.880439 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:19.880416 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c0d417af-5977-4ea9-b7d7-53532cf3f29c-metrics-client-ca\") pod \"prometheus-operator-64785b6675-vcm7j\" (UID: \"c0d417af-5977-4ea9-b7d7-53532cf3f29c\") " pod="openshift-monitoring/prometheus-operator-64785b6675-vcm7j" Jul 12 19:23:19.881984 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:19.881962 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c0d417af-5977-4ea9-b7d7-53532cf3f29c-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-64785b6675-vcm7j\" (UID: \"c0d417af-5977-4ea9-b7d7-53532cf3f29c\") " pod="openshift-monitoring/prometheus-operator-64785b6675-vcm7j" Jul 12 19:23:19.889719 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:19.889696 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mtfg6\" (UniqueName: \"kubernetes.io/projected/c0d417af-5977-4ea9-b7d7-53532cf3f29c-kube-api-access-mtfg6\") pod \"prometheus-operator-64785b6675-vcm7j\" (UID: \"c0d417af-5977-4ea9-b7d7-53532cf3f29c\") " pod="openshift-monitoring/prometheus-operator-64785b6675-vcm7j" Jul 12 19:23:20.079356 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:20.079270 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:23:20.383162 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:20.383071 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/c0d417af-5977-4ea9-b7d7-53532cf3f29c-prometheus-operator-tls\") pod \"prometheus-operator-64785b6675-vcm7j\" (UID: \"c0d417af-5977-4ea9-b7d7-53532cf3f29c\") " pod="openshift-monitoring/prometheus-operator-64785b6675-vcm7j" Jul 12 19:23:20.385373 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:20.385349 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/c0d417af-5977-4ea9-b7d7-53532cf3f29c-prometheus-operator-tls\") pod \"prometheus-operator-64785b6675-vcm7j\" (UID: \"c0d417af-5977-4ea9-b7d7-53532cf3f29c\") " pod="openshift-monitoring/prometheus-operator-64785b6675-vcm7j" Jul 12 19:23:20.633846 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:20.633765 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-64785b6675-vcm7j" Jul 12 19:23:20.748597 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:20.748569 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-64785b6675-vcm7j"] Jul 12 19:23:20.751812 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:23:20.751784 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0d417af_5977_4ea9_b7d7_53532cf3f29c.slice/crio-d94c105d99883321adfd6b8cbcf7ce9643cfe495fd0f6ecf4cffd4b6bbe11cef WatchSource:0}: Error finding container d94c105d99883321adfd6b8cbcf7ce9643cfe495fd0f6ecf4cffd4b6bbe11cef: Status 404 returned error can't find the container with id d94c105d99883321adfd6b8cbcf7ce9643cfe495fd0f6ecf4cffd4b6bbe11cef Jul 12 19:23:21.266667 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:21.266632 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-64785b6675-vcm7j" event={"ID":"c0d417af-5977-4ea9-b7d7-53532cf3f29c","Type":"ContainerStarted","Data":"d94c105d99883321adfd6b8cbcf7ce9643cfe495fd0f6ecf4cffd4b6bbe11cef"} Jul 12 19:23:21.491561 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:21.491517 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e7722ae2-e293-478e-9ac8-ea376f8d583f-metrics-certs\") pod \"network-metrics-daemon-2xmfq\" (UID: \"e7722ae2-e293-478e-9ac8-ea376f8d583f\") " pod="openshift-multus/network-metrics-daemon-2xmfq" Jul 12 19:23:21.494151 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:21.494133 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 12 19:23:21.504061 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:21.504038 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e7722ae2-e293-478e-9ac8-ea376f8d583f-metrics-certs\") pod \"network-metrics-daemon-2xmfq\" (UID: \"e7722ae2-e293-478e-9ac8-ea376f8d583f\") " pod="openshift-multus/network-metrics-daemon-2xmfq" Jul 12 19:23:21.580463 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:21.580381 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-dcr6j\"" Jul 12 19:23:21.588808 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:21.588779 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2xmfq" Jul 12 19:23:21.592789 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:21.592770 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a7a0abe6-09ba-429e-a7f2-8de5b3f5d620-original-pull-secret\") pod \"global-pull-secret-syncer-288bq\" (UID: \"a7a0abe6-09ba-429e-a7f2-8de5b3f5d620\") " pod="kube-system/global-pull-secret-syncer-288bq" Jul 12 19:23:21.594919 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:21.594897 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 12 19:23:21.606313 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:21.606275 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a7a0abe6-09ba-429e-a7f2-8de5b3f5d620-original-pull-secret\") pod \"global-pull-secret-syncer-288bq\" (UID: \"a7a0abe6-09ba-429e-a7f2-8de5b3f5d620\") " pod="kube-system/global-pull-secret-syncer-288bq" Jul 12 19:23:21.693825 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:21.693781 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bp96h\" (UniqueName: \"kubernetes.io/projected/10c67d44-bdd0-4530-9f5d-fa10238ee66e-kube-api-access-bp96h\") pod \"network-check-target-76gfm\" (UID: \"10c67d44-bdd0-4530-9f5d-fa10238ee66e\") " pod="openshift-network-diagnostics/network-check-target-76gfm" Jul 12 19:23:21.696388 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:21.696362 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 12 19:23:21.706328 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:21.706303 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 12 19:23:21.717332 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:21.717306 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bp96h\" (UniqueName: \"kubernetes.io/projected/10c67d44-bdd0-4530-9f5d-fa10238ee66e-kube-api-access-bp96h\") pod \"network-check-target-76gfm\" (UID: \"10c67d44-bdd0-4530-9f5d-fa10238ee66e\") " pod="openshift-network-diagnostics/network-check-target-76gfm" Jul 12 19:23:21.726603 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:21.726575 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-2xmfq"] Jul 12 19:23:21.762507 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:21.762479 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-cl9vq\"" Jul 12 19:23:21.770289 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:21.770264 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-76gfm" Jul 12 19:23:21.871644 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:21.871567 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-288bq" Jul 12 19:23:21.995994 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:23:21.995960 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7722ae2_e293_478e_9ac8_ea376f8d583f.slice/crio-1ffa83cdba44832dd5383919c9151cf0ef39c0729ee3db03ef05315eaf978a37 WatchSource:0}: Error finding container 1ffa83cdba44832dd5383919c9151cf0ef39c0729ee3db03ef05315eaf978a37: Status 404 returned error can't find the container with id 1ffa83cdba44832dd5383919c9151cf0ef39c0729ee3db03ef05315eaf978a37 Jul 12 19:23:22.135247 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:22.135174 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-76gfm"] Jul 12 19:23:22.139270 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:23:22.139245 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10c67d44_bdd0_4530_9f5d_fa10238ee66e.slice/crio-38412ff012bfd9726cf772ebfe73919b3f153ca4bdf3a3c906c147c4cba49957 WatchSource:0}: Error finding container 38412ff012bfd9726cf772ebfe73919b3f153ca4bdf3a3c906c147c4cba49957: Status 404 returned error can't find the container with id 38412ff012bfd9726cf772ebfe73919b3f153ca4bdf3a3c906c147c4cba49957 Jul 12 19:23:22.151101 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:22.151073 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-288bq"] Jul 12 19:23:22.156725 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:23:22.156695 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7a0abe6_09ba_429e_a7f2_8de5b3f5d620.slice/crio-2dc9d84fd871c58085c0ee09374550259b174b34222e55ae317aa7635777c0fb WatchSource:0}: Error finding container 2dc9d84fd871c58085c0ee09374550259b174b34222e55ae317aa7635777c0fb: Status 404 returned error can't find the container with id 2dc9d84fd871c58085c0ee09374550259b174b34222e55ae317aa7635777c0fb Jul 12 19:23:22.247472 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:22.247430 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7b5759fdf9-9hlqp"] Jul 12 19:23:22.250605 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:22.250570 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7b5759fdf9-9hlqp" Jul 12 19:23:22.259504 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:22.259270 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7b5759fdf9-9hlqp"] Jul 12 19:23:22.261033 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:22.261004 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 12 19:23:22.271135 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:22.271094 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-288bq" event={"ID":"a7a0abe6-09ba-429e-a7f2-8de5b3f5d620","Type":"ContainerStarted","Data":"2dc9d84fd871c58085c0ee09374550259b174b34222e55ae317aa7635777c0fb"} Jul 12 19:23:22.272788 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:22.272761 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-64785b6675-vcm7j" event={"ID":"c0d417af-5977-4ea9-b7d7-53532cf3f29c","Type":"ContainerStarted","Data":"b233483270b7d6b9a98f65b2ab28a963003b68e166eaa8f13b8ae1bfea7cb45c"} Jul 12 19:23:22.272992 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:22.272809 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-64785b6675-vcm7j" event={"ID":"c0d417af-5977-4ea9-b7d7-53532cf3f29c","Type":"ContainerStarted","Data":"af5ee566b2ae4de8664a3acaa939c3b0ad50456703550b3eb50d11c964d24b07"} Jul 12 19:23:22.273199 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:22.273162 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:23:22.273933 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:22.273914 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-76gfm" event={"ID":"10c67d44-bdd0-4530-9f5d-fa10238ee66e","Type":"ContainerStarted","Data":"38412ff012bfd9726cf772ebfe73919b3f153ca4bdf3a3c906c147c4cba49957"} Jul 12 19:23:22.274862 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:22.274842 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-2xmfq" event={"ID":"e7722ae2-e293-478e-9ac8-ea376f8d583f","Type":"ContainerStarted","Data":"1ffa83cdba44832dd5383919c9151cf0ef39c0729ee3db03ef05315eaf978a37"} Jul 12 19:23:22.293810 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:22.293753 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-64785b6675-vcm7j" podStartSLOduration=2.025214391 podStartE2EDuration="3.293737878s" totalImagesPullingTime="1.268523487s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 19:23:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 19:23:22.293406411 +0000 UTC m=+67.025978901" watchObservedRunningTime="2026-07-12 19:23:22.293737878 +0000 UTC m=+67.026310367" Jul 12 19:23:22.297705 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:22.297681 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/abed6323-d374-454c-bbee-80e9403ff351-service-ca\") pod \"console-7b5759fdf9-9hlqp\" (UID: \"abed6323-d374-454c-bbee-80e9403ff351\") " pod="openshift-console/console-7b5759fdf9-9hlqp" Jul 12 19:23:22.297797 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:22.297729 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/abed6323-d374-454c-bbee-80e9403ff351-console-oauth-config\") pod \"console-7b5759fdf9-9hlqp\" (UID: \"abed6323-d374-454c-bbee-80e9403ff351\") " pod="openshift-console/console-7b5759fdf9-9hlqp" Jul 12 19:23:22.297853 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:22.297795 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/abed6323-d374-454c-bbee-80e9403ff351-trusted-ca-bundle\") pod \"console-7b5759fdf9-9hlqp\" (UID: \"abed6323-d374-454c-bbee-80e9403ff351\") " pod="openshift-console/console-7b5759fdf9-9hlqp" Jul 12 19:23:22.297853 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:22.297839 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rg4tr\" (UniqueName: \"kubernetes.io/projected/abed6323-d374-454c-bbee-80e9403ff351-kube-api-access-rg4tr\") pod \"console-7b5759fdf9-9hlqp\" (UID: \"abed6323-d374-454c-bbee-80e9403ff351\") " pod="openshift-console/console-7b5759fdf9-9hlqp" Jul 12 19:23:22.297973 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:22.297954 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/abed6323-d374-454c-bbee-80e9403ff351-console-serving-cert\") pod \"console-7b5759fdf9-9hlqp\" (UID: \"abed6323-d374-454c-bbee-80e9403ff351\") " pod="openshift-console/console-7b5759fdf9-9hlqp" Jul 12 19:23:22.298027 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:22.298010 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/abed6323-d374-454c-bbee-80e9403ff351-console-config\") pod \"console-7b5759fdf9-9hlqp\" (UID: \"abed6323-d374-454c-bbee-80e9403ff351\") " pod="openshift-console/console-7b5759fdf9-9hlqp" Jul 12 19:23:22.298061 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:22.298046 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/abed6323-d374-454c-bbee-80e9403ff351-oauth-serving-cert\") pod \"console-7b5759fdf9-9hlqp\" (UID: \"abed6323-d374-454c-bbee-80e9403ff351\") " pod="openshift-console/console-7b5759fdf9-9hlqp" Jul 12 19:23:22.327191 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:22.327156 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:23:22.399374 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:22.399342 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/abed6323-d374-454c-bbee-80e9403ff351-trusted-ca-bundle\") pod \"console-7b5759fdf9-9hlqp\" (UID: \"abed6323-d374-454c-bbee-80e9403ff351\") " pod="openshift-console/console-7b5759fdf9-9hlqp" Jul 12 19:23:22.399526 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:22.399382 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rg4tr\" (UniqueName: \"kubernetes.io/projected/abed6323-d374-454c-bbee-80e9403ff351-kube-api-access-rg4tr\") pod \"console-7b5759fdf9-9hlqp\" (UID: \"abed6323-d374-454c-bbee-80e9403ff351\") " pod="openshift-console/console-7b5759fdf9-9hlqp" Jul 12 19:23:22.399526 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:22.399405 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/abed6323-d374-454c-bbee-80e9403ff351-console-serving-cert\") pod \"console-7b5759fdf9-9hlqp\" (UID: \"abed6323-d374-454c-bbee-80e9403ff351\") " pod="openshift-console/console-7b5759fdf9-9hlqp" Jul 12 19:23:22.399526 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:22.399425 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/abed6323-d374-454c-bbee-80e9403ff351-console-config\") pod \"console-7b5759fdf9-9hlqp\" (UID: \"abed6323-d374-454c-bbee-80e9403ff351\") " pod="openshift-console/console-7b5759fdf9-9hlqp" Jul 12 19:23:22.399526 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:22.399441 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/abed6323-d374-454c-bbee-80e9403ff351-oauth-serving-cert\") pod \"console-7b5759fdf9-9hlqp\" (UID: \"abed6323-d374-454c-bbee-80e9403ff351\") " pod="openshift-console/console-7b5759fdf9-9hlqp" Jul 12 19:23:22.399526 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:22.399468 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/abed6323-d374-454c-bbee-80e9403ff351-service-ca\") pod \"console-7b5759fdf9-9hlqp\" (UID: \"abed6323-d374-454c-bbee-80e9403ff351\") " pod="openshift-console/console-7b5759fdf9-9hlqp" Jul 12 19:23:22.399526 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:22.399486 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/abed6323-d374-454c-bbee-80e9403ff351-console-oauth-config\") pod \"console-7b5759fdf9-9hlqp\" (UID: \"abed6323-d374-454c-bbee-80e9403ff351\") " pod="openshift-console/console-7b5759fdf9-9hlqp" Jul 12 19:23:22.400333 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:22.400292 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/abed6323-d374-454c-bbee-80e9403ff351-service-ca\") pod \"console-7b5759fdf9-9hlqp\" (UID: \"abed6323-d374-454c-bbee-80e9403ff351\") " pod="openshift-console/console-7b5759fdf9-9hlqp" Jul 12 19:23:22.400436 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:22.400333 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/abed6323-d374-454c-bbee-80e9403ff351-console-config\") pod \"console-7b5759fdf9-9hlqp\" (UID: \"abed6323-d374-454c-bbee-80e9403ff351\") " pod="openshift-console/console-7b5759fdf9-9hlqp" Jul 12 19:23:22.400436 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:22.400359 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/abed6323-d374-454c-bbee-80e9403ff351-oauth-serving-cert\") pod \"console-7b5759fdf9-9hlqp\" (UID: \"abed6323-d374-454c-bbee-80e9403ff351\") " pod="openshift-console/console-7b5759fdf9-9hlqp" Jul 12 19:23:22.400557 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:22.400464 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/abed6323-d374-454c-bbee-80e9403ff351-trusted-ca-bundle\") pod \"console-7b5759fdf9-9hlqp\" (UID: \"abed6323-d374-454c-bbee-80e9403ff351\") " pod="openshift-console/console-7b5759fdf9-9hlqp" Jul 12 19:23:22.401959 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:22.401942 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/abed6323-d374-454c-bbee-80e9403ff351-console-oauth-config\") pod \"console-7b5759fdf9-9hlqp\" (UID: \"abed6323-d374-454c-bbee-80e9403ff351\") " pod="openshift-console/console-7b5759fdf9-9hlqp" Jul 12 19:23:22.402178 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:22.402163 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/abed6323-d374-454c-bbee-80e9403ff351-console-serving-cert\") pod \"console-7b5759fdf9-9hlqp\" (UID: \"abed6323-d374-454c-bbee-80e9403ff351\") " pod="openshift-console/console-7b5759fdf9-9hlqp" Jul 12 19:23:22.406394 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:22.406371 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rg4tr\" (UniqueName: \"kubernetes.io/projected/abed6323-d374-454c-bbee-80e9403ff351-kube-api-access-rg4tr\") pod \"console-7b5759fdf9-9hlqp\" (UID: \"abed6323-d374-454c-bbee-80e9403ff351\") " pod="openshift-console/console-7b5759fdf9-9hlqp" Jul 12 19:23:22.433611 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:22.433576 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:23:22.562868 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:22.562836 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7b5759fdf9-9hlqp" Jul 12 19:23:22.665456 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:22.665423 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-b6d9ddcc-2fvdr" Jul 12 19:23:22.666778 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:22.666712 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-b6d9ddcc-2fvdr" Jul 12 19:23:22.673132 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:22.673108 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-b6d9ddcc-2fvdr" Jul 12 19:23:22.720392 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:22.720364 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7b5759fdf9-9hlqp"] Jul 12 19:23:22.969428 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:23:22.969351 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabed6323_d374_454c_bbee_80e9403ff351.slice/crio-e3fa1ad7b5a0401e1b46f7a533f1f635f2c562028ea4344fffa6958008f67475 WatchSource:0}: Error finding container e3fa1ad7b5a0401e1b46f7a533f1f635f2c562028ea4344fffa6958008f67475: Status 404 returned error can't find the container with id e3fa1ad7b5a0401e1b46f7a533f1f635f2c562028ea4344fffa6958008f67475 Jul 12 19:23:23.282203 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:23.282120 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-2xmfq" event={"ID":"e7722ae2-e293-478e-9ac8-ea376f8d583f","Type":"ContainerStarted","Data":"435f25bb91665a816e24a19fde84e7b286bc63a061d4095daa7c85c7652dd6ed"} Jul 12 19:23:23.282203 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:23.282186 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-2xmfq" event={"ID":"e7722ae2-e293-478e-9ac8-ea376f8d583f","Type":"ContainerStarted","Data":"954c442500fc7172ebd01142ea9df51c2fea18e6480f02a204bc2792efebc4b7"} Jul 12 19:23:23.284245 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:23.284164 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7b5759fdf9-9hlqp" event={"ID":"abed6323-d374-454c-bbee-80e9403ff351","Type":"ContainerStarted","Data":"afafeb2d77b14e69085ff278a2120d81cc433666f3376e6d1c55e16b4d1cd1e0"} Jul 12 19:23:23.284245 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:23.284212 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7b5759fdf9-9hlqp" event={"ID":"abed6323-d374-454c-bbee-80e9403ff351","Type":"ContainerStarted","Data":"e3fa1ad7b5a0401e1b46f7a533f1f635f2c562028ea4344fffa6958008f67475"} Jul 12 19:23:23.289224 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:23.289195 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-b6d9ddcc-2fvdr" Jul 12 19:23:23.296498 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:23.296449 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-2xmfq" podStartSLOduration=67.294361977 podStartE2EDuration="1m8.296431417s" totalImagesPullingTime="1.00206944s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 19:22:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 19:23:23.29583384 +0000 UTC m=+68.028406343" watchObservedRunningTime="2026-07-12 19:23:23.296431417 +0000 UTC m=+68.029003909" Jul 12 19:23:23.327145 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:23.327092 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7b5759fdf9-9hlqp" podStartSLOduration=1.327070451 podStartE2EDuration="1.327070451s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 19:23:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 19:23:23.3260937 +0000 UTC m=+68.058666205" watchObservedRunningTime="2026-07-12 19:23:23.327070451 +0000 UTC m=+68.059642943" Jul 12 19:23:23.382848 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:23.382799 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:23:23.458513 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:23.458476 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:23:23.530612 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:23.530576 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:23:23.583012 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:23.582874 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:23:23.804630 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:23.804594 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:23:24.259417 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:24.259363 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:23:24.315012 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:24.314977 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:23:25.164490 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:25.164418 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:23:26.222011 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:26.221973 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:23:27.300837 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:27.300808 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-76gfm" event={"ID":"10c67d44-bdd0-4530-9f5d-fa10238ee66e","Type":"ContainerStarted","Data":"70a692418c23eac57cd533daaa16548bd1fd6a2badd4bb35aa77c484f8110d91"} Jul 12 19:23:27.301292 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:27.300905 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-76gfm" Jul 12 19:23:27.302269 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:27.302248 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-288bq" event={"ID":"a7a0abe6-09ba-429e-a7f2-8de5b3f5d620","Type":"ContainerStarted","Data":"3d8bd6db6b30bd2e92c323a99ba52a14e9f6b05cd9f89f872e8d6d6277fb7190"} Jul 12 19:23:27.314538 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:27.314487 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-76gfm" podStartSLOduration=67.821937284 podStartE2EDuration="1m12.314474794s" totalImagesPullingTime="4.49253751s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 19:22:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 19:23:27.314010879 +0000 UTC m=+72.046583369" watchObservedRunningTime="2026-07-12 19:23:27.314474794 +0000 UTC m=+72.047047281" Jul 12 19:23:27.329618 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:27.329579 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-288bq" podStartSLOduration=66.851097965 podStartE2EDuration="1m11.329567446s" totalImagesPullingTime="4.478469481s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-12 19:22:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 19:23:27.328364222 +0000 UTC m=+72.060936720" watchObservedRunningTime="2026-07-12 19:23:27.329567446 +0000 UTC m=+72.062139936" Jul 12 19:23:27.833508 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:27.833468 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:23:27.876520 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:27.876484 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-785778ff74-vdl5x"] Jul 12 19:23:27.879058 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:27.879035 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-785778ff74-vdl5x" Jul 12 19:23:27.881129 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:27.881095 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Jul 12 19:23:27.881253 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:27.881193 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-nf8w9\"" Jul 12 19:23:27.882178 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:27.882157 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Jul 12 19:23:27.882295 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:27.882158 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Jul 12 19:23:27.885019 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:27.884991 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-r5rxf"] Jul 12 19:23:27.887141 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:27.887118 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-r5rxf" Jul 12 19:23:27.889085 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:27.889065 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 12 19:23:27.889085 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:27.889083 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-hchhq\"" Jul 12 19:23:27.889216 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:27.889179 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 12 19:23:27.889377 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:27.889360 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 12 19:23:27.891631 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:27.891606 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-785778ff74-vdl5x"] Jul 12 19:23:27.916318 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:27.916276 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:23:27.946876 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:27.946841 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2b24f43c-e196-4d40-a20b-469489818be0-metrics-client-ca\") pod \"kube-state-metrics-785778ff74-vdl5x\" (UID: \"2b24f43c-e196-4d40-a20b-469489818be0\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-vdl5x" Jul 12 19:23:27.946876 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:27.946878 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1d52b8f7-a1db-4ab3-bd17-7462950f76ae-sys\") pod \"node-exporter-r5rxf\" (UID: \"1d52b8f7-a1db-4ab3-bd17-7462950f76ae\") " pod="openshift-monitoring/node-exporter-r5rxf" Jul 12 19:23:27.947113 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:27.946922 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1d52b8f7-a1db-4ab3-bd17-7462950f76ae-metrics-client-ca\") pod \"node-exporter-r5rxf\" (UID: \"1d52b8f7-a1db-4ab3-bd17-7462950f76ae\") " pod="openshift-monitoring/node-exporter-r5rxf" Jul 12 19:23:27.947113 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:27.946939 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1d52b8f7-a1db-4ab3-bd17-7462950f76ae-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-r5rxf\" (UID: \"1d52b8f7-a1db-4ab3-bd17-7462950f76ae\") " pod="openshift-monitoring/node-exporter-r5rxf" Jul 12 19:23:27.947113 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:27.946975 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/1d52b8f7-a1db-4ab3-bd17-7462950f76ae-node-exporter-accelerators-collector-config\") pod \"node-exporter-r5rxf\" (UID: \"1d52b8f7-a1db-4ab3-bd17-7462950f76ae\") " pod="openshift-monitoring/node-exporter-r5rxf" Jul 12 19:23:27.947113 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:27.947011 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/1d52b8f7-a1db-4ab3-bd17-7462950f76ae-node-exporter-textfile\") pod \"node-exporter-r5rxf\" (UID: \"1d52b8f7-a1db-4ab3-bd17-7462950f76ae\") " pod="openshift-monitoring/node-exporter-r5rxf" Jul 12 19:23:27.947113 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:27.947032 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/1d52b8f7-a1db-4ab3-bd17-7462950f76ae-root\") pod \"node-exporter-r5rxf\" (UID: \"1d52b8f7-a1db-4ab3-bd17-7462950f76ae\") " pod="openshift-monitoring/node-exporter-r5rxf" Jul 12 19:23:27.947113 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:27.947072 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/2b24f43c-e196-4d40-a20b-469489818be0-volume-directive-shadow\") pod \"kube-state-metrics-785778ff74-vdl5x\" (UID: \"2b24f43c-e196-4d40-a20b-469489818be0\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-vdl5x" Jul 12 19:23:27.947113 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:27.947091 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2b24f43c-e196-4d40-a20b-469489818be0-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-785778ff74-vdl5x\" (UID: \"2b24f43c-e196-4d40-a20b-469489818be0\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-vdl5x" Jul 12 19:23:27.947366 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:27.947146 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/1d52b8f7-a1db-4ab3-bd17-7462950f76ae-node-exporter-wtmp\") pod \"node-exporter-r5rxf\" (UID: \"1d52b8f7-a1db-4ab3-bd17-7462950f76ae\") " pod="openshift-monitoring/node-exporter-r5rxf" Jul 12 19:23:27.947366 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:27.947168 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/2b24f43c-e196-4d40-a20b-469489818be0-kube-state-metrics-tls\") pod \"kube-state-metrics-785778ff74-vdl5x\" (UID: \"2b24f43c-e196-4d40-a20b-469489818be0\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-vdl5x" Jul 12 19:23:27.947366 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:27.947196 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ts7fx\" (UniqueName: \"kubernetes.io/projected/2b24f43c-e196-4d40-a20b-469489818be0-kube-api-access-ts7fx\") pod \"kube-state-metrics-785778ff74-vdl5x\" (UID: \"2b24f43c-e196-4d40-a20b-469489818be0\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-vdl5x" Jul 12 19:23:27.947366 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:27.947212 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/1d52b8f7-a1db-4ab3-bd17-7462950f76ae-node-exporter-tls\") pod \"node-exporter-r5rxf\" (UID: \"1d52b8f7-a1db-4ab3-bd17-7462950f76ae\") " pod="openshift-monitoring/node-exporter-r5rxf" Jul 12 19:23:27.947366 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:27.947227 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6hj6\" (UniqueName: \"kubernetes.io/projected/1d52b8f7-a1db-4ab3-bd17-7462950f76ae-kube-api-access-w6hj6\") pod \"node-exporter-r5rxf\" (UID: \"1d52b8f7-a1db-4ab3-bd17-7462950f76ae\") " pod="openshift-monitoring/node-exporter-r5rxf" Jul 12 19:23:27.947366 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:27.947255 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/2b24f43c-e196-4d40-a20b-469489818be0-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-785778ff74-vdl5x\" (UID: \"2b24f43c-e196-4d40-a20b-469489818be0\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-vdl5x" Jul 12 19:23:28.048248 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:28.048217 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ts7fx\" (UniqueName: \"kubernetes.io/projected/2b24f43c-e196-4d40-a20b-469489818be0-kube-api-access-ts7fx\") pod \"kube-state-metrics-785778ff74-vdl5x\" (UID: \"2b24f43c-e196-4d40-a20b-469489818be0\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-vdl5x" Jul 12 19:23:28.048248 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:28.048252 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/1d52b8f7-a1db-4ab3-bd17-7462950f76ae-node-exporter-tls\") pod \"node-exporter-r5rxf\" (UID: \"1d52b8f7-a1db-4ab3-bd17-7462950f76ae\") " pod="openshift-monitoring/node-exporter-r5rxf" Jul 12 19:23:28.048580 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:28.048274 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6hj6\" (UniqueName: \"kubernetes.io/projected/1d52b8f7-a1db-4ab3-bd17-7462950f76ae-kube-api-access-w6hj6\") pod \"node-exporter-r5rxf\" (UID: \"1d52b8f7-a1db-4ab3-bd17-7462950f76ae\") " pod="openshift-monitoring/node-exporter-r5rxf" Jul 12 19:23:28.048580 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:28.048303 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/2b24f43c-e196-4d40-a20b-469489818be0-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-785778ff74-vdl5x\" (UID: \"2b24f43c-e196-4d40-a20b-469489818be0\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-vdl5x" Jul 12 19:23:28.048580 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:28.048351 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2b24f43c-e196-4d40-a20b-469489818be0-metrics-client-ca\") pod \"kube-state-metrics-785778ff74-vdl5x\" (UID: \"2b24f43c-e196-4d40-a20b-469489818be0\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-vdl5x" Jul 12 19:23:28.048580 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:28.048403 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1d52b8f7-a1db-4ab3-bd17-7462950f76ae-sys\") pod \"node-exporter-r5rxf\" (UID: \"1d52b8f7-a1db-4ab3-bd17-7462950f76ae\") " pod="openshift-monitoring/node-exporter-r5rxf" Jul 12 19:23:28.048580 ip-10-0-139-170 kubenswrapper[2565]: E0712 19:23:28.048406 2565 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 12 19:23:28.048580 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:28.048429 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1d52b8f7-a1db-4ab3-bd17-7462950f76ae-metrics-client-ca\") pod \"node-exporter-r5rxf\" (UID: \"1d52b8f7-a1db-4ab3-bd17-7462950f76ae\") " pod="openshift-monitoring/node-exporter-r5rxf" Jul 12 19:23:28.048580 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:28.048455 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1d52b8f7-a1db-4ab3-bd17-7462950f76ae-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-r5rxf\" (UID: \"1d52b8f7-a1db-4ab3-bd17-7462950f76ae\") " pod="openshift-monitoring/node-exporter-r5rxf" Jul 12 19:23:28.048580 ip-10-0-139-170 kubenswrapper[2565]: E0712 19:23:28.048485 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1d52b8f7-a1db-4ab3-bd17-7462950f76ae-node-exporter-tls podName:1d52b8f7-a1db-4ab3-bd17-7462950f76ae nodeName:}" failed. No retries permitted until 2026-07-12 19:23:28.548462776 +0000 UTC m=+73.281035245 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/1d52b8f7-a1db-4ab3-bd17-7462950f76ae-node-exporter-tls") pod "node-exporter-r5rxf" (UID: "1d52b8f7-a1db-4ab3-bd17-7462950f76ae") : secret "node-exporter-tls" not found Jul 12 19:23:28.048580 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:28.048517 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/1d52b8f7-a1db-4ab3-bd17-7462950f76ae-node-exporter-accelerators-collector-config\") pod \"node-exporter-r5rxf\" (UID: \"1d52b8f7-a1db-4ab3-bd17-7462950f76ae\") " pod="openshift-monitoring/node-exporter-r5rxf" Jul 12 19:23:28.048580 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:28.048536 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1d52b8f7-a1db-4ab3-bd17-7462950f76ae-sys\") pod \"node-exporter-r5rxf\" (UID: \"1d52b8f7-a1db-4ab3-bd17-7462950f76ae\") " pod="openshift-monitoring/node-exporter-r5rxf" Jul 12 19:23:28.048580 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:28.048549 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/1d52b8f7-a1db-4ab3-bd17-7462950f76ae-node-exporter-textfile\") pod \"node-exporter-r5rxf\" (UID: \"1d52b8f7-a1db-4ab3-bd17-7462950f76ae\") " pod="openshift-monitoring/node-exporter-r5rxf" Jul 12 19:23:28.049199 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:28.048591 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/1d52b8f7-a1db-4ab3-bd17-7462950f76ae-root\") pod \"node-exporter-r5rxf\" (UID: \"1d52b8f7-a1db-4ab3-bd17-7462950f76ae\") " pod="openshift-monitoring/node-exporter-r5rxf" Jul 12 19:23:28.049199 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:28.048671 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/2b24f43c-e196-4d40-a20b-469489818be0-volume-directive-shadow\") pod \"kube-state-metrics-785778ff74-vdl5x\" (UID: \"2b24f43c-e196-4d40-a20b-469489818be0\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-vdl5x" Jul 12 19:23:28.049199 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:28.048702 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2b24f43c-e196-4d40-a20b-469489818be0-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-785778ff74-vdl5x\" (UID: \"2b24f43c-e196-4d40-a20b-469489818be0\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-vdl5x" Jul 12 19:23:28.049199 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:28.048789 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/1d52b8f7-a1db-4ab3-bd17-7462950f76ae-node-exporter-wtmp\") pod \"node-exporter-r5rxf\" (UID: \"1d52b8f7-a1db-4ab3-bd17-7462950f76ae\") " pod="openshift-monitoring/node-exporter-r5rxf" Jul 12 19:23:28.049199 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:28.048799 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/1d52b8f7-a1db-4ab3-bd17-7462950f76ae-root\") pod \"node-exporter-r5rxf\" (UID: \"1d52b8f7-a1db-4ab3-bd17-7462950f76ae\") " pod="openshift-monitoring/node-exporter-r5rxf" Jul 12 19:23:28.049199 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:28.048823 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/2b24f43c-e196-4d40-a20b-469489818be0-kube-state-metrics-tls\") pod \"kube-state-metrics-785778ff74-vdl5x\" (UID: \"2b24f43c-e196-4d40-a20b-469489818be0\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-vdl5x" Jul 12 19:23:28.049199 ip-10-0-139-170 kubenswrapper[2565]: E0712 19:23:28.048975 2565 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found Jul 12 19:23:28.049199 ip-10-0-139-170 kubenswrapper[2565]: E0712 19:23:28.049025 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2b24f43c-e196-4d40-a20b-469489818be0-kube-state-metrics-tls podName:2b24f43c-e196-4d40-a20b-469489818be0 nodeName:}" failed. No retries permitted until 2026-07-12 19:23:28.549008496 +0000 UTC m=+73.281580978 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/2b24f43c-e196-4d40-a20b-469489818be0-kube-state-metrics-tls") pod "kube-state-metrics-785778ff74-vdl5x" (UID: "2b24f43c-e196-4d40-a20b-469489818be0") : secret "kube-state-metrics-tls" not found Jul 12 19:23:28.049199 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:28.049143 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/1d52b8f7-a1db-4ab3-bd17-7462950f76ae-node-exporter-textfile\") pod \"node-exporter-r5rxf\" (UID: \"1d52b8f7-a1db-4ab3-bd17-7462950f76ae\") " pod="openshift-monitoring/node-exporter-r5rxf" Jul 12 19:23:28.049199 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:28.049150 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/2b24f43c-e196-4d40-a20b-469489818be0-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-785778ff74-vdl5x\" (UID: \"2b24f43c-e196-4d40-a20b-469489818be0\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-vdl5x" Jul 12 19:23:28.049199 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:28.049150 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2b24f43c-e196-4d40-a20b-469489818be0-metrics-client-ca\") pod \"kube-state-metrics-785778ff74-vdl5x\" (UID: \"2b24f43c-e196-4d40-a20b-469489818be0\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-vdl5x" Jul 12 19:23:28.049199 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:28.049152 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/1d52b8f7-a1db-4ab3-bd17-7462950f76ae-node-exporter-wtmp\") pod \"node-exporter-r5rxf\" (UID: \"1d52b8f7-a1db-4ab3-bd17-7462950f76ae\") " pod="openshift-monitoring/node-exporter-r5rxf" Jul 12 19:23:28.049199 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:28.049177 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1d52b8f7-a1db-4ab3-bd17-7462950f76ae-metrics-client-ca\") pod \"node-exporter-r5rxf\" (UID: \"1d52b8f7-a1db-4ab3-bd17-7462950f76ae\") " pod="openshift-monitoring/node-exporter-r5rxf" Jul 12 19:23:28.049678 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:28.049177 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/1d52b8f7-a1db-4ab3-bd17-7462950f76ae-node-exporter-accelerators-collector-config\") pod \"node-exporter-r5rxf\" (UID: \"1d52b8f7-a1db-4ab3-bd17-7462950f76ae\") " pod="openshift-monitoring/node-exporter-r5rxf" Jul 12 19:23:28.049678 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:28.049391 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/2b24f43c-e196-4d40-a20b-469489818be0-volume-directive-shadow\") pod \"kube-state-metrics-785778ff74-vdl5x\" (UID: \"2b24f43c-e196-4d40-a20b-469489818be0\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-vdl5x" Jul 12 19:23:28.051117 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:28.051100 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1d52b8f7-a1db-4ab3-bd17-7462950f76ae-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-r5rxf\" (UID: \"1d52b8f7-a1db-4ab3-bd17-7462950f76ae\") " pod="openshift-monitoring/node-exporter-r5rxf" Jul 12 19:23:28.051256 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:28.051239 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2b24f43c-e196-4d40-a20b-469489818be0-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-785778ff74-vdl5x\" (UID: \"2b24f43c-e196-4d40-a20b-469489818be0\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-vdl5x" Jul 12 19:23:28.061902 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:28.061859 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6hj6\" (UniqueName: \"kubernetes.io/projected/1d52b8f7-a1db-4ab3-bd17-7462950f76ae-kube-api-access-w6hj6\") pod \"node-exporter-r5rxf\" (UID: \"1d52b8f7-a1db-4ab3-bd17-7462950f76ae\") " pod="openshift-monitoring/node-exporter-r5rxf" Jul 12 19:23:28.062032 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:28.061956 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ts7fx\" (UniqueName: \"kubernetes.io/projected/2b24f43c-e196-4d40-a20b-469489818be0-kube-api-access-ts7fx\") pod \"kube-state-metrics-785778ff74-vdl5x\" (UID: \"2b24f43c-e196-4d40-a20b-469489818be0\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-vdl5x" Jul 12 19:23:28.095444 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:28.095342 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:23:28.195033 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:28.194994 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:23:28.281552 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:28.281520 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:23:28.388682 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:28.388588 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:23:28.488972 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:28.488925 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:23:28.552825 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:28.552793 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/2b24f43c-e196-4d40-a20b-469489818be0-kube-state-metrics-tls\") pod \"kube-state-metrics-785778ff74-vdl5x\" (UID: \"2b24f43c-e196-4d40-a20b-469489818be0\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-vdl5x" Jul 12 19:23:28.553033 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:28.552833 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/1d52b8f7-a1db-4ab3-bd17-7462950f76ae-node-exporter-tls\") pod \"node-exporter-r5rxf\" (UID: \"1d52b8f7-a1db-4ab3-bd17-7462950f76ae\") " pod="openshift-monitoring/node-exporter-r5rxf" Jul 12 19:23:28.555120 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:28.555092 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/1d52b8f7-a1db-4ab3-bd17-7462950f76ae-node-exporter-tls\") pod \"node-exporter-r5rxf\" (UID: \"1d52b8f7-a1db-4ab3-bd17-7462950f76ae\") " pod="openshift-monitoring/node-exporter-r5rxf" Jul 12 19:23:28.555279 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:28.555248 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/2b24f43c-e196-4d40-a20b-469489818be0-kube-state-metrics-tls\") pod \"kube-state-metrics-785778ff74-vdl5x\" (UID: \"2b24f43c-e196-4d40-a20b-469489818be0\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-vdl5x" Jul 12 19:23:28.789088 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:28.788991 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-785778ff74-vdl5x" Jul 12 19:23:28.796751 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:28.796725 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-r5rxf" Jul 12 19:23:28.810834 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:23:28.810805 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d52b8f7_a1db_4ab3_bd17_7462950f76ae.slice/crio-d6aad2e5be648511158549cde995101b6a430c4deb461d140e153005bbe0814d WatchSource:0}: Error finding container d6aad2e5be648511158549cde995101b6a430c4deb461d140e153005bbe0814d: Status 404 returned error can't find the container with id d6aad2e5be648511158549cde995101b6a430c4deb461d140e153005bbe0814d Jul 12 19:23:28.830518 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:28.830487 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:23:28.914984 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:28.914954 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:23:28.927820 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:28.927550 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-785778ff74-vdl5x"] Jul 12 19:23:28.930467 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:23:28.930440 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b24f43c_e196_4d40_a20b_469489818be0.slice/crio-e6d60f8aedb4ebf12b52fe5668fd346d70201abe2072e0e5758b8645f6f8ea29 WatchSource:0}: Error finding container e6d60f8aedb4ebf12b52fe5668fd346d70201abe2072e0e5758b8645f6f8ea29: Status 404 returned error can't find the container with id e6d60f8aedb4ebf12b52fe5668fd346d70201abe2072e0e5758b8645f6f8ea29 Jul 12 19:23:29.280167 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:29.280127 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:23:29.310128 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:29.310092 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-r5rxf" event={"ID":"1d52b8f7-a1db-4ab3-bd17-7462950f76ae","Type":"ContainerStarted","Data":"d6aad2e5be648511158549cde995101b6a430c4deb461d140e153005bbe0814d"} Jul 12 19:23:29.311075 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:29.311057 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-vdl5x" event={"ID":"2b24f43c-e196-4d40-a20b-469489818be0","Type":"ContainerStarted","Data":"e6d60f8aedb4ebf12b52fe5668fd346d70201abe2072e0e5758b8645f6f8ea29"} Jul 12 19:23:30.053119 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:30.053073 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:23:30.316134 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:30.316101 2565 generic.go:356] "Generic (PLEG): container finished" podID="1d52b8f7-a1db-4ab3-bd17-7462950f76ae" containerID="8ec07dd39ba84b03b567ee1cf925869215fd2fa9309205b09c7988edba46b397" exitCode=0 Jul 12 19:23:30.316286 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:30.316187 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-r5rxf" event={"ID":"1d52b8f7-a1db-4ab3-bd17-7462950f76ae","Type":"ContainerDied","Data":"8ec07dd39ba84b03b567ee1cf925869215fd2fa9309205b09c7988edba46b397"} Jul 12 19:23:30.317747 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:30.317723 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-vdl5x" event={"ID":"2b24f43c-e196-4d40-a20b-469489818be0","Type":"ContainerStarted","Data":"1144b44d616d98f375c6f1d0f3fcaf6f6f227b798c689c718a5e3b74c82e15ab"} Jul 12 19:23:30.758388 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:30.758300 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:23:30.813405 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:30.813367 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:23:31.039392 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:31.039301 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:23:31.247462 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:31.247435 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:23:31.300006 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:31.299939 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:23:31.323878 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:31.323847 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-vdl5x" event={"ID":"2b24f43c-e196-4d40-a20b-469489818be0","Type":"ContainerStarted","Data":"b1a92584532d1d625874ecb0a15628ad8c5322eca7721f64031b56a7c6ffb5a6"} Jul 12 19:23:31.324004 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:31.323921 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-vdl5x" event={"ID":"2b24f43c-e196-4d40-a20b-469489818be0","Type":"ContainerStarted","Data":"ccc0560f2bfd02908448b08df6fc73816a5b6df4d50dbd56f64bb60c3b9fe620"} Jul 12 19:23:31.325859 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:31.325836 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-r5rxf" event={"ID":"1d52b8f7-a1db-4ab3-bd17-7462950f76ae","Type":"ContainerStarted","Data":"b1b43fd98371a05d9046bcc609abad6ba45e0179b59ec6fdf2fa0f768e1eb661"} Jul 12 19:23:31.325977 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:31.325875 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-r5rxf" event={"ID":"1d52b8f7-a1db-4ab3-bd17-7462950f76ae","Type":"ContainerStarted","Data":"7490e87cdc341d08bfaac4756a5f588a74bffe16676d6443749580fdc0005582"} Jul 12 19:23:31.340787 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:31.340741 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-785778ff74-vdl5x" podStartSLOduration=3.04288177 podStartE2EDuration="4.340726383s" totalImagesPullingTime="1.297844613s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 19:23:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 19:23:31.339389641 +0000 UTC m=+76.071962130" watchObservedRunningTime="2026-07-12 19:23:31.340726383 +0000 UTC m=+76.073298919" Jul 12 19:23:31.355629 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:31.355591 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-r5rxf" podStartSLOduration=3.243623537 podStartE2EDuration="4.355578194s" totalImagesPullingTime="1.041651698s" totalInitContainerRuntime="70.302959ms" isStatefulPod=true podCreationTimestamp="2026-07-12 19:23:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 19:23:31.354455481 +0000 UTC m=+76.087027983" watchObservedRunningTime="2026-07-12 19:23:31.355578194 +0000 UTC m=+76.088150684" Jul 12 19:23:31.524390 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:31.524356 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:23:32.151575 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:32.151538 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:23:32.167574 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:32.167544 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-75f45dd54d-p5rzj"] Jul 12 19:23:32.170863 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:32.170840 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-75f45dd54d-p5rzj" Jul 12 19:23:32.173132 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:32.173107 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Jul 12 19:23:32.173263 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:32.173190 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Jul 12 19:23:32.173336 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:32.173278 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 12 19:23:32.173416 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:32.173401 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Jul 12 19:23:32.174287 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:32.174271 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-bkggj08k8sbo6\"" Jul 12 19:23:32.174400 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:32.174274 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-k4nfg\"" Jul 12 19:23:32.180423 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:32.180390 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-75f45dd54d-p5rzj"] Jul 12 19:23:32.246029 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:32.245989 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:23:32.285350 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:32.285318 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/ef3fe7d8-3ba3-453d-b939-ba590b2c24a9-audit-log\") pod \"metrics-server-75f45dd54d-p5rzj\" (UID: \"ef3fe7d8-3ba3-453d-b939-ba590b2c24a9\") " pod="openshift-monitoring/metrics-server-75f45dd54d-p5rzj" Jul 12 19:23:32.285728 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:32.285355 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ef3fe7d8-3ba3-453d-b939-ba590b2c24a9-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-75f45dd54d-p5rzj\" (UID: \"ef3fe7d8-3ba3-453d-b939-ba590b2c24a9\") " pod="openshift-monitoring/metrics-server-75f45dd54d-p5rzj" Jul 12 19:23:32.285728 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:32.285393 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vgb9f\" (UniqueName: \"kubernetes.io/projected/ef3fe7d8-3ba3-453d-b939-ba590b2c24a9-kube-api-access-vgb9f\") pod \"metrics-server-75f45dd54d-p5rzj\" (UID: \"ef3fe7d8-3ba3-453d-b939-ba590b2c24a9\") " pod="openshift-monitoring/metrics-server-75f45dd54d-p5rzj" Jul 12 19:23:32.285728 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:32.285417 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/ef3fe7d8-3ba3-453d-b939-ba590b2c24a9-secret-metrics-server-client-certs\") pod \"metrics-server-75f45dd54d-p5rzj\" (UID: \"ef3fe7d8-3ba3-453d-b939-ba590b2c24a9\") " pod="openshift-monitoring/metrics-server-75f45dd54d-p5rzj" Jul 12 19:23:32.285728 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:32.285433 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/ef3fe7d8-3ba3-453d-b939-ba590b2c24a9-metrics-server-audit-profiles\") pod \"metrics-server-75f45dd54d-p5rzj\" (UID: \"ef3fe7d8-3ba3-453d-b939-ba590b2c24a9\") " pod="openshift-monitoring/metrics-server-75f45dd54d-p5rzj" Jul 12 19:23:32.285728 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:32.285465 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/ef3fe7d8-3ba3-453d-b939-ba590b2c24a9-client-ca-bundle\") pod \"metrics-server-75f45dd54d-p5rzj\" (UID: \"ef3fe7d8-3ba3-453d-b939-ba590b2c24a9\") " pod="openshift-monitoring/metrics-server-75f45dd54d-p5rzj" Jul 12 19:23:32.285728 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:32.285487 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/ef3fe7d8-3ba3-453d-b939-ba590b2c24a9-secret-metrics-server-tls\") pod \"metrics-server-75f45dd54d-p5rzj\" (UID: \"ef3fe7d8-3ba3-453d-b939-ba590b2c24a9\") " pod="openshift-monitoring/metrics-server-75f45dd54d-p5rzj" Jul 12 19:23:32.386001 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:32.385963 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vgb9f\" (UniqueName: \"kubernetes.io/projected/ef3fe7d8-3ba3-453d-b939-ba590b2c24a9-kube-api-access-vgb9f\") pod \"metrics-server-75f45dd54d-p5rzj\" (UID: \"ef3fe7d8-3ba3-453d-b939-ba590b2c24a9\") " pod="openshift-monitoring/metrics-server-75f45dd54d-p5rzj" Jul 12 19:23:32.386001 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:32.386004 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/ef3fe7d8-3ba3-453d-b939-ba590b2c24a9-secret-metrics-server-client-certs\") pod \"metrics-server-75f45dd54d-p5rzj\" (UID: \"ef3fe7d8-3ba3-453d-b939-ba590b2c24a9\") " pod="openshift-monitoring/metrics-server-75f45dd54d-p5rzj" Jul 12 19:23:32.386246 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:32.386030 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/ef3fe7d8-3ba3-453d-b939-ba590b2c24a9-metrics-server-audit-profiles\") pod \"metrics-server-75f45dd54d-p5rzj\" (UID: \"ef3fe7d8-3ba3-453d-b939-ba590b2c24a9\") " pod="openshift-monitoring/metrics-server-75f45dd54d-p5rzj" Jul 12 19:23:32.386303 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:32.386272 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/ef3fe7d8-3ba3-453d-b939-ba590b2c24a9-client-ca-bundle\") pod \"metrics-server-75f45dd54d-p5rzj\" (UID: \"ef3fe7d8-3ba3-453d-b939-ba590b2c24a9\") " pod="openshift-monitoring/metrics-server-75f45dd54d-p5rzj" Jul 12 19:23:32.386517 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:32.386488 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/ef3fe7d8-3ba3-453d-b939-ba590b2c24a9-secret-metrics-server-tls\") pod \"metrics-server-75f45dd54d-p5rzj\" (UID: \"ef3fe7d8-3ba3-453d-b939-ba590b2c24a9\") " pod="openshift-monitoring/metrics-server-75f45dd54d-p5rzj" Jul 12 19:23:32.386659 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:32.386616 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/ef3fe7d8-3ba3-453d-b939-ba590b2c24a9-audit-log\") pod \"metrics-server-75f45dd54d-p5rzj\" (UID: \"ef3fe7d8-3ba3-453d-b939-ba590b2c24a9\") " pod="openshift-monitoring/metrics-server-75f45dd54d-p5rzj" Jul 12 19:23:32.386659 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:32.386649 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ef3fe7d8-3ba3-453d-b939-ba590b2c24a9-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-75f45dd54d-p5rzj\" (UID: \"ef3fe7d8-3ba3-453d-b939-ba590b2c24a9\") " pod="openshift-monitoring/metrics-server-75f45dd54d-p5rzj" Jul 12 19:23:32.387122 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:32.387097 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/ef3fe7d8-3ba3-453d-b939-ba590b2c24a9-metrics-server-audit-profiles\") pod \"metrics-server-75f45dd54d-p5rzj\" (UID: \"ef3fe7d8-3ba3-453d-b939-ba590b2c24a9\") " pod="openshift-monitoring/metrics-server-75f45dd54d-p5rzj" Jul 12 19:23:32.387292 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:32.387253 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/ef3fe7d8-3ba3-453d-b939-ba590b2c24a9-audit-log\") pod \"metrics-server-75f45dd54d-p5rzj\" (UID: \"ef3fe7d8-3ba3-453d-b939-ba590b2c24a9\") " pod="openshift-monitoring/metrics-server-75f45dd54d-p5rzj" Jul 12 19:23:32.387559 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:32.387536 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ef3fe7d8-3ba3-453d-b939-ba590b2c24a9-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-75f45dd54d-p5rzj\" (UID: \"ef3fe7d8-3ba3-453d-b939-ba590b2c24a9\") " pod="openshift-monitoring/metrics-server-75f45dd54d-p5rzj" Jul 12 19:23:32.388704 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:32.388683 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/ef3fe7d8-3ba3-453d-b939-ba590b2c24a9-secret-metrics-server-client-certs\") pod \"metrics-server-75f45dd54d-p5rzj\" (UID: \"ef3fe7d8-3ba3-453d-b939-ba590b2c24a9\") " pod="openshift-monitoring/metrics-server-75f45dd54d-p5rzj" Jul 12 19:23:32.388861 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:32.388843 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/ef3fe7d8-3ba3-453d-b939-ba590b2c24a9-client-ca-bundle\") pod \"metrics-server-75f45dd54d-p5rzj\" (UID: \"ef3fe7d8-3ba3-453d-b939-ba590b2c24a9\") " pod="openshift-monitoring/metrics-server-75f45dd54d-p5rzj" Jul 12 19:23:32.389194 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:32.389176 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/ef3fe7d8-3ba3-453d-b939-ba590b2c24a9-secret-metrics-server-tls\") pod \"metrics-server-75f45dd54d-p5rzj\" (UID: \"ef3fe7d8-3ba3-453d-b939-ba590b2c24a9\") " pod="openshift-monitoring/metrics-server-75f45dd54d-p5rzj" Jul 12 19:23:32.395836 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:32.395817 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vgb9f\" (UniqueName: \"kubernetes.io/projected/ef3fe7d8-3ba3-453d-b939-ba590b2c24a9-kube-api-access-vgb9f\") pod \"metrics-server-75f45dd54d-p5rzj\" (UID: \"ef3fe7d8-3ba3-453d-b939-ba590b2c24a9\") " pod="openshift-monitoring/metrics-server-75f45dd54d-p5rzj" Jul 12 19:23:32.481078 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:32.480992 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-75f45dd54d-p5rzj" Jul 12 19:23:32.563899 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:32.563849 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7b5759fdf9-9hlqp" Jul 12 19:23:32.564076 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:32.563933 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7b5759fdf9-9hlqp" Jul 12 19:23:32.568854 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:32.568822 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7b5759fdf9-9hlqp" Jul 12 19:23:32.626852 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:32.626814 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-75f45dd54d-p5rzj"] Jul 12 19:23:32.630838 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:23:32.630812 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef3fe7d8_3ba3_453d_b939_ba590b2c24a9.slice/crio-1e5263e34c2515e0ee3b408ffc7695a12053bf6d8a50fec956ccf1c9d8c33c48 WatchSource:0}: Error finding container 1e5263e34c2515e0ee3b408ffc7695a12053bf6d8a50fec956ccf1c9d8c33c48: Status 404 returned error can't find the container with id 1e5263e34c2515e0ee3b408ffc7695a12053bf6d8a50fec956ccf1c9d8c33c48 Jul 12 19:23:32.638049 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:32.638023 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:23:32.641081 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:32.641061 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-5f68df7f58-l5mf4"] Jul 12 19:23:32.645018 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:32.645001 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-l5mf4" Jul 12 19:23:32.647380 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:32.647363 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Jul 12 19:23:32.647466 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:32.647400 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-89tcv\"" Jul 12 19:23:32.651128 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:32.651107 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5f68df7f58-l5mf4"] Jul 12 19:23:32.688665 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:32.688626 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/502d5209-08a9-431f-a3f0-ab55309635e1-monitoring-plugin-cert\") pod \"monitoring-plugin-5f68df7f58-l5mf4\" (UID: \"502d5209-08a9-431f-a3f0-ab55309635e1\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-l5mf4" Jul 12 19:23:32.688971 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:32.688927 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7wjlm\" (UniqueName: \"kubernetes.io/projected/502d5209-08a9-431f-a3f0-ab55309635e1-kube-api-access-7wjlm\") pod \"monitoring-plugin-5f68df7f58-l5mf4\" (UID: \"502d5209-08a9-431f-a3f0-ab55309635e1\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-l5mf4" Jul 12 19:23:32.690699 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:32.690676 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:23:32.790051 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:32.789958 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7wjlm\" (UniqueName: \"kubernetes.io/projected/502d5209-08a9-431f-a3f0-ab55309635e1-kube-api-access-7wjlm\") pod \"monitoring-plugin-5f68df7f58-l5mf4\" (UID: \"502d5209-08a9-431f-a3f0-ab55309635e1\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-l5mf4" Jul 12 19:23:32.790051 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:32.790010 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/502d5209-08a9-431f-a3f0-ab55309635e1-monitoring-plugin-cert\") pod \"monitoring-plugin-5f68df7f58-l5mf4\" (UID: \"502d5209-08a9-431f-a3f0-ab55309635e1\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-l5mf4" Jul 12 19:23:32.790230 ip-10-0-139-170 kubenswrapper[2565]: E0712 19:23:32.790104 2565 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found Jul 12 19:23:32.790230 ip-10-0-139-170 kubenswrapper[2565]: E0712 19:23:32.790168 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/502d5209-08a9-431f-a3f0-ab55309635e1-monitoring-plugin-cert podName:502d5209-08a9-431f-a3f0-ab55309635e1 nodeName:}" failed. No retries permitted until 2026-07-12 19:23:33.290151528 +0000 UTC m=+78.022723996 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/502d5209-08a9-431f-a3f0-ab55309635e1-monitoring-plugin-cert") pod "monitoring-plugin-5f68df7f58-l5mf4" (UID: "502d5209-08a9-431f-a3f0-ab55309635e1") : secret "monitoring-plugin-cert" not found Jul 12 19:23:32.797867 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:32.797833 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7wjlm\" (UniqueName: \"kubernetes.io/projected/502d5209-08a9-431f-a3f0-ab55309635e1-kube-api-access-7wjlm\") pod \"monitoring-plugin-5f68df7f58-l5mf4\" (UID: \"502d5209-08a9-431f-a3f0-ab55309635e1\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-l5mf4" Jul 12 19:23:32.847273 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:32.847239 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:23:33.293993 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:33.293962 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/502d5209-08a9-431f-a3f0-ab55309635e1-monitoring-plugin-cert\") pod \"monitoring-plugin-5f68df7f58-l5mf4\" (UID: \"502d5209-08a9-431f-a3f0-ab55309635e1\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-l5mf4" Jul 12 19:23:33.296861 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:33.296831 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/502d5209-08a9-431f-a3f0-ab55309635e1-monitoring-plugin-cert\") pod \"monitoring-plugin-5f68df7f58-l5mf4\" (UID: \"502d5209-08a9-431f-a3f0-ab55309635e1\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-l5mf4" Jul 12 19:23:33.333787 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:33.333752 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-75f45dd54d-p5rzj" event={"ID":"ef3fe7d8-3ba3-453d-b939-ba590b2c24a9","Type":"ContainerStarted","Data":"1e5263e34c2515e0ee3b408ffc7695a12053bf6d8a50fec956ccf1c9d8c33c48"} Jul 12 19:23:33.338339 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:33.338314 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7b5759fdf9-9hlqp" Jul 12 19:23:33.380055 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:33.380016 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-b6d9ddcc-2fvdr"] Jul 12 19:23:33.554927 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:33.554829 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-l5mf4" Jul 12 19:23:33.654866 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:33.654834 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-65564fb587-24z67"] Jul 12 19:23:33.659202 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:33.659175 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-65564fb587-24z67" Jul 12 19:23:33.670353 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:33.670284 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-65564fb587-24z67"] Jul 12 19:23:33.694877 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:33.694844 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5f68df7f58-l5mf4"] Jul 12 19:23:33.697347 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:33.697325 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c00c020c-dd2b-45bd-a69c-7600f9bec905-console-oauth-config\") pod \"console-65564fb587-24z67\" (UID: \"c00c020c-dd2b-45bd-a69c-7600f9bec905\") " pod="openshift-console/console-65564fb587-24z67" Jul 12 19:23:33.697440 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:33.697381 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c00c020c-dd2b-45bd-a69c-7600f9bec905-console-serving-cert\") pod \"console-65564fb587-24z67\" (UID: \"c00c020c-dd2b-45bd-a69c-7600f9bec905\") " pod="openshift-console/console-65564fb587-24z67" Jul 12 19:23:33.697500 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:33.697443 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-crxmq\" (UniqueName: \"kubernetes.io/projected/c00c020c-dd2b-45bd-a69c-7600f9bec905-kube-api-access-crxmq\") pod \"console-65564fb587-24z67\" (UID: \"c00c020c-dd2b-45bd-a69c-7600f9bec905\") " pod="openshift-console/console-65564fb587-24z67" Jul 12 19:23:33.697566 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:33.697509 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c00c020c-dd2b-45bd-a69c-7600f9bec905-service-ca\") pod \"console-65564fb587-24z67\" (UID: \"c00c020c-dd2b-45bd-a69c-7600f9bec905\") " pod="openshift-console/console-65564fb587-24z67" Jul 12 19:23:33.697566 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:33.697542 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c00c020c-dd2b-45bd-a69c-7600f9bec905-trusted-ca-bundle\") pod \"console-65564fb587-24z67\" (UID: \"c00c020c-dd2b-45bd-a69c-7600f9bec905\") " pod="openshift-console/console-65564fb587-24z67" Jul 12 19:23:33.697654 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:33.697581 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c00c020c-dd2b-45bd-a69c-7600f9bec905-console-config\") pod \"console-65564fb587-24z67\" (UID: \"c00c020c-dd2b-45bd-a69c-7600f9bec905\") " pod="openshift-console/console-65564fb587-24z67" Jul 12 19:23:33.697707 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:33.697663 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c00c020c-dd2b-45bd-a69c-7600f9bec905-oauth-serving-cert\") pod \"console-65564fb587-24z67\" (UID: \"c00c020c-dd2b-45bd-a69c-7600f9bec905\") " pod="openshift-console/console-65564fb587-24z67" Jul 12 19:23:33.699316 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:23:33.699292 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod502d5209_08a9_431f_a3f0_ab55309635e1.slice/crio-3003096782c594c26194bbf67eefe3f4470d010df2ca38c5c1fb65b4880f176e WatchSource:0}: Error finding container 3003096782c594c26194bbf67eefe3f4470d010df2ca38c5c1fb65b4880f176e: Status 404 returned error can't find the container with id 3003096782c594c26194bbf67eefe3f4470d010df2ca38c5c1fb65b4880f176e Jul 12 19:23:33.798356 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:33.798308 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c00c020c-dd2b-45bd-a69c-7600f9bec905-console-serving-cert\") pod \"console-65564fb587-24z67\" (UID: \"c00c020c-dd2b-45bd-a69c-7600f9bec905\") " pod="openshift-console/console-65564fb587-24z67" Jul 12 19:23:33.798356 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:33.798354 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crxmq\" (UniqueName: \"kubernetes.io/projected/c00c020c-dd2b-45bd-a69c-7600f9bec905-kube-api-access-crxmq\") pod \"console-65564fb587-24z67\" (UID: \"c00c020c-dd2b-45bd-a69c-7600f9bec905\") " pod="openshift-console/console-65564fb587-24z67" Jul 12 19:23:33.798579 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:33.798393 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c00c020c-dd2b-45bd-a69c-7600f9bec905-service-ca\") pod \"console-65564fb587-24z67\" (UID: \"c00c020c-dd2b-45bd-a69c-7600f9bec905\") " pod="openshift-console/console-65564fb587-24z67" Jul 12 19:23:33.798579 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:33.798421 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c00c020c-dd2b-45bd-a69c-7600f9bec905-trusted-ca-bundle\") pod \"console-65564fb587-24z67\" (UID: \"c00c020c-dd2b-45bd-a69c-7600f9bec905\") " pod="openshift-console/console-65564fb587-24z67" Jul 12 19:23:33.798579 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:33.798458 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c00c020c-dd2b-45bd-a69c-7600f9bec905-console-config\") pod \"console-65564fb587-24z67\" (UID: \"c00c020c-dd2b-45bd-a69c-7600f9bec905\") " pod="openshift-console/console-65564fb587-24z67" Jul 12 19:23:33.798579 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:33.798503 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c00c020c-dd2b-45bd-a69c-7600f9bec905-oauth-serving-cert\") pod \"console-65564fb587-24z67\" (UID: \"c00c020c-dd2b-45bd-a69c-7600f9bec905\") " pod="openshift-console/console-65564fb587-24z67" Jul 12 19:23:33.798579 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:33.798532 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c00c020c-dd2b-45bd-a69c-7600f9bec905-console-oauth-config\") pod \"console-65564fb587-24z67\" (UID: \"c00c020c-dd2b-45bd-a69c-7600f9bec905\") " pod="openshift-console/console-65564fb587-24z67" Jul 12 19:23:33.800002 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:33.799669 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c00c020c-dd2b-45bd-a69c-7600f9bec905-service-ca\") pod \"console-65564fb587-24z67\" (UID: \"c00c020c-dd2b-45bd-a69c-7600f9bec905\") " pod="openshift-console/console-65564fb587-24z67" Jul 12 19:23:33.800968 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:33.800920 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c00c020c-dd2b-45bd-a69c-7600f9bec905-trusted-ca-bundle\") pod \"console-65564fb587-24z67\" (UID: \"c00c020c-dd2b-45bd-a69c-7600f9bec905\") " pod="openshift-console/console-65564fb587-24z67" Jul 12 19:23:33.801369 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:33.801345 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c00c020c-dd2b-45bd-a69c-7600f9bec905-console-config\") pod \"console-65564fb587-24z67\" (UID: \"c00c020c-dd2b-45bd-a69c-7600f9bec905\") " pod="openshift-console/console-65564fb587-24z67" Jul 12 19:23:33.801465 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:33.801434 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c00c020c-dd2b-45bd-a69c-7600f9bec905-oauth-serving-cert\") pod \"console-65564fb587-24z67\" (UID: \"c00c020c-dd2b-45bd-a69c-7600f9bec905\") " pod="openshift-console/console-65564fb587-24z67" Jul 12 19:23:33.803605 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:33.803578 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c00c020c-dd2b-45bd-a69c-7600f9bec905-console-oauth-config\") pod \"console-65564fb587-24z67\" (UID: \"c00c020c-dd2b-45bd-a69c-7600f9bec905\") " pod="openshift-console/console-65564fb587-24z67" Jul 12 19:23:33.803727 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:33.803684 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c00c020c-dd2b-45bd-a69c-7600f9bec905-console-serving-cert\") pod \"console-65564fb587-24z67\" (UID: \"c00c020c-dd2b-45bd-a69c-7600f9bec905\") " pod="openshift-console/console-65564fb587-24z67" Jul 12 19:23:33.809913 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:33.809840 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-crxmq\" (UniqueName: \"kubernetes.io/projected/c00c020c-dd2b-45bd-a69c-7600f9bec905-kube-api-access-crxmq\") pod \"console-65564fb587-24z67\" (UID: \"c00c020c-dd2b-45bd-a69c-7600f9bec905\") " pod="openshift-console/console-65564fb587-24z67" Jul 12 19:23:33.971180 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:33.971139 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-65564fb587-24z67" Jul 12 19:23:34.028912 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:34.028849 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 12 19:23:34.031465 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:34.031387 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:23:34.033577 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:34.033514 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 12 19:23:34.036036 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:34.036010 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-4fj3jjfj75avh\"" Jul 12 19:23:34.036144 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:34.036056 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 12 19:23:34.036569 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:34.036535 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 12 19:23:34.036713 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:34.036589 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 12 19:23:34.036713 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:34.036589 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 12 19:23:34.036713 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:34.036691 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 12 19:23:34.036713 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:34.036689 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 12 19:23:34.036917 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:34.036820 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 12 19:23:34.036973 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:34.036925 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 12 19:23:34.037019 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:34.036996 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 12 19:23:34.037161 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:34.037130 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-fqbbf\"" Jul 12 19:23:34.037354 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:34.037252 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 12 19:23:34.038436 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:34.037499 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 12 19:23:34.038436 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:34.037578 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 12 19:23:34.041013 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:34.040993 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 12 19:23:34.043557 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:34.043534 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 12 19:23:34.047417 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:34.047395 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 12 19:23:34.103919 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:34.103806 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1588d7eb-6773-4bcb-bce3-cc83d47485b5-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"1588d7eb-6773-4bcb-bce3-cc83d47485b5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 12 19:23:34.103919 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:34.103854 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1588d7eb-6773-4bcb-bce3-cc83d47485b5-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"1588d7eb-6773-4bcb-bce3-cc83d47485b5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 12 19:23:34.103919 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:34.103900 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1588d7eb-6773-4bcb-bce3-cc83d47485b5-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1588d7eb-6773-4bcb-bce3-cc83d47485b5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 12 19:23:34.104159 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:34.103934 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/1588d7eb-6773-4bcb-bce3-cc83d47485b5-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"1588d7eb-6773-4bcb-bce3-cc83d47485b5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 12 19:23:34.104159 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:34.103957 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/1588d7eb-6773-4bcb-bce3-cc83d47485b5-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"1588d7eb-6773-4bcb-bce3-cc83d47485b5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 12 19:23:34.104159 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:34.103981 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/1588d7eb-6773-4bcb-bce3-cc83d47485b5-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"1588d7eb-6773-4bcb-bce3-cc83d47485b5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 12 19:23:34.104159 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:34.104031 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/1588d7eb-6773-4bcb-bce3-cc83d47485b5-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"1588d7eb-6773-4bcb-bce3-cc83d47485b5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 12 19:23:34.104159 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:34.104087 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1588d7eb-6773-4bcb-bce3-cc83d47485b5-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"1588d7eb-6773-4bcb-bce3-cc83d47485b5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 12 19:23:34.104159 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:34.104108 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/1588d7eb-6773-4bcb-bce3-cc83d47485b5-config\") pod \"prometheus-k8s-0\" (UID: \"1588d7eb-6773-4bcb-bce3-cc83d47485b5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 12 19:23:34.104159 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:34.104128 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1588d7eb-6773-4bcb-bce3-cc83d47485b5-web-config\") pod \"prometheus-k8s-0\" (UID: \"1588d7eb-6773-4bcb-bce3-cc83d47485b5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 12 19:23:34.104159 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:34.104150 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1588d7eb-6773-4bcb-bce3-cc83d47485b5-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"1588d7eb-6773-4bcb-bce3-cc83d47485b5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 12 19:23:34.104536 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:34.104195 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1588d7eb-6773-4bcb-bce3-cc83d47485b5-config-out\") pod \"prometheus-k8s-0\" (UID: \"1588d7eb-6773-4bcb-bce3-cc83d47485b5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 12 19:23:34.104536 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:34.104244 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l25wz\" (UniqueName: \"kubernetes.io/projected/1588d7eb-6773-4bcb-bce3-cc83d47485b5-kube-api-access-l25wz\") pod \"prometheus-k8s-0\" (UID: \"1588d7eb-6773-4bcb-bce3-cc83d47485b5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 12 19:23:34.104536 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:34.104305 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1588d7eb-6773-4bcb-bce3-cc83d47485b5-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1588d7eb-6773-4bcb-bce3-cc83d47485b5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 12 19:23:34.104536 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:34.104336 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/1588d7eb-6773-4bcb-bce3-cc83d47485b5-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"1588d7eb-6773-4bcb-bce3-cc83d47485b5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 12 19:23:34.104536 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:34.104370 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/1588d7eb-6773-4bcb-bce3-cc83d47485b5-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"1588d7eb-6773-4bcb-bce3-cc83d47485b5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 12 19:23:34.104536 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:34.104394 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/1588d7eb-6773-4bcb-bce3-cc83d47485b5-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"1588d7eb-6773-4bcb-bce3-cc83d47485b5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 12 19:23:34.104536 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:34.104424 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/1588d7eb-6773-4bcb-bce3-cc83d47485b5-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"1588d7eb-6773-4bcb-bce3-cc83d47485b5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 12 19:23:34.104536 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:34.104447 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1588d7eb-6773-4bcb-bce3-cc83d47485b5-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1588d7eb-6773-4bcb-bce3-cc83d47485b5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 12 19:23:34.104536 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:34.104473 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/1588d7eb-6773-4bcb-bce3-cc83d47485b5-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"1588d7eb-6773-4bcb-bce3-cc83d47485b5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 12 19:23:34.206138 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:34.205854 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1588d7eb-6773-4bcb-bce3-cc83d47485b5-config-out\") pod \"prometheus-k8s-0\" (UID: \"1588d7eb-6773-4bcb-bce3-cc83d47485b5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 12 19:23:34.206138 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:34.205928 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l25wz\" (UniqueName: \"kubernetes.io/projected/1588d7eb-6773-4bcb-bce3-cc83d47485b5-kube-api-access-l25wz\") pod \"prometheus-k8s-0\" (UID: \"1588d7eb-6773-4bcb-bce3-cc83d47485b5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 12 19:23:34.206138 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:34.205957 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1588d7eb-6773-4bcb-bce3-cc83d47485b5-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1588d7eb-6773-4bcb-bce3-cc83d47485b5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 12 19:23:34.206138 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:34.205982 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/1588d7eb-6773-4bcb-bce3-cc83d47485b5-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"1588d7eb-6773-4bcb-bce3-cc83d47485b5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 12 19:23:34.206138 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:34.206010 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/1588d7eb-6773-4bcb-bce3-cc83d47485b5-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"1588d7eb-6773-4bcb-bce3-cc83d47485b5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 12 19:23:34.206138 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:34.206034 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/1588d7eb-6773-4bcb-bce3-cc83d47485b5-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"1588d7eb-6773-4bcb-bce3-cc83d47485b5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 12 19:23:34.206138 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:34.206063 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/1588d7eb-6773-4bcb-bce3-cc83d47485b5-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"1588d7eb-6773-4bcb-bce3-cc83d47485b5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 12 19:23:34.206138 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:34.206085 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1588d7eb-6773-4bcb-bce3-cc83d47485b5-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1588d7eb-6773-4bcb-bce3-cc83d47485b5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 12 19:23:34.206138 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:34.206111 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/1588d7eb-6773-4bcb-bce3-cc83d47485b5-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"1588d7eb-6773-4bcb-bce3-cc83d47485b5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 12 19:23:34.206566 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:34.206149 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1588d7eb-6773-4bcb-bce3-cc83d47485b5-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"1588d7eb-6773-4bcb-bce3-cc83d47485b5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 12 19:23:34.206566 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:34.206171 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1588d7eb-6773-4bcb-bce3-cc83d47485b5-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"1588d7eb-6773-4bcb-bce3-cc83d47485b5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 12 19:23:34.206566 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:34.206262 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1588d7eb-6773-4bcb-bce3-cc83d47485b5-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1588d7eb-6773-4bcb-bce3-cc83d47485b5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 12 19:23:34.206566 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:34.206319 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/1588d7eb-6773-4bcb-bce3-cc83d47485b5-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"1588d7eb-6773-4bcb-bce3-cc83d47485b5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 12 19:23:34.206566 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:34.206349 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/1588d7eb-6773-4bcb-bce3-cc83d47485b5-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"1588d7eb-6773-4bcb-bce3-cc83d47485b5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 12 19:23:34.208163 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:34.206948 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/1588d7eb-6773-4bcb-bce3-cc83d47485b5-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"1588d7eb-6773-4bcb-bce3-cc83d47485b5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 12 19:23:34.208163 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:34.207228 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1588d7eb-6773-4bcb-bce3-cc83d47485b5-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1588d7eb-6773-4bcb-bce3-cc83d47485b5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 12 19:23:34.208163 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:34.207356 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1588d7eb-6773-4bcb-bce3-cc83d47485b5-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"1588d7eb-6773-4bcb-bce3-cc83d47485b5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 12 19:23:34.208163 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:34.207527 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1588d7eb-6773-4bcb-bce3-cc83d47485b5-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1588d7eb-6773-4bcb-bce3-cc83d47485b5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 12 19:23:34.208163 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:34.207571 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/1588d7eb-6773-4bcb-bce3-cc83d47485b5-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"1588d7eb-6773-4bcb-bce3-cc83d47485b5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 12 19:23:34.208163 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:34.207602 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/1588d7eb-6773-4bcb-bce3-cc83d47485b5-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"1588d7eb-6773-4bcb-bce3-cc83d47485b5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 12 19:23:34.208163 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:34.207657 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1588d7eb-6773-4bcb-bce3-cc83d47485b5-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"1588d7eb-6773-4bcb-bce3-cc83d47485b5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 12 19:23:34.208163 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:34.207691 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/1588d7eb-6773-4bcb-bce3-cc83d47485b5-config\") pod \"prometheus-k8s-0\" (UID: \"1588d7eb-6773-4bcb-bce3-cc83d47485b5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 12 19:23:34.208163 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:34.207729 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1588d7eb-6773-4bcb-bce3-cc83d47485b5-web-config\") pod \"prometheus-k8s-0\" (UID: \"1588d7eb-6773-4bcb-bce3-cc83d47485b5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 12 19:23:34.208163 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:34.207761 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1588d7eb-6773-4bcb-bce3-cc83d47485b5-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"1588d7eb-6773-4bcb-bce3-cc83d47485b5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 12 19:23:34.208163 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:34.208139 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/1588d7eb-6773-4bcb-bce3-cc83d47485b5-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"1588d7eb-6773-4bcb-bce3-cc83d47485b5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 12 19:23:34.208788 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:34.208527 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1588d7eb-6773-4bcb-bce3-cc83d47485b5-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1588d7eb-6773-4bcb-bce3-cc83d47485b5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 12 19:23:34.208788 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:34.208609 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1588d7eb-6773-4bcb-bce3-cc83d47485b5-config-out\") pod \"prometheus-k8s-0\" (UID: \"1588d7eb-6773-4bcb-bce3-cc83d47485b5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 12 19:23:34.209636 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:34.209606 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/1588d7eb-6773-4bcb-bce3-cc83d47485b5-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"1588d7eb-6773-4bcb-bce3-cc83d47485b5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 12 19:23:34.210235 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:34.210209 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/1588d7eb-6773-4bcb-bce3-cc83d47485b5-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"1588d7eb-6773-4bcb-bce3-cc83d47485b5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 12 19:23:34.211707 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:34.211620 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1588d7eb-6773-4bcb-bce3-cc83d47485b5-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"1588d7eb-6773-4bcb-bce3-cc83d47485b5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 12 19:23:34.211707 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:34.211642 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1588d7eb-6773-4bcb-bce3-cc83d47485b5-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"1588d7eb-6773-4bcb-bce3-cc83d47485b5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 12 19:23:34.211915 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:34.211716 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/1588d7eb-6773-4bcb-bce3-cc83d47485b5-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"1588d7eb-6773-4bcb-bce3-cc83d47485b5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 12 19:23:34.211915 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:34.211808 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/1588d7eb-6773-4bcb-bce3-cc83d47485b5-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"1588d7eb-6773-4bcb-bce3-cc83d47485b5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 12 19:23:34.212512 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:34.212388 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/1588d7eb-6773-4bcb-bce3-cc83d47485b5-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"1588d7eb-6773-4bcb-bce3-cc83d47485b5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 12 19:23:34.212512 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:34.212493 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/1588d7eb-6773-4bcb-bce3-cc83d47485b5-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"1588d7eb-6773-4bcb-bce3-cc83d47485b5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 12 19:23:34.212665 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:34.212600 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/1588d7eb-6773-4bcb-bce3-cc83d47485b5-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"1588d7eb-6773-4bcb-bce3-cc83d47485b5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 12 19:23:34.212721 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:34.212693 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/1588d7eb-6773-4bcb-bce3-cc83d47485b5-config\") pod \"prometheus-k8s-0\" (UID: \"1588d7eb-6773-4bcb-bce3-cc83d47485b5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 12 19:23:34.213370 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:34.213330 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1588d7eb-6773-4bcb-bce3-cc83d47485b5-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"1588d7eb-6773-4bcb-bce3-cc83d47485b5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 12 19:23:34.215817 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:34.215346 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1588d7eb-6773-4bcb-bce3-cc83d47485b5-web-config\") pod \"prometheus-k8s-0\" (UID: \"1588d7eb-6773-4bcb-bce3-cc83d47485b5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 12 19:23:34.217399 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:34.217367 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l25wz\" (UniqueName: \"kubernetes.io/projected/1588d7eb-6773-4bcb-bce3-cc83d47485b5-kube-api-access-l25wz\") pod \"prometheus-k8s-0\" (UID: \"1588d7eb-6773-4bcb-bce3-cc83d47485b5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 12 19:23:34.293664 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:34.293502 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:23:34.343578 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:34.343532 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-l5mf4" event={"ID":"502d5209-08a9-431f-a3f0-ab55309635e1","Type":"ContainerStarted","Data":"3003096782c594c26194bbf67eefe3f4470d010df2ca38c5c1fb65b4880f176e"} Jul 12 19:23:34.344063 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:34.344041 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-65564fb587-24z67"] Jul 12 19:23:34.346082 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:23:34.346057 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc00c020c_dd2b_45bd_a69c_7600f9bec905.slice/crio-d3657ec129153fa08af54d9df06f095b9f2b6e4fc1abbfed68d6db0f9cfbc684 WatchSource:0}: Error finding container d3657ec129153fa08af54d9df06f095b9f2b6e4fc1abbfed68d6db0f9cfbc684: Status 404 returned error can't find the container with id d3657ec129153fa08af54d9df06f095b9f2b6e4fc1abbfed68d6db0f9cfbc684 Jul 12 19:23:34.351063 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:34.351043 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 12 19:23:34.533575 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:34.533539 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 12 19:23:35.349949 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:35.349905 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-65564fb587-24z67" event={"ID":"c00c020c-dd2b-45bd-a69c-7600f9bec905","Type":"ContainerStarted","Data":"62732d3361becbf94146b0de33f251bb4b282b662bbf1df9d8b4d306a1ed84fc"} Jul 12 19:23:35.350437 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:35.349971 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-65564fb587-24z67" event={"ID":"c00c020c-dd2b-45bd-a69c-7600f9bec905","Type":"ContainerStarted","Data":"d3657ec129153fa08af54d9df06f095b9f2b6e4fc1abbfed68d6db0f9cfbc684"} Jul 12 19:23:35.351718 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:35.351689 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-75f45dd54d-p5rzj" event={"ID":"ef3fe7d8-3ba3-453d-b939-ba590b2c24a9","Type":"ContainerStarted","Data":"00879cecd8981ff13c334a8f07ff1ba0a7f4b020e4cf20b7c0e9a72fd7367b4e"} Jul 12 19:23:35.353077 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:35.353050 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1588d7eb-6773-4bcb-bce3-cc83d47485b5","Type":"ContainerStarted","Data":"6004c2943677efc43fa50eca5c075f9305f67b90d813f8de421548aeebc820b1"} Jul 12 19:23:35.366253 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:35.366194 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-65564fb587-24z67" podStartSLOduration=2.366176848 podStartE2EDuration="2.366176848s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 19:23:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 19:23:35.36510607 +0000 UTC m=+80.097678560" watchObservedRunningTime="2026-07-12 19:23:35.366176848 +0000 UTC m=+80.098749337" Jul 12 19:23:35.380776 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:35.380725 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-75f45dd54d-p5rzj" podStartSLOduration=1.7476751130000001 podStartE2EDuration="3.380708964s" totalImagesPullingTime="1.633033851s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 19:23:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 19:23:35.379173355 +0000 UTC m=+80.111745856" watchObservedRunningTime="2026-07-12 19:23:35.380708964 +0000 UTC m=+80.113281450" Jul 12 19:23:35.533909 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:35.533859 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:23:35.634454 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:35.634423 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:23:36.357418 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:36.357382 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-l5mf4" event={"ID":"502d5209-08a9-431f-a3f0-ab55309635e1","Type":"ContainerStarted","Data":"b3edc8caadf6647ed8cabcc5060777c6ed407ef5c5a983066fab5e0d8e87247d"} Jul 12 19:23:36.357839 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:36.357478 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-l5mf4" Jul 12 19:23:36.359029 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:36.359003 2565 generic.go:356] "Generic (PLEG): container finished" podID="1588d7eb-6773-4bcb-bce3-cc83d47485b5" containerID="11805770b29f472bf1c9fea04ed051c0725b69db8154dfdba72a5102b5b4445f" exitCode=0 Jul 12 19:23:36.359145 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:36.359096 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1588d7eb-6773-4bcb-bce3-cc83d47485b5","Type":"ContainerDied","Data":"11805770b29f472bf1c9fea04ed051c0725b69db8154dfdba72a5102b5b4445f"} Jul 12 19:23:36.362532 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:36.362512 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-l5mf4" Jul 12 19:23:36.374286 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:36.374243 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-l5mf4" podStartSLOduration=2.535516307 podStartE2EDuration="4.374231549s" totalImagesPullingTime="1.838715242s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 19:23:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 19:23:36.373134924 +0000 UTC m=+81.105707414" watchObservedRunningTime="2026-07-12 19:23:36.374231549 +0000 UTC m=+81.106804074" Jul 12 19:23:36.651571 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:36.651536 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:23:36.699188 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:36.699155 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:23:37.583184 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:37.583146 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:23:37.632150 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:37.632104 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:23:40.377907 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:40.377857 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1588d7eb-6773-4bcb-bce3-cc83d47485b5","Type":"ContainerStarted","Data":"01ea109191f641473ad72c6ff894f160c054478bd92b7977fd7b850541a61289"} Jul 12 19:23:40.378242 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:40.377938 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1588d7eb-6773-4bcb-bce3-cc83d47485b5","Type":"ContainerStarted","Data":"63ec2021dd7ecb08e2ebe0bfb52c6f6545efb0be96bf4ee3252d4ff8b4f53f26"} Jul 12 19:23:42.390712 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:42.390678 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1588d7eb-6773-4bcb-bce3-cc83d47485b5","Type":"ContainerStarted","Data":"a770455cbf605db396178218fa5b6e1886319a460c9aa6bd4119498e5a916a69"} Jul 12 19:23:42.391086 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:42.390741 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1588d7eb-6773-4bcb-bce3-cc83d47485b5","Type":"ContainerStarted","Data":"c491dc653526a0e43d6377f36498c0ea22076d47f86e3171add74c8e2471b0b0"} Jul 12 19:23:42.391086 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:42.390772 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1588d7eb-6773-4bcb-bce3-cc83d47485b5","Type":"ContainerStarted","Data":"87ace26f1731444ad87e55a79b011e475514430c6fc9b455f35795e9d85eb53f"} Jul 12 19:23:42.391086 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:42.390798 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1588d7eb-6773-4bcb-bce3-cc83d47485b5","Type":"ContainerStarted","Data":"af60f7ebac1b540cce7cab738f4ce53f396af27b6acecf25d2c90397817475d0"} Jul 12 19:23:42.418384 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:42.418274 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.098808586 podStartE2EDuration="8.418257368s" totalImagesPullingTime="6.296585419s" totalInitContainerRuntime="22.863363ms" isStatefulPod=false podCreationTimestamp="2026-07-12 19:23:34 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 19:23:42.416343493 +0000 UTC m=+87.148915985" watchObservedRunningTime="2026-07-12 19:23:42.418257368 +0000 UTC m=+87.150829911" Jul 12 19:23:43.971696 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:43.971657 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-65564fb587-24z67" Jul 12 19:23:43.972306 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:43.971961 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-65564fb587-24z67" Jul 12 19:23:43.977305 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:43.977281 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-65564fb587-24z67" Jul 12 19:23:44.351669 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:44.351576 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 12 19:23:44.402624 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:44.402594 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-65564fb587-24z67" Jul 12 19:23:44.444504 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:44.444475 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7b5759fdf9-9hlqp"] Jul 12 19:23:44.553711 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:44.553673 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:23:44.653965 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:44.653932 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:23:44.753815 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:44.753784 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:23:44.812697 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:44.812651 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:23:46.660157 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:46.660098 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:23:46.850528 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:46.850500 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:23:51.654114 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:51.654076 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:23:51.754801 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:51.754773 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:23:52.481102 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:52.481067 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-75f45dd54d-p5rzj" Jul 12 19:23:52.481272 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:52.481141 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-75f45dd54d-p5rzj" Jul 12 19:23:58.307995 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:58.307964 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-76gfm" Jul 12 19:23:58.405018 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:58.404959 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-b6d9ddcc-2fvdr" podUID="dc6274fb-accb-4f81-ae6e-8133c6ed4132" containerName="console" containerID="cri-o://01938b77bfc3c666db2904818b37798c50537c4f9353b74b6bc6c71d5a1d00d7" gracePeriod=15 Jul 12 19:23:58.644461 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:58.644437 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-b6d9ddcc-2fvdr" Jul 12 19:23:58.796012 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:58.795974 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:23:58.826020 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:58.825958 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/dc6274fb-accb-4f81-ae6e-8133c6ed4132-service-ca\") pod \"dc6274fb-accb-4f81-ae6e-8133c6ed4132\" (UID: \"dc6274fb-accb-4f81-ae6e-8133c6ed4132\") " Jul 12 19:23:58.826131 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:58.826026 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/dc6274fb-accb-4f81-ae6e-8133c6ed4132-oauth-serving-cert\") pod \"dc6274fb-accb-4f81-ae6e-8133c6ed4132\" (UID: \"dc6274fb-accb-4f81-ae6e-8133c6ed4132\") " Jul 12 19:23:58.826131 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:58.826049 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/dc6274fb-accb-4f81-ae6e-8133c6ed4132-console-serving-cert\") pod \"dc6274fb-accb-4f81-ae6e-8133c6ed4132\" (UID: \"dc6274fb-accb-4f81-ae6e-8133c6ed4132\") " Jul 12 19:23:58.826237 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:58.826219 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/dc6274fb-accb-4f81-ae6e-8133c6ed4132-console-config\") pod \"dc6274fb-accb-4f81-ae6e-8133c6ed4132\" (UID: \"dc6274fb-accb-4f81-ae6e-8133c6ed4132\") " Jul 12 19:23:58.826324 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:58.826309 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/dc6274fb-accb-4f81-ae6e-8133c6ed4132-console-oauth-config\") pod \"dc6274fb-accb-4f81-ae6e-8133c6ed4132\" (UID: \"dc6274fb-accb-4f81-ae6e-8133c6ed4132\") " Jul 12 19:23:58.826375 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:58.826343 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sdkff\" (UniqueName: \"kubernetes.io/projected/dc6274fb-accb-4f81-ae6e-8133c6ed4132-kube-api-access-sdkff\") pod \"dc6274fb-accb-4f81-ae6e-8133c6ed4132\" (UID: \"dc6274fb-accb-4f81-ae6e-8133c6ed4132\") " Jul 12 19:23:58.826435 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:58.826394 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc6274fb-accb-4f81-ae6e-8133c6ed4132-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "dc6274fb-accb-4f81-ae6e-8133c6ed4132" (UID: "dc6274fb-accb-4f81-ae6e-8133c6ed4132"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 12 19:23:58.826482 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:58.826404 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc6274fb-accb-4f81-ae6e-8133c6ed4132-service-ca" (OuterVolumeSpecName: "service-ca") pod "dc6274fb-accb-4f81-ae6e-8133c6ed4132" (UID: "dc6274fb-accb-4f81-ae6e-8133c6ed4132"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 12 19:23:58.826631 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:58.826607 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc6274fb-accb-4f81-ae6e-8133c6ed4132-console-config" (OuterVolumeSpecName: "console-config") pod "dc6274fb-accb-4f81-ae6e-8133c6ed4132" (UID: "dc6274fb-accb-4f81-ae6e-8133c6ed4132"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 12 19:23:58.826758 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:58.826745 2565 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/dc6274fb-accb-4f81-ae6e-8133c6ed4132-console-config\") on node \"ip-10-0-139-170.ec2.internal\" DevicePath \"\"" Jul 12 19:23:58.826817 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:58.826760 2565 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/dc6274fb-accb-4f81-ae6e-8133c6ed4132-service-ca\") on node \"ip-10-0-139-170.ec2.internal\" DevicePath \"\"" Jul 12 19:23:58.826817 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:58.826770 2565 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/dc6274fb-accb-4f81-ae6e-8133c6ed4132-oauth-serving-cert\") on node \"ip-10-0-139-170.ec2.internal\" DevicePath \"\"" Jul 12 19:23:58.828427 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:58.828399 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc6274fb-accb-4f81-ae6e-8133c6ed4132-kube-api-access-sdkff" (OuterVolumeSpecName: "kube-api-access-sdkff") pod "dc6274fb-accb-4f81-ae6e-8133c6ed4132" (UID: "dc6274fb-accb-4f81-ae6e-8133c6ed4132"). InnerVolumeSpecName "kube-api-access-sdkff". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 12 19:23:58.828427 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:58.828413 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc6274fb-accb-4f81-ae6e-8133c6ed4132-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "dc6274fb-accb-4f81-ae6e-8133c6ed4132" (UID: "dc6274fb-accb-4f81-ae6e-8133c6ed4132"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 12 19:23:58.828552 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:58.828440 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc6274fb-accb-4f81-ae6e-8133c6ed4132-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "dc6274fb-accb-4f81-ae6e-8133c6ed4132" (UID: "dc6274fb-accb-4f81-ae6e-8133c6ed4132"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 12 19:23:58.847589 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:58.847563 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:23:58.928186 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:58.928160 2565 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/dc6274fb-accb-4f81-ae6e-8133c6ed4132-console-serving-cert\") on node \"ip-10-0-139-170.ec2.internal\" DevicePath \"\"" Jul 12 19:23:58.928186 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:58.928183 2565 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/dc6274fb-accb-4f81-ae6e-8133c6ed4132-console-oauth-config\") on node \"ip-10-0-139-170.ec2.internal\" DevicePath \"\"" Jul 12 19:23:58.928343 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:58.928197 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sdkff\" (UniqueName: \"kubernetes.io/projected/dc6274fb-accb-4f81-ae6e-8133c6ed4132-kube-api-access-sdkff\") on node \"ip-10-0-139-170.ec2.internal\" DevicePath \"\"" Jul 12 19:23:59.462101 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:59.462067 2565 generic.go:356] "Generic (PLEG): container finished" podID="dc6274fb-accb-4f81-ae6e-8133c6ed4132" containerID="01938b77bfc3c666db2904818b37798c50537c4f9353b74b6bc6c71d5a1d00d7" exitCode=0 Jul 12 19:23:59.462514 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:59.462127 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-b6d9ddcc-2fvdr" Jul 12 19:23:59.462514 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:59.462145 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-b6d9ddcc-2fvdr" event={"ID":"dc6274fb-accb-4f81-ae6e-8133c6ed4132","Type":"ContainerDied","Data":"01938b77bfc3c666db2904818b37798c50537c4f9353b74b6bc6c71d5a1d00d7"} Jul 12 19:23:59.462514 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:59.462229 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-b6d9ddcc-2fvdr" event={"ID":"dc6274fb-accb-4f81-ae6e-8133c6ed4132","Type":"ContainerDied","Data":"0cf3f0d43f68b6e6e436ee375388a6f4a6fcd23780b9ec4585f4d387e7b6c983"} Jul 12 19:23:59.462514 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:59.462254 2565 scope.go:117] "RemoveContainer" containerID="01938b77bfc3c666db2904818b37798c50537c4f9353b74b6bc6c71d5a1d00d7" Jul 12 19:23:59.470690 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:59.470674 2565 scope.go:117] "RemoveContainer" containerID="01938b77bfc3c666db2904818b37798c50537c4f9353b74b6bc6c71d5a1d00d7" Jul 12 19:23:59.470948 ip-10-0-139-170 kubenswrapper[2565]: E0712 19:23:59.470929 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01938b77bfc3c666db2904818b37798c50537c4f9353b74b6bc6c71d5a1d00d7\": container with ID starting with 01938b77bfc3c666db2904818b37798c50537c4f9353b74b6bc6c71d5a1d00d7 not found: ID does not exist" containerID="01938b77bfc3c666db2904818b37798c50537c4f9353b74b6bc6c71d5a1d00d7" Jul 12 19:23:59.470996 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:59.470957 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01938b77bfc3c666db2904818b37798c50537c4f9353b74b6bc6c71d5a1d00d7"} err="failed to get container status \"01938b77bfc3c666db2904818b37798c50537c4f9353b74b6bc6c71d5a1d00d7\": rpc error: code = NotFound desc = could not find container \"01938b77bfc3c666db2904818b37798c50537c4f9353b74b6bc6c71d5a1d00d7\": container with ID starting with 01938b77bfc3c666db2904818b37798c50537c4f9353b74b6bc6c71d5a1d00d7 not found: ID does not exist" Jul 12 19:23:59.481729 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:59.481706 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-b6d9ddcc-2fvdr"] Jul 12 19:23:59.483801 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:59.483784 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-b6d9ddcc-2fvdr"] Jul 12 19:23:59.856876 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:23:59.856783 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dc6274fb-accb-4f81-ae6e-8133c6ed4132" path="/var/lib/kubelet/pods/dc6274fb-accb-4f81-ae6e-8133c6ed4132/volumes" Jul 12 19:24:06.648743 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:24:06.648705 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:24:06.698061 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:24:06.698030 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:24:09.465240 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:24:09.465205 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-7b5759fdf9-9hlqp" podUID="abed6323-d374-454c-bbee-80e9403ff351" containerName="console" containerID="cri-o://afafeb2d77b14e69085ff278a2120d81cc433666f3376e6d1c55e16b4d1cd1e0" gracePeriod=15 Jul 12 19:24:09.733667 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:24:09.733604 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7b5759fdf9-9hlqp" Jul 12 19:24:09.808238 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:24:09.808206 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/abed6323-d374-454c-bbee-80e9403ff351-console-oauth-config\") pod \"abed6323-d374-454c-bbee-80e9403ff351\" (UID: \"abed6323-d374-454c-bbee-80e9403ff351\") " Jul 12 19:24:09.808393 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:24:09.808246 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/abed6323-d374-454c-bbee-80e9403ff351-console-config\") pod \"abed6323-d374-454c-bbee-80e9403ff351\" (UID: \"abed6323-d374-454c-bbee-80e9403ff351\") " Jul 12 19:24:09.808393 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:24:09.808277 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/abed6323-d374-454c-bbee-80e9403ff351-oauth-serving-cert\") pod \"abed6323-d374-454c-bbee-80e9403ff351\" (UID: \"abed6323-d374-454c-bbee-80e9403ff351\") " Jul 12 19:24:09.808393 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:24:09.808297 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/abed6323-d374-454c-bbee-80e9403ff351-service-ca\") pod \"abed6323-d374-454c-bbee-80e9403ff351\" (UID: \"abed6323-d374-454c-bbee-80e9403ff351\") " Jul 12 19:24:09.808393 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:24:09.808383 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rg4tr\" (UniqueName: \"kubernetes.io/projected/abed6323-d374-454c-bbee-80e9403ff351-kube-api-access-rg4tr\") pod \"abed6323-d374-454c-bbee-80e9403ff351\" (UID: \"abed6323-d374-454c-bbee-80e9403ff351\") " Jul 12 19:24:09.808575 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:24:09.808416 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/abed6323-d374-454c-bbee-80e9403ff351-trusted-ca-bundle\") pod \"abed6323-d374-454c-bbee-80e9403ff351\" (UID: \"abed6323-d374-454c-bbee-80e9403ff351\") " Jul 12 19:24:09.808575 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:24:09.808512 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/abed6323-d374-454c-bbee-80e9403ff351-console-serving-cert\") pod \"abed6323-d374-454c-bbee-80e9403ff351\" (UID: \"abed6323-d374-454c-bbee-80e9403ff351\") " Jul 12 19:24:09.808719 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:24:09.808692 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/abed6323-d374-454c-bbee-80e9403ff351-console-config" (OuterVolumeSpecName: "console-config") pod "abed6323-d374-454c-bbee-80e9403ff351" (UID: "abed6323-d374-454c-bbee-80e9403ff351"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 12 19:24:09.808803 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:24:09.808718 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/abed6323-d374-454c-bbee-80e9403ff351-service-ca" (OuterVolumeSpecName: "service-ca") pod "abed6323-d374-454c-bbee-80e9403ff351" (UID: "abed6323-d374-454c-bbee-80e9403ff351"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 12 19:24:09.808803 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:24:09.808758 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/abed6323-d374-454c-bbee-80e9403ff351-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "abed6323-d374-454c-bbee-80e9403ff351" (UID: "abed6323-d374-454c-bbee-80e9403ff351"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 12 19:24:09.808927 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:24:09.808815 2565 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/abed6323-d374-454c-bbee-80e9403ff351-console-config\") on node \"ip-10-0-139-170.ec2.internal\" DevicePath \"\"" Jul 12 19:24:09.808927 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:24:09.808827 2565 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/abed6323-d374-454c-bbee-80e9403ff351-oauth-serving-cert\") on node \"ip-10-0-139-170.ec2.internal\" DevicePath \"\"" Jul 12 19:24:09.808927 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:24:09.808840 2565 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/abed6323-d374-454c-bbee-80e9403ff351-service-ca\") on node \"ip-10-0-139-170.ec2.internal\" DevicePath \"\"" Jul 12 19:24:09.808927 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:24:09.808851 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/abed6323-d374-454c-bbee-80e9403ff351-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "abed6323-d374-454c-bbee-80e9403ff351" (UID: "abed6323-d374-454c-bbee-80e9403ff351"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 12 19:24:09.810630 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:24:09.810603 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/abed6323-d374-454c-bbee-80e9403ff351-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "abed6323-d374-454c-bbee-80e9403ff351" (UID: "abed6323-d374-454c-bbee-80e9403ff351"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 12 19:24:09.810704 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:24:09.810660 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abed6323-d374-454c-bbee-80e9403ff351-kube-api-access-rg4tr" (OuterVolumeSpecName: "kube-api-access-rg4tr") pod "abed6323-d374-454c-bbee-80e9403ff351" (UID: "abed6323-d374-454c-bbee-80e9403ff351"). InnerVolumeSpecName "kube-api-access-rg4tr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 12 19:24:09.810740 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:24:09.810703 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/abed6323-d374-454c-bbee-80e9403ff351-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "abed6323-d374-454c-bbee-80e9403ff351" (UID: "abed6323-d374-454c-bbee-80e9403ff351"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 12 19:24:09.909462 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:24:09.909436 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rg4tr\" (UniqueName: \"kubernetes.io/projected/abed6323-d374-454c-bbee-80e9403ff351-kube-api-access-rg4tr\") on node \"ip-10-0-139-170.ec2.internal\" DevicePath \"\"" Jul 12 19:24:09.909462 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:24:09.909458 2565 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/abed6323-d374-454c-bbee-80e9403ff351-trusted-ca-bundle\") on node \"ip-10-0-139-170.ec2.internal\" DevicePath \"\"" Jul 12 19:24:09.909462 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:24:09.909467 2565 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/abed6323-d374-454c-bbee-80e9403ff351-console-serving-cert\") on node \"ip-10-0-139-170.ec2.internal\" DevicePath \"\"" Jul 12 19:24:09.909668 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:24:09.909476 2565 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/abed6323-d374-454c-bbee-80e9403ff351-console-oauth-config\") on node \"ip-10-0-139-170.ec2.internal\" DevicePath \"\"" Jul 12 19:24:10.501545 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:24:10.501509 2565 generic.go:356] "Generic (PLEG): container finished" podID="abed6323-d374-454c-bbee-80e9403ff351" containerID="afafeb2d77b14e69085ff278a2120d81cc433666f3376e6d1c55e16b4d1cd1e0" exitCode=0 Jul 12 19:24:10.501984 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:24:10.501560 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7b5759fdf9-9hlqp" Jul 12 19:24:10.501984 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:24:10.501578 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7b5759fdf9-9hlqp" event={"ID":"abed6323-d374-454c-bbee-80e9403ff351","Type":"ContainerDied","Data":"afafeb2d77b14e69085ff278a2120d81cc433666f3376e6d1c55e16b4d1cd1e0"} Jul 12 19:24:10.501984 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:24:10.501620 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7b5759fdf9-9hlqp" event={"ID":"abed6323-d374-454c-bbee-80e9403ff351","Type":"ContainerDied","Data":"e3fa1ad7b5a0401e1b46f7a533f1f635f2c562028ea4344fffa6958008f67475"} Jul 12 19:24:10.501984 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:24:10.501640 2565 scope.go:117] "RemoveContainer" containerID="afafeb2d77b14e69085ff278a2120d81cc433666f3376e6d1c55e16b4d1cd1e0" Jul 12 19:24:10.509617 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:24:10.509588 2565 scope.go:117] "RemoveContainer" containerID="afafeb2d77b14e69085ff278a2120d81cc433666f3376e6d1c55e16b4d1cd1e0" Jul 12 19:24:10.509848 ip-10-0-139-170 kubenswrapper[2565]: E0712 19:24:10.509832 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"afafeb2d77b14e69085ff278a2120d81cc433666f3376e6d1c55e16b4d1cd1e0\": container with ID starting with afafeb2d77b14e69085ff278a2120d81cc433666f3376e6d1c55e16b4d1cd1e0 not found: ID does not exist" containerID="afafeb2d77b14e69085ff278a2120d81cc433666f3376e6d1c55e16b4d1cd1e0" Jul 12 19:24:10.509938 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:24:10.509855 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"afafeb2d77b14e69085ff278a2120d81cc433666f3376e6d1c55e16b4d1cd1e0"} err="failed to get container status \"afafeb2d77b14e69085ff278a2120d81cc433666f3376e6d1c55e16b4d1cd1e0\": rpc error: code = NotFound desc = could not find container \"afafeb2d77b14e69085ff278a2120d81cc433666f3376e6d1c55e16b4d1cd1e0\": container with ID starting with afafeb2d77b14e69085ff278a2120d81cc433666f3376e6d1c55e16b4d1cd1e0 not found: ID does not exist" Jul 12 19:24:10.519098 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:24:10.519076 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7b5759fdf9-9hlqp"] Jul 12 19:24:10.521476 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:24:10.521458 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7b5759fdf9-9hlqp"] Jul 12 19:24:11.856764 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:24:11.856721 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="abed6323-d374-454c-bbee-80e9403ff351" path="/var/lib/kubelet/pods/abed6323-d374-454c-bbee-80e9403ff351/volumes" Jul 12 19:24:12.486910 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:24:12.486866 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-75f45dd54d-p5rzj" Jul 12 19:24:12.490678 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:24:12.490650 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-75f45dd54d-p5rzj" Jul 12 19:24:34.351429 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:24:34.351393 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 12 19:24:34.371852 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:24:34.371828 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 12 19:24:34.611161 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:24:34.611090 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 12 19:24:46.003307 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:24:46.003273 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:24:47.024613 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:24:47.024564 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:24:47.218173 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:24:47.218134 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:24:55.648129 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:24:55.648094 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-878fd74cf-dv9g6"] Jul 12 19:24:55.648513 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:24:55.648385 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dc6274fb-accb-4f81-ae6e-8133c6ed4132" containerName="console" Jul 12 19:24:55.648513 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:24:55.648403 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="dc6274fb-accb-4f81-ae6e-8133c6ed4132" containerName="console" Jul 12 19:24:55.648513 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:24:55.648413 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="abed6323-d374-454c-bbee-80e9403ff351" containerName="console" Jul 12 19:24:55.648513 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:24:55.648419 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="abed6323-d374-454c-bbee-80e9403ff351" containerName="console" Jul 12 19:24:55.648513 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:24:55.648465 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="dc6274fb-accb-4f81-ae6e-8133c6ed4132" containerName="console" Jul 12 19:24:55.648513 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:24:55.648474 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="abed6323-d374-454c-bbee-80e9403ff351" containerName="console" Jul 12 19:24:55.651482 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:24:55.651466 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-878fd74cf-dv9g6" Jul 12 19:24:55.661819 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:24:55.661792 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-878fd74cf-dv9g6"] Jul 12 19:24:55.703230 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:24:55.703189 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:24:55.757086 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:24:55.757048 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:24:55.777172 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:24:55.777149 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sq4ws\" (UniqueName: \"kubernetes.io/projected/a8adde5e-7b8f-43ff-ba96-67bbd1603d93-kube-api-access-sq4ws\") pod \"console-878fd74cf-dv9g6\" (UID: \"a8adde5e-7b8f-43ff-ba96-67bbd1603d93\") " pod="openshift-console/console-878fd74cf-dv9g6" Jul 12 19:24:55.777251 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:24:55.777183 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a8adde5e-7b8f-43ff-ba96-67bbd1603d93-console-config\") pod \"console-878fd74cf-dv9g6\" (UID: \"a8adde5e-7b8f-43ff-ba96-67bbd1603d93\") " pod="openshift-console/console-878fd74cf-dv9g6" Jul 12 19:24:55.777251 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:24:55.777208 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a8adde5e-7b8f-43ff-ba96-67bbd1603d93-console-oauth-config\") pod \"console-878fd74cf-dv9g6\" (UID: \"a8adde5e-7b8f-43ff-ba96-67bbd1603d93\") " pod="openshift-console/console-878fd74cf-dv9g6" Jul 12 19:24:55.777251 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:24:55.777228 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a8adde5e-7b8f-43ff-ba96-67bbd1603d93-oauth-serving-cert\") pod \"console-878fd74cf-dv9g6\" (UID: \"a8adde5e-7b8f-43ff-ba96-67bbd1603d93\") " pod="openshift-console/console-878fd74cf-dv9g6" Jul 12 19:24:55.777343 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:24:55.777292 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a8adde5e-7b8f-43ff-ba96-67bbd1603d93-trusted-ca-bundle\") pod \"console-878fd74cf-dv9g6\" (UID: \"a8adde5e-7b8f-43ff-ba96-67bbd1603d93\") " pod="openshift-console/console-878fd74cf-dv9g6" Jul 12 19:24:55.777375 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:24:55.777342 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a8adde5e-7b8f-43ff-ba96-67bbd1603d93-service-ca\") pod \"console-878fd74cf-dv9g6\" (UID: \"a8adde5e-7b8f-43ff-ba96-67bbd1603d93\") " pod="openshift-console/console-878fd74cf-dv9g6" Jul 12 19:24:55.777405 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:24:55.777387 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a8adde5e-7b8f-43ff-ba96-67bbd1603d93-console-serving-cert\") pod \"console-878fd74cf-dv9g6\" (UID: \"a8adde5e-7b8f-43ff-ba96-67bbd1603d93\") " pod="openshift-console/console-878fd74cf-dv9g6" Jul 12 19:24:55.850589 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:24:55.850555 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:24:55.877761 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:24:55.877735 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a8adde5e-7b8f-43ff-ba96-67bbd1603d93-service-ca\") pod \"console-878fd74cf-dv9g6\" (UID: \"a8adde5e-7b8f-43ff-ba96-67bbd1603d93\") " pod="openshift-console/console-878fd74cf-dv9g6" Jul 12 19:24:55.877949 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:24:55.877773 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a8adde5e-7b8f-43ff-ba96-67bbd1603d93-console-serving-cert\") pod \"console-878fd74cf-dv9g6\" (UID: \"a8adde5e-7b8f-43ff-ba96-67bbd1603d93\") " pod="openshift-console/console-878fd74cf-dv9g6" Jul 12 19:24:55.877949 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:24:55.877800 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sq4ws\" (UniqueName: \"kubernetes.io/projected/a8adde5e-7b8f-43ff-ba96-67bbd1603d93-kube-api-access-sq4ws\") pod \"console-878fd74cf-dv9g6\" (UID: \"a8adde5e-7b8f-43ff-ba96-67bbd1603d93\") " pod="openshift-console/console-878fd74cf-dv9g6" Jul 12 19:24:55.877949 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:24:55.877829 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a8adde5e-7b8f-43ff-ba96-67bbd1603d93-console-config\") pod \"console-878fd74cf-dv9g6\" (UID: \"a8adde5e-7b8f-43ff-ba96-67bbd1603d93\") " pod="openshift-console/console-878fd74cf-dv9g6" Jul 12 19:24:55.877949 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:24:55.877862 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a8adde5e-7b8f-43ff-ba96-67bbd1603d93-console-oauth-config\") pod \"console-878fd74cf-dv9g6\" (UID: \"a8adde5e-7b8f-43ff-ba96-67bbd1603d93\") " pod="openshift-console/console-878fd74cf-dv9g6" Jul 12 19:24:55.877949 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:24:55.877907 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a8adde5e-7b8f-43ff-ba96-67bbd1603d93-oauth-serving-cert\") pod \"console-878fd74cf-dv9g6\" (UID: \"a8adde5e-7b8f-43ff-ba96-67bbd1603d93\") " pod="openshift-console/console-878fd74cf-dv9g6" Jul 12 19:24:55.878203 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:24:55.877949 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a8adde5e-7b8f-43ff-ba96-67bbd1603d93-trusted-ca-bundle\") pod \"console-878fd74cf-dv9g6\" (UID: \"a8adde5e-7b8f-43ff-ba96-67bbd1603d93\") " pod="openshift-console/console-878fd74cf-dv9g6" Jul 12 19:24:55.878609 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:24:55.878580 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a8adde5e-7b8f-43ff-ba96-67bbd1603d93-console-config\") pod \"console-878fd74cf-dv9g6\" (UID: \"a8adde5e-7b8f-43ff-ba96-67bbd1603d93\") " pod="openshift-console/console-878fd74cf-dv9g6" Jul 12 19:24:55.878731 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:24:55.878580 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a8adde5e-7b8f-43ff-ba96-67bbd1603d93-service-ca\") pod \"console-878fd74cf-dv9g6\" (UID: \"a8adde5e-7b8f-43ff-ba96-67bbd1603d93\") " pod="openshift-console/console-878fd74cf-dv9g6" Jul 12 19:24:55.878731 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:24:55.878683 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a8adde5e-7b8f-43ff-ba96-67bbd1603d93-oauth-serving-cert\") pod \"console-878fd74cf-dv9g6\" (UID: \"a8adde5e-7b8f-43ff-ba96-67bbd1603d93\") " pod="openshift-console/console-878fd74cf-dv9g6" Jul 12 19:24:55.878731 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:24:55.878712 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a8adde5e-7b8f-43ff-ba96-67bbd1603d93-trusted-ca-bundle\") pod \"console-878fd74cf-dv9g6\" (UID: \"a8adde5e-7b8f-43ff-ba96-67bbd1603d93\") " pod="openshift-console/console-878fd74cf-dv9g6" Jul 12 19:24:55.880219 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:24:55.880203 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a8adde5e-7b8f-43ff-ba96-67bbd1603d93-console-serving-cert\") pod \"console-878fd74cf-dv9g6\" (UID: \"a8adde5e-7b8f-43ff-ba96-67bbd1603d93\") " pod="openshift-console/console-878fd74cf-dv9g6" Jul 12 19:24:55.880310 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:24:55.880251 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a8adde5e-7b8f-43ff-ba96-67bbd1603d93-console-oauth-config\") pod \"console-878fd74cf-dv9g6\" (UID: \"a8adde5e-7b8f-43ff-ba96-67bbd1603d93\") " pod="openshift-console/console-878fd74cf-dv9g6" Jul 12 19:24:55.885485 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:24:55.885464 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sq4ws\" (UniqueName: \"kubernetes.io/projected/a8adde5e-7b8f-43ff-ba96-67bbd1603d93-kube-api-access-sq4ws\") pod \"console-878fd74cf-dv9g6\" (UID: \"a8adde5e-7b8f-43ff-ba96-67bbd1603d93\") " pod="openshift-console/console-878fd74cf-dv9g6" Jul 12 19:24:55.901528 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:24:55.901477 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:24:55.960358 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:24:55.960322 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-878fd74cf-dv9g6" Jul 12 19:24:56.087802 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:24:56.087666 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-878fd74cf-dv9g6"] Jul 12 19:24:56.090376 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:24:56.090347 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8adde5e_7b8f_43ff_ba96_67bbd1603d93.slice/crio-355b70cbf7584690c31908a258d0a1950105077b9a53071042d572b913f2d148 WatchSource:0}: Error finding container 355b70cbf7584690c31908a258d0a1950105077b9a53071042d572b913f2d148: Status 404 returned error can't find the container with id 355b70cbf7584690c31908a258d0a1950105077b9a53071042d572b913f2d148 Jul 12 19:24:56.448058 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:24:56.448025 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:24:56.500220 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:24:56.500191 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:24:56.677543 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:24:56.677508 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-878fd74cf-dv9g6" event={"ID":"a8adde5e-7b8f-43ff-ba96-67bbd1603d93","Type":"ContainerStarted","Data":"c3c1622931f36cafc485fbeacfc328ad3ce63745b2ee0ed5f859f77c5d98933d"} Jul 12 19:24:56.678043 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:24:56.677563 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-878fd74cf-dv9g6" event={"ID":"a8adde5e-7b8f-43ff-ba96-67bbd1603d93","Type":"ContainerStarted","Data":"355b70cbf7584690c31908a258d0a1950105077b9a53071042d572b913f2d148"} Jul 12 19:24:56.694057 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:24:56.694004 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-878fd74cf-dv9g6" podStartSLOduration=1.693986647 podStartE2EDuration="1.693986647s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 19:24:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 19:24:56.692985901 +0000 UTC m=+161.425558393" watchObservedRunningTime="2026-07-12 19:24:56.693986647 +0000 UTC m=+161.426559137" Jul 12 19:25:02.219985 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:25:02.219947 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:25:02.318252 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:25:02.318220 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:25:05.961391 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:25:05.961353 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-878fd74cf-dv9g6" Jul 12 19:25:05.961859 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:25:05.961408 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-878fd74cf-dv9g6" Jul 12 19:25:05.966029 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:25:05.966006 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-878fd74cf-dv9g6" Jul 12 19:25:06.715871 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:25:06.715843 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-878fd74cf-dv9g6" Jul 12 19:25:06.756390 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:25:06.756359 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-65564fb587-24z67"] Jul 12 19:25:06.890523 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:25:06.890491 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-554ff9c978-8zmg2"] Jul 12 19:25:06.893593 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:25:06.893573 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-554ff9c978-8zmg2" Jul 12 19:25:06.896751 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:25:06.896718 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:25:06.902788 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:25:06.902764 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-554ff9c978-8zmg2"] Jul 12 19:25:06.946639 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:25:06.946610 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:25:07.068160 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:25:07.068080 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e1f9c14d-7d55-423b-802d-086f272ec50b-console-config\") pod \"console-554ff9c978-8zmg2\" (UID: \"e1f9c14d-7d55-423b-802d-086f272ec50b\") " pod="openshift-console/console-554ff9c978-8zmg2" Jul 12 19:25:07.068160 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:25:07.068116 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e1f9c14d-7d55-423b-802d-086f272ec50b-trusted-ca-bundle\") pod \"console-554ff9c978-8zmg2\" (UID: \"e1f9c14d-7d55-423b-802d-086f272ec50b\") " pod="openshift-console/console-554ff9c978-8zmg2" Jul 12 19:25:07.068160 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:25:07.068150 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e1f9c14d-7d55-423b-802d-086f272ec50b-oauth-serving-cert\") pod \"console-554ff9c978-8zmg2\" (UID: \"e1f9c14d-7d55-423b-802d-086f272ec50b\") " pod="openshift-console/console-554ff9c978-8zmg2" Jul 12 19:25:07.068557 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:25:07.068220 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6kht\" (UniqueName: \"kubernetes.io/projected/e1f9c14d-7d55-423b-802d-086f272ec50b-kube-api-access-b6kht\") pod \"console-554ff9c978-8zmg2\" (UID: \"e1f9c14d-7d55-423b-802d-086f272ec50b\") " pod="openshift-console/console-554ff9c978-8zmg2" Jul 12 19:25:07.068557 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:25:07.068280 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e1f9c14d-7d55-423b-802d-086f272ec50b-console-oauth-config\") pod \"console-554ff9c978-8zmg2\" (UID: \"e1f9c14d-7d55-423b-802d-086f272ec50b\") " pod="openshift-console/console-554ff9c978-8zmg2" Jul 12 19:25:07.068557 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:25:07.068320 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e1f9c14d-7d55-423b-802d-086f272ec50b-service-ca\") pod \"console-554ff9c978-8zmg2\" (UID: \"e1f9c14d-7d55-423b-802d-086f272ec50b\") " pod="openshift-console/console-554ff9c978-8zmg2" Jul 12 19:25:07.068557 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:25:07.068370 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e1f9c14d-7d55-423b-802d-086f272ec50b-console-serving-cert\") pod \"console-554ff9c978-8zmg2\" (UID: \"e1f9c14d-7d55-423b-802d-086f272ec50b\") " pod="openshift-console/console-554ff9c978-8zmg2" Jul 12 19:25:07.169726 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:25:07.169690 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e1f9c14d-7d55-423b-802d-086f272ec50b-console-serving-cert\") pod \"console-554ff9c978-8zmg2\" (UID: \"e1f9c14d-7d55-423b-802d-086f272ec50b\") " pod="openshift-console/console-554ff9c978-8zmg2" Jul 12 19:25:07.169726 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:25:07.169731 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e1f9c14d-7d55-423b-802d-086f272ec50b-console-config\") pod \"console-554ff9c978-8zmg2\" (UID: \"e1f9c14d-7d55-423b-802d-086f272ec50b\") " pod="openshift-console/console-554ff9c978-8zmg2" Jul 12 19:25:07.169981 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:25:07.169751 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e1f9c14d-7d55-423b-802d-086f272ec50b-trusted-ca-bundle\") pod \"console-554ff9c978-8zmg2\" (UID: \"e1f9c14d-7d55-423b-802d-086f272ec50b\") " pod="openshift-console/console-554ff9c978-8zmg2" Jul 12 19:25:07.169981 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:25:07.169798 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e1f9c14d-7d55-423b-802d-086f272ec50b-oauth-serving-cert\") pod \"console-554ff9c978-8zmg2\" (UID: \"e1f9c14d-7d55-423b-802d-086f272ec50b\") " pod="openshift-console/console-554ff9c978-8zmg2" Jul 12 19:25:07.169981 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:25:07.169839 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6kht\" (UniqueName: \"kubernetes.io/projected/e1f9c14d-7d55-423b-802d-086f272ec50b-kube-api-access-b6kht\") pod \"console-554ff9c978-8zmg2\" (UID: \"e1f9c14d-7d55-423b-802d-086f272ec50b\") " pod="openshift-console/console-554ff9c978-8zmg2" Jul 12 19:25:07.169981 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:25:07.169878 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e1f9c14d-7d55-423b-802d-086f272ec50b-console-oauth-config\") pod \"console-554ff9c978-8zmg2\" (UID: \"e1f9c14d-7d55-423b-802d-086f272ec50b\") " pod="openshift-console/console-554ff9c978-8zmg2" Jul 12 19:25:07.169981 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:25:07.169925 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e1f9c14d-7d55-423b-802d-086f272ec50b-service-ca\") pod \"console-554ff9c978-8zmg2\" (UID: \"e1f9c14d-7d55-423b-802d-086f272ec50b\") " pod="openshift-console/console-554ff9c978-8zmg2" Jul 12 19:25:07.170605 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:25:07.170575 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e1f9c14d-7d55-423b-802d-086f272ec50b-console-config\") pod \"console-554ff9c978-8zmg2\" (UID: \"e1f9c14d-7d55-423b-802d-086f272ec50b\") " pod="openshift-console/console-554ff9c978-8zmg2" Jul 12 19:25:07.170741 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:25:07.170584 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e1f9c14d-7d55-423b-802d-086f272ec50b-oauth-serving-cert\") pod \"console-554ff9c978-8zmg2\" (UID: \"e1f9c14d-7d55-423b-802d-086f272ec50b\") " pod="openshift-console/console-554ff9c978-8zmg2" Jul 12 19:25:07.170741 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:25:07.170665 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e1f9c14d-7d55-423b-802d-086f272ec50b-service-ca\") pod \"console-554ff9c978-8zmg2\" (UID: \"e1f9c14d-7d55-423b-802d-086f272ec50b\") " pod="openshift-console/console-554ff9c978-8zmg2" Jul 12 19:25:07.170874 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:25:07.170852 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e1f9c14d-7d55-423b-802d-086f272ec50b-trusted-ca-bundle\") pod \"console-554ff9c978-8zmg2\" (UID: \"e1f9c14d-7d55-423b-802d-086f272ec50b\") " pod="openshift-console/console-554ff9c978-8zmg2" Jul 12 19:25:07.172624 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:25:07.172600 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e1f9c14d-7d55-423b-802d-086f272ec50b-console-oauth-config\") pod \"console-554ff9c978-8zmg2\" (UID: \"e1f9c14d-7d55-423b-802d-086f272ec50b\") " pod="openshift-console/console-554ff9c978-8zmg2" Jul 12 19:25:07.172707 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:25:07.172689 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e1f9c14d-7d55-423b-802d-086f272ec50b-console-serving-cert\") pod \"console-554ff9c978-8zmg2\" (UID: \"e1f9c14d-7d55-423b-802d-086f272ec50b\") " pod="openshift-console/console-554ff9c978-8zmg2" Jul 12 19:25:07.179685 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:25:07.179658 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6kht\" (UniqueName: \"kubernetes.io/projected/e1f9c14d-7d55-423b-802d-086f272ec50b-kube-api-access-b6kht\") pod \"console-554ff9c978-8zmg2\" (UID: \"e1f9c14d-7d55-423b-802d-086f272ec50b\") " pod="openshift-console/console-554ff9c978-8zmg2" Jul 12 19:25:07.203046 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:25:07.203007 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:25:07.203195 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:25:07.203188 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-554ff9c978-8zmg2" Jul 12 19:25:07.252360 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:25:07.252327 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:25:07.350454 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:25:07.350366 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-554ff9c978-8zmg2"] Jul 12 19:25:07.353121 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:25:07.353095 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1f9c14d_7d55_423b_802d_086f272ec50b.slice/crio-2541486d1cbead7266992d508b5db2d310f0493ab3ccc10e4cb1e3c7646ab6e8 WatchSource:0}: Error finding container 2541486d1cbead7266992d508b5db2d310f0493ab3ccc10e4cb1e3c7646ab6e8: Status 404 returned error can't find the container with id 2541486d1cbead7266992d508b5db2d310f0493ab3ccc10e4cb1e3c7646ab6e8 Jul 12 19:25:07.717149 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:25:07.717105 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-554ff9c978-8zmg2" event={"ID":"e1f9c14d-7d55-423b-802d-086f272ec50b","Type":"ContainerStarted","Data":"bd70ea03af01f2987f13887e795f0c49c028659beb515fd5a4397af48c8c9005"} Jul 12 19:25:07.717338 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:25:07.717190 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-554ff9c978-8zmg2" event={"ID":"e1f9c14d-7d55-423b-802d-086f272ec50b","Type":"ContainerStarted","Data":"2541486d1cbead7266992d508b5db2d310f0493ab3ccc10e4cb1e3c7646ab6e8"} Jul 12 19:25:07.733845 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:25:07.733793 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-554ff9c978-8zmg2" podStartSLOduration=1.733778891 podStartE2EDuration="1.733778891s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 19:25:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 19:25:07.732284814 +0000 UTC m=+172.464857305" watchObservedRunningTime="2026-07-12 19:25:07.733778891 +0000 UTC m=+172.466351382" Jul 12 19:25:07.804679 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:25:07.804646 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:25:08.801831 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:25:08.801794 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:25:14.168647 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:25:14.168617 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:25:14.269745 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:25:14.269709 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:25:15.148661 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:25:15.148624 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:25:17.203606 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:25:17.203567 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-554ff9c978-8zmg2" Jul 12 19:25:17.204142 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:25:17.203657 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-554ff9c978-8zmg2" Jul 12 19:25:17.208334 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:25:17.208311 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-554ff9c978-8zmg2" Jul 12 19:25:17.762707 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:25:17.762677 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-554ff9c978-8zmg2" Jul 12 19:25:17.803748 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:25:17.803717 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-878fd74cf-dv9g6"] Jul 12 19:25:17.902626 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:25:17.902596 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:25:17.952847 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:25:17.952822 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:25:18.003030 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:25:18.002984 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:25:21.668090 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:25:21.668057 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:25:21.762044 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:25:21.762009 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:25:24.732623 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:25:24.732588 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:25:31.778169 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:25:31.778128 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-65564fb587-24z67" podUID="c00c020c-dd2b-45bd-a69c-7600f9bec905" containerName="console" containerID="cri-o://62732d3361becbf94146b0de33f251bb4b282b662bbf1df9d8b4d306a1ed84fc" gracePeriod=15 Jul 12 19:25:32.027598 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:25:32.027576 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-65564fb587-24z67" Jul 12 19:25:32.079399 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:25:32.079324 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c00c020c-dd2b-45bd-a69c-7600f9bec905-oauth-serving-cert\") pod \"c00c020c-dd2b-45bd-a69c-7600f9bec905\" (UID: \"c00c020c-dd2b-45bd-a69c-7600f9bec905\") " Jul 12 19:25:32.079399 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:25:32.079363 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c00c020c-dd2b-45bd-a69c-7600f9bec905-service-ca\") pod \"c00c020c-dd2b-45bd-a69c-7600f9bec905\" (UID: \"c00c020c-dd2b-45bd-a69c-7600f9bec905\") " Jul 12 19:25:32.079399 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:25:32.079396 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c00c020c-dd2b-45bd-a69c-7600f9bec905-console-oauth-config\") pod \"c00c020c-dd2b-45bd-a69c-7600f9bec905\" (UID: \"c00c020c-dd2b-45bd-a69c-7600f9bec905\") " Jul 12 19:25:32.079621 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:25:32.079441 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-crxmq\" (UniqueName: \"kubernetes.io/projected/c00c020c-dd2b-45bd-a69c-7600f9bec905-kube-api-access-crxmq\") pod \"c00c020c-dd2b-45bd-a69c-7600f9bec905\" (UID: \"c00c020c-dd2b-45bd-a69c-7600f9bec905\") " Jul 12 19:25:32.079621 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:25:32.079476 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c00c020c-dd2b-45bd-a69c-7600f9bec905-console-config\") pod \"c00c020c-dd2b-45bd-a69c-7600f9bec905\" (UID: \"c00c020c-dd2b-45bd-a69c-7600f9bec905\") " Jul 12 19:25:32.079621 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:25:32.079493 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c00c020c-dd2b-45bd-a69c-7600f9bec905-trusted-ca-bundle\") pod \"c00c020c-dd2b-45bd-a69c-7600f9bec905\" (UID: \"c00c020c-dd2b-45bd-a69c-7600f9bec905\") " Jul 12 19:25:32.079621 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:25:32.079520 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c00c020c-dd2b-45bd-a69c-7600f9bec905-console-serving-cert\") pod \"c00c020c-dd2b-45bd-a69c-7600f9bec905\" (UID: \"c00c020c-dd2b-45bd-a69c-7600f9bec905\") " Jul 12 19:25:32.079793 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:25:32.079689 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c00c020c-dd2b-45bd-a69c-7600f9bec905-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "c00c020c-dd2b-45bd-a69c-7600f9bec905" (UID: "c00c020c-dd2b-45bd-a69c-7600f9bec905"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 12 19:25:32.079852 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:25:32.079804 2565 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c00c020c-dd2b-45bd-a69c-7600f9bec905-oauth-serving-cert\") on node \"ip-10-0-139-170.ec2.internal\" DevicePath \"\"" Jul 12 19:25:32.079964 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:25:32.079937 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c00c020c-dd2b-45bd-a69c-7600f9bec905-console-config" (OuterVolumeSpecName: "console-config") pod "c00c020c-dd2b-45bd-a69c-7600f9bec905" (UID: "c00c020c-dd2b-45bd-a69c-7600f9bec905"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 12 19:25:32.079964 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:25:32.079949 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c00c020c-dd2b-45bd-a69c-7600f9bec905-service-ca" (OuterVolumeSpecName: "service-ca") pod "c00c020c-dd2b-45bd-a69c-7600f9bec905" (UID: "c00c020c-dd2b-45bd-a69c-7600f9bec905"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 12 19:25:32.080075 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:25:32.080028 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c00c020c-dd2b-45bd-a69c-7600f9bec905-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "c00c020c-dd2b-45bd-a69c-7600f9bec905" (UID: "c00c020c-dd2b-45bd-a69c-7600f9bec905"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 12 19:25:32.081510 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:25:32.081484 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c00c020c-dd2b-45bd-a69c-7600f9bec905-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "c00c020c-dd2b-45bd-a69c-7600f9bec905" (UID: "c00c020c-dd2b-45bd-a69c-7600f9bec905"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 12 19:25:32.081630 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:25:32.081558 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c00c020c-dd2b-45bd-a69c-7600f9bec905-kube-api-access-crxmq" (OuterVolumeSpecName: "kube-api-access-crxmq") pod "c00c020c-dd2b-45bd-a69c-7600f9bec905" (UID: "c00c020c-dd2b-45bd-a69c-7600f9bec905"). InnerVolumeSpecName "kube-api-access-crxmq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 12 19:25:32.081687 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:25:32.081644 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c00c020c-dd2b-45bd-a69c-7600f9bec905-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "c00c020c-dd2b-45bd-a69c-7600f9bec905" (UID: "c00c020c-dd2b-45bd-a69c-7600f9bec905"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 12 19:25:32.180767 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:25:32.180727 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-crxmq\" (UniqueName: \"kubernetes.io/projected/c00c020c-dd2b-45bd-a69c-7600f9bec905-kube-api-access-crxmq\") on node \"ip-10-0-139-170.ec2.internal\" DevicePath \"\"" Jul 12 19:25:32.180767 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:25:32.180762 2565 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c00c020c-dd2b-45bd-a69c-7600f9bec905-console-config\") on node \"ip-10-0-139-170.ec2.internal\" DevicePath \"\"" Jul 12 19:25:32.180767 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:25:32.180773 2565 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c00c020c-dd2b-45bd-a69c-7600f9bec905-trusted-ca-bundle\") on node \"ip-10-0-139-170.ec2.internal\" DevicePath \"\"" Jul 12 19:25:32.181039 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:25:32.180786 2565 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c00c020c-dd2b-45bd-a69c-7600f9bec905-console-serving-cert\") on node \"ip-10-0-139-170.ec2.internal\" DevicePath \"\"" Jul 12 19:25:32.181039 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:25:32.180798 2565 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c00c020c-dd2b-45bd-a69c-7600f9bec905-service-ca\") on node \"ip-10-0-139-170.ec2.internal\" DevicePath \"\"" Jul 12 19:25:32.181039 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:25:32.180809 2565 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c00c020c-dd2b-45bd-a69c-7600f9bec905-console-oauth-config\") on node \"ip-10-0-139-170.ec2.internal\" DevicePath \"\"" Jul 12 19:25:32.814066 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:25:32.814028 2565 generic.go:356] "Generic (PLEG): container finished" podID="c00c020c-dd2b-45bd-a69c-7600f9bec905" containerID="62732d3361becbf94146b0de33f251bb4b282b662bbf1df9d8b4d306a1ed84fc" exitCode=0 Jul 12 19:25:32.814066 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:25:32.814077 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-65564fb587-24z67" Jul 12 19:25:32.814510 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:25:32.814084 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-65564fb587-24z67" event={"ID":"c00c020c-dd2b-45bd-a69c-7600f9bec905","Type":"ContainerDied","Data":"62732d3361becbf94146b0de33f251bb4b282b662bbf1df9d8b4d306a1ed84fc"} Jul 12 19:25:32.814510 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:25:32.814132 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-65564fb587-24z67" event={"ID":"c00c020c-dd2b-45bd-a69c-7600f9bec905","Type":"ContainerDied","Data":"d3657ec129153fa08af54d9df06f095b9f2b6e4fc1abbfed68d6db0f9cfbc684"} Jul 12 19:25:32.814510 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:25:32.814157 2565 scope.go:117] "RemoveContainer" containerID="62732d3361becbf94146b0de33f251bb4b282b662bbf1df9d8b4d306a1ed84fc" Jul 12 19:25:32.822177 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:25:32.822159 2565 scope.go:117] "RemoveContainer" containerID="62732d3361becbf94146b0de33f251bb4b282b662bbf1df9d8b4d306a1ed84fc" Jul 12 19:25:32.822451 ip-10-0-139-170 kubenswrapper[2565]: E0712 19:25:32.822433 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62732d3361becbf94146b0de33f251bb4b282b662bbf1df9d8b4d306a1ed84fc\": container with ID starting with 62732d3361becbf94146b0de33f251bb4b282b662bbf1df9d8b4d306a1ed84fc not found: ID does not exist" containerID="62732d3361becbf94146b0de33f251bb4b282b662bbf1df9d8b4d306a1ed84fc" Jul 12 19:25:32.822497 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:25:32.822460 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62732d3361becbf94146b0de33f251bb4b282b662bbf1df9d8b4d306a1ed84fc"} err="failed to get container status \"62732d3361becbf94146b0de33f251bb4b282b662bbf1df9d8b4d306a1ed84fc\": rpc error: code = NotFound desc = could not find container \"62732d3361becbf94146b0de33f251bb4b282b662bbf1df9d8b4d306a1ed84fc\": container with ID starting with 62732d3361becbf94146b0de33f251bb4b282b662bbf1df9d8b4d306a1ed84fc not found: ID does not exist" Jul 12 19:25:32.833402 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:25:32.833383 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-65564fb587-24z67"] Jul 12 19:25:32.836781 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:25:32.836761 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-65564fb587-24z67"] Jul 12 19:25:33.861063 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:25:33.861027 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c00c020c-dd2b-45bd-a69c-7600f9bec905" path="/var/lib/kubelet/pods/c00c020c-dd2b-45bd-a69c-7600f9bec905/volumes" Jul 12 19:25:40.041544 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:25:40.041506 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:25:40.141642 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:25:40.141602 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:25:41.342600 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:25:41.342566 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:25:41.393016 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:25:41.392979 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:25:42.824491 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:25:42.824444 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-878fd74cf-dv9g6" podUID="a8adde5e-7b8f-43ff-ba96-67bbd1603d93" containerName="console" containerID="cri-o://c3c1622931f36cafc485fbeacfc328ad3ce63745b2ee0ed5f859f77c5d98933d" gracePeriod=15 Jul 12 19:25:43.069250 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:25:43.069224 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-878fd74cf-dv9g6" Jul 12 19:25:43.181022 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:25:43.180989 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a8adde5e-7b8f-43ff-ba96-67bbd1603d93-console-config\") pod \"a8adde5e-7b8f-43ff-ba96-67bbd1603d93\" (UID: \"a8adde5e-7b8f-43ff-ba96-67bbd1603d93\") " Jul 12 19:25:43.181163 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:25:43.181053 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sq4ws\" (UniqueName: \"kubernetes.io/projected/a8adde5e-7b8f-43ff-ba96-67bbd1603d93-kube-api-access-sq4ws\") pod \"a8adde5e-7b8f-43ff-ba96-67bbd1603d93\" (UID: \"a8adde5e-7b8f-43ff-ba96-67bbd1603d93\") " Jul 12 19:25:43.181163 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:25:43.181077 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a8adde5e-7b8f-43ff-ba96-67bbd1603d93-trusted-ca-bundle\") pod \"a8adde5e-7b8f-43ff-ba96-67bbd1603d93\" (UID: \"a8adde5e-7b8f-43ff-ba96-67bbd1603d93\") " Jul 12 19:25:43.181253 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:25:43.181194 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a8adde5e-7b8f-43ff-ba96-67bbd1603d93-oauth-serving-cert\") pod \"a8adde5e-7b8f-43ff-ba96-67bbd1603d93\" (UID: \"a8adde5e-7b8f-43ff-ba96-67bbd1603d93\") " Jul 12 19:25:43.181253 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:25:43.181235 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a8adde5e-7b8f-43ff-ba96-67bbd1603d93-console-serving-cert\") pod \"a8adde5e-7b8f-43ff-ba96-67bbd1603d93\" (UID: \"a8adde5e-7b8f-43ff-ba96-67bbd1603d93\") " Jul 12 19:25:43.181328 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:25:43.181266 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a8adde5e-7b8f-43ff-ba96-67bbd1603d93-service-ca\") pod \"a8adde5e-7b8f-43ff-ba96-67bbd1603d93\" (UID: \"a8adde5e-7b8f-43ff-ba96-67bbd1603d93\") " Jul 12 19:25:43.181372 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:25:43.181339 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a8adde5e-7b8f-43ff-ba96-67bbd1603d93-console-oauth-config\") pod \"a8adde5e-7b8f-43ff-ba96-67bbd1603d93\" (UID: \"a8adde5e-7b8f-43ff-ba96-67bbd1603d93\") " Jul 12 19:25:43.181553 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:25:43.181527 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8adde5e-7b8f-43ff-ba96-67bbd1603d93-console-config" (OuterVolumeSpecName: "console-config") pod "a8adde5e-7b8f-43ff-ba96-67bbd1603d93" (UID: "a8adde5e-7b8f-43ff-ba96-67bbd1603d93"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 12 19:25:43.181631 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:25:43.181569 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8adde5e-7b8f-43ff-ba96-67bbd1603d93-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "a8adde5e-7b8f-43ff-ba96-67bbd1603d93" (UID: "a8adde5e-7b8f-43ff-ba96-67bbd1603d93"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 12 19:25:43.181631 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:25:43.181614 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8adde5e-7b8f-43ff-ba96-67bbd1603d93-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "a8adde5e-7b8f-43ff-ba96-67bbd1603d93" (UID: "a8adde5e-7b8f-43ff-ba96-67bbd1603d93"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 12 19:25:43.181723 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:25:43.181674 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8adde5e-7b8f-43ff-ba96-67bbd1603d93-service-ca" (OuterVolumeSpecName: "service-ca") pod "a8adde5e-7b8f-43ff-ba96-67bbd1603d93" (UID: "a8adde5e-7b8f-43ff-ba96-67bbd1603d93"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 12 19:25:43.183340 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:25:43.183311 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8adde5e-7b8f-43ff-ba96-67bbd1603d93-kube-api-access-sq4ws" (OuterVolumeSpecName: "kube-api-access-sq4ws") pod "a8adde5e-7b8f-43ff-ba96-67bbd1603d93" (UID: "a8adde5e-7b8f-43ff-ba96-67bbd1603d93"). InnerVolumeSpecName "kube-api-access-sq4ws". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 12 19:25:43.183340 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:25:43.183311 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8adde5e-7b8f-43ff-ba96-67bbd1603d93-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "a8adde5e-7b8f-43ff-ba96-67bbd1603d93" (UID: "a8adde5e-7b8f-43ff-ba96-67bbd1603d93"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 12 19:25:43.183472 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:25:43.183334 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8adde5e-7b8f-43ff-ba96-67bbd1603d93-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "a8adde5e-7b8f-43ff-ba96-67bbd1603d93" (UID: "a8adde5e-7b8f-43ff-ba96-67bbd1603d93"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 12 19:25:43.282577 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:25:43.282538 2565 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a8adde5e-7b8f-43ff-ba96-67bbd1603d93-console-serving-cert\") on node \"ip-10-0-139-170.ec2.internal\" DevicePath \"\"" Jul 12 19:25:43.282577 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:25:43.282570 2565 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a8adde5e-7b8f-43ff-ba96-67bbd1603d93-service-ca\") on node \"ip-10-0-139-170.ec2.internal\" DevicePath \"\"" Jul 12 19:25:43.282577 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:25:43.282579 2565 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a8adde5e-7b8f-43ff-ba96-67bbd1603d93-console-oauth-config\") on node \"ip-10-0-139-170.ec2.internal\" DevicePath \"\"" Jul 12 19:25:43.282577 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:25:43.282587 2565 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a8adde5e-7b8f-43ff-ba96-67bbd1603d93-console-config\") on node \"ip-10-0-139-170.ec2.internal\" DevicePath \"\"" Jul 12 19:25:43.282823 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:25:43.282596 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sq4ws\" (UniqueName: \"kubernetes.io/projected/a8adde5e-7b8f-43ff-ba96-67bbd1603d93-kube-api-access-sq4ws\") on node \"ip-10-0-139-170.ec2.internal\" DevicePath \"\"" Jul 12 19:25:43.282823 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:25:43.282604 2565 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a8adde5e-7b8f-43ff-ba96-67bbd1603d93-trusted-ca-bundle\") on node \"ip-10-0-139-170.ec2.internal\" DevicePath \"\"" Jul 12 19:25:43.282823 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:25:43.282612 2565 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a8adde5e-7b8f-43ff-ba96-67bbd1603d93-oauth-serving-cert\") on node \"ip-10-0-139-170.ec2.internal\" DevicePath \"\"" Jul 12 19:25:43.855494 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:25:43.855461 2565 generic.go:356] "Generic (PLEG): container finished" podID="a8adde5e-7b8f-43ff-ba96-67bbd1603d93" containerID="c3c1622931f36cafc485fbeacfc328ad3ce63745b2ee0ed5f859f77c5d98933d" exitCode=0 Jul 12 19:25:43.855858 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:25:43.855536 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-878fd74cf-dv9g6" Jul 12 19:25:43.856067 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:25:43.856041 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-878fd74cf-dv9g6" event={"ID":"a8adde5e-7b8f-43ff-ba96-67bbd1603d93","Type":"ContainerDied","Data":"c3c1622931f36cafc485fbeacfc328ad3ce63745b2ee0ed5f859f77c5d98933d"} Jul 12 19:25:43.856110 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:25:43.856091 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-878fd74cf-dv9g6" event={"ID":"a8adde5e-7b8f-43ff-ba96-67bbd1603d93","Type":"ContainerDied","Data":"355b70cbf7584690c31908a258d0a1950105077b9a53071042d572b913f2d148"} Jul 12 19:25:43.856147 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:25:43.856120 2565 scope.go:117] "RemoveContainer" containerID="c3c1622931f36cafc485fbeacfc328ad3ce63745b2ee0ed5f859f77c5d98933d" Jul 12 19:25:43.864232 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:25:43.864211 2565 scope.go:117] "RemoveContainer" containerID="c3c1622931f36cafc485fbeacfc328ad3ce63745b2ee0ed5f859f77c5d98933d" Jul 12 19:25:43.864671 ip-10-0-139-170 kubenswrapper[2565]: E0712 19:25:43.864494 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3c1622931f36cafc485fbeacfc328ad3ce63745b2ee0ed5f859f77c5d98933d\": container with ID starting with c3c1622931f36cafc485fbeacfc328ad3ce63745b2ee0ed5f859f77c5d98933d not found: ID does not exist" containerID="c3c1622931f36cafc485fbeacfc328ad3ce63745b2ee0ed5f859f77c5d98933d" Jul 12 19:25:43.864671 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:25:43.864521 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3c1622931f36cafc485fbeacfc328ad3ce63745b2ee0ed5f859f77c5d98933d"} err="failed to get container status \"c3c1622931f36cafc485fbeacfc328ad3ce63745b2ee0ed5f859f77c5d98933d\": rpc error: code = NotFound desc = could not find container \"c3c1622931f36cafc485fbeacfc328ad3ce63745b2ee0ed5f859f77c5d98933d\": container with ID starting with c3c1622931f36cafc485fbeacfc328ad3ce63745b2ee0ed5f859f77c5d98933d not found: ID does not exist" Jul 12 19:25:43.877856 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:25:43.877824 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-878fd74cf-dv9g6"] Jul 12 19:25:43.881301 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:25:43.881279 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-878fd74cf-dv9g6"] Jul 12 19:25:45.856542 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:25:45.856507 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a8adde5e-7b8f-43ff-ba96-67bbd1603d93" path="/var/lib/kubelet/pods/a8adde5e-7b8f-43ff-ba96-67bbd1603d93/volumes" Jul 12 19:25:56.934653 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:25:56.934618 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:26:01.310970 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:26:01.310938 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:26:02.673117 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:26:02.673079 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:26:04.911444 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:26:04.911410 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:26:04.964482 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:26:04.964447 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:26:06.461370 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:26:06.461338 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:26:06.567752 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:26:06.567717 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:26:08.796043 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:26:08.796015 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:26:09.206237 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:26:09.206203 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:26:11.721958 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:26:11.721924 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:26:11.822199 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:26:11.822173 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:26:14.582158 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:26:14.582123 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-j7fwj"] Jul 12 19:26:14.582514 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:26:14.582403 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a8adde5e-7b8f-43ff-ba96-67bbd1603d93" containerName="console" Jul 12 19:26:14.582514 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:26:14.582414 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="a8adde5e-7b8f-43ff-ba96-67bbd1603d93" containerName="console" Jul 12 19:26:14.582514 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:26:14.582426 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c00c020c-dd2b-45bd-a69c-7600f9bec905" containerName="console" Jul 12 19:26:14.582514 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:26:14.582431 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="c00c020c-dd2b-45bd-a69c-7600f9bec905" containerName="console" Jul 12 19:26:14.582514 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:26:14.582478 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="a8adde5e-7b8f-43ff-ba96-67bbd1603d93" containerName="console" Jul 12 19:26:14.582514 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:26:14.582486 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="c00c020c-dd2b-45bd-a69c-7600f9bec905" containerName="console" Jul 12 19:26:14.585378 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:26:14.585363 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-j7fwj" Jul 12 19:26:14.587605 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:26:14.587582 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 12 19:26:14.587734 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:26:14.587615 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 12 19:26:14.588371 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:26:14.588359 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-hnfr4\"" Jul 12 19:26:14.594472 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:26:14.594451 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-j7fwj"] Jul 12 19:26:14.627352 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:26:14.627329 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/16a6250d-2928-4b32-b29b-70911da322bd-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-j7fwj\" (UID: \"16a6250d-2928-4b32-b29b-70911da322bd\") " pod="cert-manager/cert-manager-webhook-6486ffd997-j7fwj" Jul 12 19:26:14.627472 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:26:14.627360 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wcjdm\" (UniqueName: \"kubernetes.io/projected/16a6250d-2928-4b32-b29b-70911da322bd-kube-api-access-wcjdm\") pod \"cert-manager-webhook-6486ffd997-j7fwj\" (UID: \"16a6250d-2928-4b32-b29b-70911da322bd\") " pod="cert-manager/cert-manager-webhook-6486ffd997-j7fwj" Jul 12 19:26:14.728322 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:26:14.728285 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/16a6250d-2928-4b32-b29b-70911da322bd-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-j7fwj\" (UID: \"16a6250d-2928-4b32-b29b-70911da322bd\") " pod="cert-manager/cert-manager-webhook-6486ffd997-j7fwj" Jul 12 19:26:14.728322 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:26:14.728322 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wcjdm\" (UniqueName: \"kubernetes.io/projected/16a6250d-2928-4b32-b29b-70911da322bd-kube-api-access-wcjdm\") pod \"cert-manager-webhook-6486ffd997-j7fwj\" (UID: \"16a6250d-2928-4b32-b29b-70911da322bd\") " pod="cert-manager/cert-manager-webhook-6486ffd997-j7fwj" Jul 12 19:26:14.735860 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:26:14.735826 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/16a6250d-2928-4b32-b29b-70911da322bd-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-j7fwj\" (UID: \"16a6250d-2928-4b32-b29b-70911da322bd\") " pod="cert-manager/cert-manager-webhook-6486ffd997-j7fwj" Jul 12 19:26:14.735990 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:26:14.735827 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wcjdm\" (UniqueName: \"kubernetes.io/projected/16a6250d-2928-4b32-b29b-70911da322bd-kube-api-access-wcjdm\") pod \"cert-manager-webhook-6486ffd997-j7fwj\" (UID: \"16a6250d-2928-4b32-b29b-70911da322bd\") " pod="cert-manager/cert-manager-webhook-6486ffd997-j7fwj" Jul 12 19:26:14.903736 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:26:14.903685 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-j7fwj" Jul 12 19:26:15.024924 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:26:15.024830 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-j7fwj"] Jul 12 19:26:15.027375 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:26:15.027340 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod16a6250d_2928_4b32_b29b_70911da322bd.slice/crio-d6283f79801b6c1df0fb58e88c3c6ce07f763a651b21cf2c54bdeda80a676aaf WatchSource:0}: Error finding container d6283f79801b6c1df0fb58e88c3c6ce07f763a651b21cf2c54bdeda80a676aaf: Status 404 returned error can't find the container with id d6283f79801b6c1df0fb58e88c3c6ce07f763a651b21cf2c54bdeda80a676aaf Jul 12 19:26:15.971201 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:26:15.971095 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-j7fwj" event={"ID":"16a6250d-2928-4b32-b29b-70911da322bd","Type":"ContainerStarted","Data":"d6283f79801b6c1df0fb58e88c3c6ce07f763a651b21cf2c54bdeda80a676aaf"} Jul 12 19:26:16.209681 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:26:16.209594 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:26:17.979287 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:26:17.979251 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-j7fwj" event={"ID":"16a6250d-2928-4b32-b29b-70911da322bd","Type":"ContainerStarted","Data":"51a3b70bfe8ff07f5e6c3a7262030a7e1caa429ebb15c2bbb2453f56bc1f3805"} Jul 12 19:26:17.979664 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:26:17.979332 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-j7fwj" Jul 12 19:26:17.994471 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:26:17.994428 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-j7fwj" podStartSLOduration=1.264453388 podStartE2EDuration="3.994415305s" totalImagesPullingTime="2.729961917s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 19:26:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 19:26:17.993475791 +0000 UTC m=+242.726048293" watchObservedRunningTime="2026-07-12 19:26:17.994415305 +0000 UTC m=+242.726987794" Jul 12 19:26:18.005315 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:26:18.005285 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:26:18.059597 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:26:18.059531 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:26:23.126266 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:26:23.126231 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:26:23.226058 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:26:23.226022 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:26:23.985915 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:26:23.985868 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-j7fwj" Jul 12 19:26:28.994589 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:26:28.994551 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:26:29.095270 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:26:29.095238 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:26:30.861141 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:26:30.861110 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:26:45.330164 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:26:45.330080 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:26:45.428505 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:26:45.428473 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:26:45.972175 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:26:45.972136 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:26:46.141123 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:26:46.141092 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:26:46.248501 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:26:46.248408 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:26:46.279965 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:26:46.279938 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5449d7969f-kr7sd"] Jul 12 19:26:46.284109 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:26:46.284085 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5449d7969f-kr7sd" Jul 12 19:26:46.287119 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:26:46.287085 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 12 19:26:46.287119 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:26:46.287116 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 12 19:26:46.287292 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:26:46.287144 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-4nbfd\"" Jul 12 19:26:46.287292 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:26:46.287086 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 12 19:26:46.287537 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:26:46.287519 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 12 19:26:46.377579 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:26:46.377543 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5449d7969f-kr7sd"] Jul 12 19:26:46.382040 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:26:46.382013 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/c46c3317-ba7f-493b-a751-961cfa421b5c-manifests\") pod \"opendatahub-operator-controller-manager-5449d7969f-kr7sd\" (UID: \"c46c3317-ba7f-493b-a751-961cfa421b5c\") " pod="opendatahub/opendatahub-operator-controller-manager-5449d7969f-kr7sd" Jul 12 19:26:46.382177 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:26:46.382045 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/c46c3317-ba7f-493b-a751-961cfa421b5c-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5449d7969f-kr7sd\" (UID: \"c46c3317-ba7f-493b-a751-961cfa421b5c\") " pod="opendatahub/opendatahub-operator-controller-manager-5449d7969f-kr7sd" Jul 12 19:26:46.382177 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:26:46.382071 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c46c3317-ba7f-493b-a751-961cfa421b5c-webhook-cert\") pod \"opendatahub-operator-controller-manager-5449d7969f-kr7sd\" (UID: \"c46c3317-ba7f-493b-a751-961cfa421b5c\") " pod="opendatahub/opendatahub-operator-controller-manager-5449d7969f-kr7sd" Jul 12 19:26:46.382177 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:26:46.382149 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c46c3317-ba7f-493b-a751-961cfa421b5c-tmp\") pod \"opendatahub-operator-controller-manager-5449d7969f-kr7sd\" (UID: \"c46c3317-ba7f-493b-a751-961cfa421b5c\") " pod="opendatahub/opendatahub-operator-controller-manager-5449d7969f-kr7sd" Jul 12 19:26:46.382406 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:26:46.382188 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7cst\" (UniqueName: \"kubernetes.io/projected/c46c3317-ba7f-493b-a751-961cfa421b5c-kube-api-access-k7cst\") pod \"opendatahub-operator-controller-manager-5449d7969f-kr7sd\" (UID: \"c46c3317-ba7f-493b-a751-961cfa421b5c\") " pod="opendatahub/opendatahub-operator-controller-manager-5449d7969f-kr7sd" Jul 12 19:26:46.455356 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:26:46.455317 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:26:46.483503 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:26:46.483476 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7cst\" (UniqueName: \"kubernetes.io/projected/c46c3317-ba7f-493b-a751-961cfa421b5c-kube-api-access-k7cst\") pod \"opendatahub-operator-controller-manager-5449d7969f-kr7sd\" (UID: \"c46c3317-ba7f-493b-a751-961cfa421b5c\") " pod="opendatahub/opendatahub-operator-controller-manager-5449d7969f-kr7sd" Jul 12 19:26:46.483598 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:26:46.483519 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/c46c3317-ba7f-493b-a751-961cfa421b5c-manifests\") pod \"opendatahub-operator-controller-manager-5449d7969f-kr7sd\" (UID: \"c46c3317-ba7f-493b-a751-961cfa421b5c\") " pod="opendatahub/opendatahub-operator-controller-manager-5449d7969f-kr7sd" Jul 12 19:26:46.483598 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:26:46.483540 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/c46c3317-ba7f-493b-a751-961cfa421b5c-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5449d7969f-kr7sd\" (UID: \"c46c3317-ba7f-493b-a751-961cfa421b5c\") " pod="opendatahub/opendatahub-operator-controller-manager-5449d7969f-kr7sd" Jul 12 19:26:46.483598 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:26:46.483565 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c46c3317-ba7f-493b-a751-961cfa421b5c-webhook-cert\") pod \"opendatahub-operator-controller-manager-5449d7969f-kr7sd\" (UID: \"c46c3317-ba7f-493b-a751-961cfa421b5c\") " pod="opendatahub/opendatahub-operator-controller-manager-5449d7969f-kr7sd" Jul 12 19:26:46.483598 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:26:46.483597 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c46c3317-ba7f-493b-a751-961cfa421b5c-tmp\") pod \"opendatahub-operator-controller-manager-5449d7969f-kr7sd\" (UID: \"c46c3317-ba7f-493b-a751-961cfa421b5c\") " pod="opendatahub/opendatahub-operator-controller-manager-5449d7969f-kr7sd" Jul 12 19:26:46.483880 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:26:46.483861 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/c46c3317-ba7f-493b-a751-961cfa421b5c-manifests\") pod \"opendatahub-operator-controller-manager-5449d7969f-kr7sd\" (UID: \"c46c3317-ba7f-493b-a751-961cfa421b5c\") " pod="opendatahub/opendatahub-operator-controller-manager-5449d7969f-kr7sd" Jul 12 19:26:46.485865 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:26:46.485835 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c46c3317-ba7f-493b-a751-961cfa421b5c-tmp\") pod \"opendatahub-operator-controller-manager-5449d7969f-kr7sd\" (UID: \"c46c3317-ba7f-493b-a751-961cfa421b5c\") " pod="opendatahub/opendatahub-operator-controller-manager-5449d7969f-kr7sd" Jul 12 19:26:46.485971 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:26:46.485947 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c46c3317-ba7f-493b-a751-961cfa421b5c-webhook-cert\") pod \"opendatahub-operator-controller-manager-5449d7969f-kr7sd\" (UID: \"c46c3317-ba7f-493b-a751-961cfa421b5c\") " pod="opendatahub/opendatahub-operator-controller-manager-5449d7969f-kr7sd" Jul 12 19:26:46.486060 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:26:46.486043 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/c46c3317-ba7f-493b-a751-961cfa421b5c-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5449d7969f-kr7sd\" (UID: \"c46c3317-ba7f-493b-a751-961cfa421b5c\") " pod="opendatahub/opendatahub-operator-controller-manager-5449d7969f-kr7sd" Jul 12 19:26:46.493322 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:26:46.493299 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7cst\" (UniqueName: \"kubernetes.io/projected/c46c3317-ba7f-493b-a751-961cfa421b5c-kube-api-access-k7cst\") pod \"opendatahub-operator-controller-manager-5449d7969f-kr7sd\" (UID: \"c46c3317-ba7f-493b-a751-961cfa421b5c\") " pod="opendatahub/opendatahub-operator-controller-manager-5449d7969f-kr7sd" Jul 12 19:26:46.555257 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:26:46.555179 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:26:46.595304 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:26:46.595273 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5449d7969f-kr7sd" Jul 12 19:26:46.655664 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:26:46.655622 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:26:46.713137 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:26:46.713104 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:26:46.724579 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:26:46.724555 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5449d7969f-kr7sd"] Jul 12 19:26:46.727297 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:26:46.727267 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc46c3317_ba7f_493b_a751_961cfa421b5c.slice/crio-5637dbe951381a2d08b240f42f45017961bbc9796fb1cfc95a2347e26cd2b788 WatchSource:0}: Error finding container 5637dbe951381a2d08b240f42f45017961bbc9796fb1cfc95a2347e26cd2b788: Status 404 returned error can't find the container with id 5637dbe951381a2d08b240f42f45017961bbc9796fb1cfc95a2347e26cd2b788 Jul 12 19:26:46.772945 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:26:46.772922 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:26:46.874346 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:26:46.874283 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:26:46.978474 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:26:46.978451 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:26:47.097935 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:26:47.097903 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:26:47.105600 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:26:47.105577 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5449d7969f-kr7sd" event={"ID":"c46c3317-ba7f-493b-a751-961cfa421b5c","Type":"ContainerStarted","Data":"5637dbe951381a2d08b240f42f45017961bbc9796fb1cfc95a2347e26cd2b788"} Jul 12 19:26:47.183417 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:26:47.183388 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:26:47.283350 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:26:47.283296 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:26:50.118259 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:26:50.118223 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5449d7969f-kr7sd" event={"ID":"c46c3317-ba7f-493b-a751-961cfa421b5c","Type":"ContainerStarted","Data":"19975f829ff264c975f5dd39b3a085e32c9e3970794f38ea4542b2b3c3b07e9e"} Jul 12 19:26:51.125553 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:26:51.125459 2565 generic.go:356] "Generic (PLEG): container finished" podID="c46c3317-ba7f-493b-a751-961cfa421b5c" containerID="19975f829ff264c975f5dd39b3a085e32c9e3970794f38ea4542b2b3c3b07e9e" exitCode=0 Jul 12 19:26:51.125908 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:26:51.125545 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5449d7969f-kr7sd" event={"ID":"c46c3317-ba7f-493b-a751-961cfa421b5c","Type":"ContainerDied","Data":"19975f829ff264c975f5dd39b3a085e32c9e3970794f38ea4542b2b3c3b07e9e"} Jul 12 19:26:52.131042 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:26:52.131009 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5449d7969f-kr7sd" event={"ID":"c46c3317-ba7f-493b-a751-961cfa421b5c","Type":"ContainerStarted","Data":"a3398265ddd18429fa25c558b5f4fe3ad872c8689b958411b8711c0f4f86a005"} Jul 12 19:26:52.131407 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:26:52.131124 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-5449d7969f-kr7sd" Jul 12 19:26:52.151086 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:26:52.151030 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-5449d7969f-kr7sd" podStartSLOduration=1.9926308769999999 podStartE2EDuration="6.151014039s" totalImagesPullingTime="3.137931234s" totalInitContainerRuntime="1.020451928s" isStatefulPod=false podCreationTimestamp="2026-07-12 19:26:46 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 19:26:52.150435381 +0000 UTC m=+276.883007870" watchObservedRunningTime="2026-07-12 19:26:52.151014039 +0000 UTC m=+276.883586530" Jul 12 19:27:03.136507 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:27:03.136474 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-5449d7969f-kr7sd" Jul 12 19:27:04.848314 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:27:04.848280 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:27:06.136743 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:27:06.136709 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:27:07.782541 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:27:07.782507 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:27:07.869211 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:27:07.869180 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-654967888b-nbxf9"] Jul 12 19:27:07.872160 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:27:07.872143 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-654967888b-nbxf9" Jul 12 19:27:07.874558 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:27:07.874521 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 12 19:27:07.874558 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:27:07.874540 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 12 19:27:07.875324 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:27:07.875309 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-wq9r5\"" Jul 12 19:27:07.875582 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:27:07.875563 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 12 19:27:07.875658 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:27:07.875610 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 12 19:27:07.880641 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:27:07.880619 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-654967888b-nbxf9"] Jul 12 19:27:07.905134 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:27:07.905103 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:27:07.963443 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:27:07.963411 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-44kfk\" (UniqueName: \"kubernetes.io/projected/f94b7de3-7cb4-4346-87eb-1ac4dfc83cc0-kube-api-access-44kfk\") pod \"kube-auth-proxy-654967888b-nbxf9\" (UID: \"f94b7de3-7cb4-4346-87eb-1ac4dfc83cc0\") " pod="openshift-ingress/kube-auth-proxy-654967888b-nbxf9" Jul 12 19:27:07.963621 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:27:07.963456 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f94b7de3-7cb4-4346-87eb-1ac4dfc83cc0-tmp\") pod \"kube-auth-proxy-654967888b-nbxf9\" (UID: \"f94b7de3-7cb4-4346-87eb-1ac4dfc83cc0\") " pod="openshift-ingress/kube-auth-proxy-654967888b-nbxf9" Jul 12 19:27:07.963621 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:27:07.963489 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f94b7de3-7cb4-4346-87eb-1ac4dfc83cc0-tls-certs\") pod \"kube-auth-proxy-654967888b-nbxf9\" (UID: \"f94b7de3-7cb4-4346-87eb-1ac4dfc83cc0\") " pod="openshift-ingress/kube-auth-proxy-654967888b-nbxf9" Jul 12 19:27:08.064732 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:27:08.064659 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f94b7de3-7cb4-4346-87eb-1ac4dfc83cc0-tls-certs\") pod \"kube-auth-proxy-654967888b-nbxf9\" (UID: \"f94b7de3-7cb4-4346-87eb-1ac4dfc83cc0\") " pod="openshift-ingress/kube-auth-proxy-654967888b-nbxf9" Jul 12 19:27:08.064732 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:27:08.064720 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44kfk\" (UniqueName: \"kubernetes.io/projected/f94b7de3-7cb4-4346-87eb-1ac4dfc83cc0-kube-api-access-44kfk\") pod \"kube-auth-proxy-654967888b-nbxf9\" (UID: \"f94b7de3-7cb4-4346-87eb-1ac4dfc83cc0\") " pod="openshift-ingress/kube-auth-proxy-654967888b-nbxf9" Jul 12 19:27:08.064910 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:27:08.064743 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f94b7de3-7cb4-4346-87eb-1ac4dfc83cc0-tmp\") pod \"kube-auth-proxy-654967888b-nbxf9\" (UID: \"f94b7de3-7cb4-4346-87eb-1ac4dfc83cc0\") " pod="openshift-ingress/kube-auth-proxy-654967888b-nbxf9" Jul 12 19:27:08.066930 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:27:08.066905 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f94b7de3-7cb4-4346-87eb-1ac4dfc83cc0-tmp\") pod \"kube-auth-proxy-654967888b-nbxf9\" (UID: \"f94b7de3-7cb4-4346-87eb-1ac4dfc83cc0\") " pod="openshift-ingress/kube-auth-proxy-654967888b-nbxf9" Jul 12 19:27:08.067050 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:27:08.067035 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f94b7de3-7cb4-4346-87eb-1ac4dfc83cc0-tls-certs\") pod \"kube-auth-proxy-654967888b-nbxf9\" (UID: \"f94b7de3-7cb4-4346-87eb-1ac4dfc83cc0\") " pod="openshift-ingress/kube-auth-proxy-654967888b-nbxf9" Jul 12 19:27:08.081983 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:27:08.081962 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-44kfk\" (UniqueName: \"kubernetes.io/projected/f94b7de3-7cb4-4346-87eb-1ac4dfc83cc0-kube-api-access-44kfk\") pod \"kube-auth-proxy-654967888b-nbxf9\" (UID: \"f94b7de3-7cb4-4346-87eb-1ac4dfc83cc0\") " pod="openshift-ingress/kube-auth-proxy-654967888b-nbxf9" Jul 12 19:27:08.117039 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:27:08.117007 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:27:08.182349 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:27:08.182325 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-654967888b-nbxf9" Jul 12 19:27:08.190501 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:27:08.190472 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:27:08.302645 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:27:08.302567 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-654967888b-nbxf9"] Jul 12 19:27:08.305286 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:27:08.305254 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf94b7de3_7cb4_4346_87eb_1ac4dfc83cc0.slice/crio-300a561a3d05fc7d45a9d8f17ff10e8389dcf0c3042ca9098981a7d6e4b3ccba WatchSource:0}: Error finding container 300a561a3d05fc7d45a9d8f17ff10e8389dcf0c3042ca9098981a7d6e4b3ccba: Status 404 returned error can't find the container with id 300a561a3d05fc7d45a9d8f17ff10e8389dcf0c3042ca9098981a7d6e4b3ccba Jul 12 19:27:08.665945 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:27:08.665916 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:27:08.716904 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:27:08.716854 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-4jntx"] Jul 12 19:27:08.721080 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:27:08.721062 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-4jntx" Jul 12 19:27:08.723517 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:27:08.723475 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Jul 12 19:27:08.723657 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:27:08.723608 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-rnkjn\"" Jul 12 19:27:08.731518 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:27:08.731498 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-4jntx"] Jul 12 19:27:08.740770 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:27:08.740747 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:27:08.770546 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:27:08.770514 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/352106d4-8ce4-442d-a232-416311e4b0a1-tls-certs\") pod \"model-serving-api-54d89bd5fc-4jntx\" (UID: \"352106d4-8ce4-442d-a232-416311e4b0a1\") " pod="opendatahub/model-serving-api-54d89bd5fc-4jntx" Jul 12 19:27:08.770696 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:27:08.770553 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dp4vv\" (UniqueName: \"kubernetes.io/projected/352106d4-8ce4-442d-a232-416311e4b0a1-kube-api-access-dp4vv\") pod \"model-serving-api-54d89bd5fc-4jntx\" (UID: \"352106d4-8ce4-442d-a232-416311e4b0a1\") " pod="opendatahub/model-serving-api-54d89bd5fc-4jntx" Jul 12 19:27:08.871310 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:27:08.871274 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/352106d4-8ce4-442d-a232-416311e4b0a1-tls-certs\") pod \"model-serving-api-54d89bd5fc-4jntx\" (UID: \"352106d4-8ce4-442d-a232-416311e4b0a1\") " pod="opendatahub/model-serving-api-54d89bd5fc-4jntx" Jul 12 19:27:08.871714 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:27:08.871320 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dp4vv\" (UniqueName: \"kubernetes.io/projected/352106d4-8ce4-442d-a232-416311e4b0a1-kube-api-access-dp4vv\") pod \"model-serving-api-54d89bd5fc-4jntx\" (UID: \"352106d4-8ce4-442d-a232-416311e4b0a1\") " pod="opendatahub/model-serving-api-54d89bd5fc-4jntx" Jul 12 19:27:08.871714 ip-10-0-139-170 kubenswrapper[2565]: E0712 19:27:08.871397 2565 secret.go:189] Couldn't get secret opendatahub/model-serving-api-tls: secret "model-serving-api-tls" not found Jul 12 19:27:08.871714 ip-10-0-139-170 kubenswrapper[2565]: E0712 19:27:08.871457 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/352106d4-8ce4-442d-a232-416311e4b0a1-tls-certs podName:352106d4-8ce4-442d-a232-416311e4b0a1 nodeName:}" failed. No retries permitted until 2026-07-12 19:27:09.371440322 +0000 UTC m=+294.104012790 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/352106d4-8ce4-442d-a232-416311e4b0a1-tls-certs") pod "model-serving-api-54d89bd5fc-4jntx" (UID: "352106d4-8ce4-442d-a232-416311e4b0a1") : secret "model-serving-api-tls" not found Jul 12 19:27:08.880032 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:27:08.880015 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dp4vv\" (UniqueName: \"kubernetes.io/projected/352106d4-8ce4-442d-a232-416311e4b0a1-kube-api-access-dp4vv\") pod \"model-serving-api-54d89bd5fc-4jntx\" (UID: \"352106d4-8ce4-442d-a232-416311e4b0a1\") " pod="opendatahub/model-serving-api-54d89bd5fc-4jntx" Jul 12 19:27:08.951804 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:27:08.951713 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:27:09.027396 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:27:09.027364 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:27:09.192234 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:27:09.192194 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-654967888b-nbxf9" event={"ID":"f94b7de3-7cb4-4346-87eb-1ac4dfc83cc0","Type":"ContainerStarted","Data":"300a561a3d05fc7d45a9d8f17ff10e8389dcf0c3042ca9098981a7d6e4b3ccba"} Jul 12 19:27:09.374945 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:27:09.374843 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/352106d4-8ce4-442d-a232-416311e4b0a1-tls-certs\") pod \"model-serving-api-54d89bd5fc-4jntx\" (UID: \"352106d4-8ce4-442d-a232-416311e4b0a1\") " pod="opendatahub/model-serving-api-54d89bd5fc-4jntx" Jul 12 19:27:09.377454 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:27:09.377423 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/352106d4-8ce4-442d-a232-416311e4b0a1-tls-certs\") pod \"model-serving-api-54d89bd5fc-4jntx\" (UID: \"352106d4-8ce4-442d-a232-416311e4b0a1\") " pod="opendatahub/model-serving-api-54d89bd5fc-4jntx" Jul 12 19:27:09.632687 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:27:09.632615 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-4jntx" Jul 12 19:27:09.782853 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:27:09.782820 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-4jntx"] Jul 12 19:27:09.786510 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:27:09.786475 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod352106d4_8ce4_442d_a232_416311e4b0a1.slice/crio-25bb2ee70f7b774af5c44380f376d67bc5cd7ec8321bb8007b0108afc76d408e WatchSource:0}: Error finding container 25bb2ee70f7b774af5c44380f376d67bc5cd7ec8321bb8007b0108afc76d408e: Status 404 returned error can't find the container with id 25bb2ee70f7b774af5c44380f376d67bc5cd7ec8321bb8007b0108afc76d408e Jul 12 19:27:09.919017 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:27:09.918969 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:27:10.198544 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:27:10.198455 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-4jntx" event={"ID":"352106d4-8ce4-442d-a232-416311e4b0a1","Type":"ContainerStarted","Data":"25bb2ee70f7b774af5c44380f376d67bc5cd7ec8321bb8007b0108afc76d408e"} Jul 12 19:27:11.203970 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:27:11.203931 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-654967888b-nbxf9" event={"ID":"f94b7de3-7cb4-4346-87eb-1ac4dfc83cc0","Type":"ContainerStarted","Data":"dba900abcce970aa2b6269fe02b6fa5f224e979150cf849c3bf315b40d4f3ec6"} Jul 12 19:27:11.220134 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:27:11.220080 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-654967888b-nbxf9" podStartSLOduration=1.944746861 podStartE2EDuration="4.220065151s" totalImagesPullingTime="2.27531829s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 19:27:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 19:27:11.21838634 +0000 UTC m=+295.950958830" watchObservedRunningTime="2026-07-12 19:27:11.220065151 +0000 UTC m=+295.952637640" Jul 12 19:27:12.209743 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:27:12.209707 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-4jntx" event={"ID":"352106d4-8ce4-442d-a232-416311e4b0a1","Type":"ContainerStarted","Data":"23e5234f2252ef9beafcec2703c1f6d3c43bfdb631a63971009ba7a377e9156a"} Jul 12 19:27:12.225297 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:27:12.225246 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-4jntx" podStartSLOduration=2.263803045 podStartE2EDuration="4.225231207s" totalImagesPullingTime="1.961428162s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 19:27:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 19:27:12.224313584 +0000 UTC m=+296.956886075" watchObservedRunningTime="2026-07-12 19:27:12.225231207 +0000 UTC m=+296.957803696" Jul 12 19:27:13.213816 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:27:13.213789 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-4jntx" Jul 12 19:27:13.979600 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:27:13.979567 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:27:14.089971 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:27:14.089937 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:27:14.613590 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:27:14.613558 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678c9jcd"] Jul 12 19:27:14.617036 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:27:14.617017 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678c9jcd" Jul 12 19:27:14.620100 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:27:14.620079 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-dnpj2\"" Jul 12 19:27:14.621842 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:27:14.621826 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 12 19:27:14.654764 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:27:14.654732 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678c9jcd"] Jul 12 19:27:14.660700 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:27:14.660676 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:27:14.715713 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:27:14.715652 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d445ad9c-2f45-4808-989a-d988ef1dbaff-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678c9jcd\" (UID: \"d445ad9c-2f45-4808-989a-d988ef1dbaff\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678c9jcd" Jul 12 19:27:14.715934 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:27:14.715715 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7622\" (UniqueName: \"kubernetes.io/projected/d445ad9c-2f45-4808-989a-d988ef1dbaff-kube-api-access-p7622\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678c9jcd\" (UID: \"d445ad9c-2f45-4808-989a-d988ef1dbaff\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678c9jcd" Jul 12 19:27:14.715934 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:27:14.715769 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d445ad9c-2f45-4808-989a-d988ef1dbaff-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678c9jcd\" (UID: \"d445ad9c-2f45-4808-989a-d988ef1dbaff\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678c9jcd" Jul 12 19:27:14.715934 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:27:14.715792 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d445ad9c-2f45-4808-989a-d988ef1dbaff-istio-token\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678c9jcd\" (UID: \"d445ad9c-2f45-4808-989a-d988ef1dbaff\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678c9jcd" Jul 12 19:27:14.715934 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:27:14.715836 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d445ad9c-2f45-4808-989a-d988ef1dbaff-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678c9jcd\" (UID: \"d445ad9c-2f45-4808-989a-d988ef1dbaff\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678c9jcd" Jul 12 19:27:14.715934 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:27:14.715864 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d445ad9c-2f45-4808-989a-d988ef1dbaff-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678c9jcd\" (UID: \"d445ad9c-2f45-4808-989a-d988ef1dbaff\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678c9jcd" Jul 12 19:27:14.715934 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:27:14.715917 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d445ad9c-2f45-4808-989a-d988ef1dbaff-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678c9jcd\" (UID: \"d445ad9c-2f45-4808-989a-d988ef1dbaff\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678c9jcd" Jul 12 19:27:14.716225 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:27:14.715957 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d445ad9c-2f45-4808-989a-d988ef1dbaff-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678c9jcd\" (UID: \"d445ad9c-2f45-4808-989a-d988ef1dbaff\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678c9jcd" Jul 12 19:27:14.716225 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:27:14.715993 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d445ad9c-2f45-4808-989a-d988ef1dbaff-istio-data\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678c9jcd\" (UID: \"d445ad9c-2f45-4808-989a-d988ef1dbaff\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678c9jcd" Jul 12 19:27:14.817252 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:27:14.817217 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d445ad9c-2f45-4808-989a-d988ef1dbaff-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678c9jcd\" (UID: \"d445ad9c-2f45-4808-989a-d988ef1dbaff\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678c9jcd" Jul 12 19:27:14.817252 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:27:14.817257 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d445ad9c-2f45-4808-989a-d988ef1dbaff-istio-data\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678c9jcd\" (UID: \"d445ad9c-2f45-4808-989a-d988ef1dbaff\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678c9jcd" Jul 12 19:27:14.817479 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:27:14.817284 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d445ad9c-2f45-4808-989a-d988ef1dbaff-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678c9jcd\" (UID: \"d445ad9c-2f45-4808-989a-d988ef1dbaff\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678c9jcd" Jul 12 19:27:14.817479 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:27:14.817313 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7622\" (UniqueName: \"kubernetes.io/projected/d445ad9c-2f45-4808-989a-d988ef1dbaff-kube-api-access-p7622\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678c9jcd\" (UID: \"d445ad9c-2f45-4808-989a-d988ef1dbaff\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678c9jcd" Jul 12 19:27:14.817479 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:27:14.817341 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d445ad9c-2f45-4808-989a-d988ef1dbaff-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678c9jcd\" (UID: \"d445ad9c-2f45-4808-989a-d988ef1dbaff\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678c9jcd" Jul 12 19:27:14.817479 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:27:14.817355 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d445ad9c-2f45-4808-989a-d988ef1dbaff-istio-token\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678c9jcd\" (UID: \"d445ad9c-2f45-4808-989a-d988ef1dbaff\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678c9jcd" Jul 12 19:27:14.817479 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:27:14.817382 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d445ad9c-2f45-4808-989a-d988ef1dbaff-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678c9jcd\" (UID: \"d445ad9c-2f45-4808-989a-d988ef1dbaff\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678c9jcd" Jul 12 19:27:14.817709 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:27:14.817554 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d445ad9c-2f45-4808-989a-d988ef1dbaff-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678c9jcd\" (UID: \"d445ad9c-2f45-4808-989a-d988ef1dbaff\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678c9jcd" Jul 12 19:27:14.817709 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:27:14.817607 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d445ad9c-2f45-4808-989a-d988ef1dbaff-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678c9jcd\" (UID: \"d445ad9c-2f45-4808-989a-d988ef1dbaff\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678c9jcd" Jul 12 19:27:14.817709 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:27:14.817640 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d445ad9c-2f45-4808-989a-d988ef1dbaff-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678c9jcd\" (UID: \"d445ad9c-2f45-4808-989a-d988ef1dbaff\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678c9jcd" Jul 12 19:27:14.817898 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:27:14.817867 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d445ad9c-2f45-4808-989a-d988ef1dbaff-istio-data\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678c9jcd\" (UID: \"d445ad9c-2f45-4808-989a-d988ef1dbaff\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678c9jcd" Jul 12 19:27:14.818009 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:27:14.817974 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d445ad9c-2f45-4808-989a-d988ef1dbaff-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678c9jcd\" (UID: \"d445ad9c-2f45-4808-989a-d988ef1dbaff\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678c9jcd" Jul 12 19:27:14.818009 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:27:14.818000 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d445ad9c-2f45-4808-989a-d988ef1dbaff-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678c9jcd\" (UID: \"d445ad9c-2f45-4808-989a-d988ef1dbaff\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678c9jcd" Jul 12 19:27:14.818425 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:27:14.818400 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d445ad9c-2f45-4808-989a-d988ef1dbaff-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678c9jcd\" (UID: \"d445ad9c-2f45-4808-989a-d988ef1dbaff\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678c9jcd" Jul 12 19:27:14.819591 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:27:14.819563 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d445ad9c-2f45-4808-989a-d988ef1dbaff-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678c9jcd\" (UID: \"d445ad9c-2f45-4808-989a-d988ef1dbaff\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678c9jcd" Jul 12 19:27:14.819846 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:27:14.819830 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d445ad9c-2f45-4808-989a-d988ef1dbaff-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678c9jcd\" (UID: \"d445ad9c-2f45-4808-989a-d988ef1dbaff\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678c9jcd" Jul 12 19:27:14.829692 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:27:14.829670 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d445ad9c-2f45-4808-989a-d988ef1dbaff-istio-token\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678c9jcd\" (UID: \"d445ad9c-2f45-4808-989a-d988ef1dbaff\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678c9jcd" Jul 12 19:27:14.831043 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:27:14.831024 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7622\" (UniqueName: \"kubernetes.io/projected/d445ad9c-2f45-4808-989a-d988ef1dbaff-kube-api-access-p7622\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678c9jcd\" (UID: \"d445ad9c-2f45-4808-989a-d988ef1dbaff\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678c9jcd" Jul 12 19:27:14.837404 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:27:14.837384 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:27:14.926832 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:27:14.926669 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678c9jcd" Jul 12 19:27:14.945414 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:27:14.945387 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:27:14.991033 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:27:14.991005 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-vgbw9"] Jul 12 19:27:14.994424 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:27:14.994399 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-vgbw9" Jul 12 19:27:15.001359 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:27:15.001327 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 12 19:27:15.001359 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:27:15.001347 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 12 19:27:15.001698 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:27:15.001680 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-zjc87\"" Jul 12 19:27:15.002696 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:27:15.002042 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 12 19:27:15.005709 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:27:15.005680 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-vgbw9"] Jul 12 19:27:15.020006 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:27:15.019982 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4f668c0d-a883-4283-bea4-fb5f56cd1481-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-vgbw9\" (UID: \"4f668c0d-a883-4283-bea4-fb5f56cd1481\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-vgbw9" Jul 12 19:27:15.020113 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:27:15.020026 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f668c0d-a883-4283-bea4-fb5f56cd1481-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-vgbw9\" (UID: \"4f668c0d-a883-4283-bea4-fb5f56cd1481\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-vgbw9" Jul 12 19:27:15.020113 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:27:15.020048 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/4f668c0d-a883-4283-bea4-fb5f56cd1481-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-vgbw9\" (UID: \"4f668c0d-a883-4283-bea4-fb5f56cd1481\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-vgbw9" Jul 12 19:27:15.020218 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:27:15.020118 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2xmgw\" (UniqueName: \"kubernetes.io/projected/4f668c0d-a883-4283-bea4-fb5f56cd1481-kube-api-access-2xmgw\") pod \"istiod-openshift-gateway-94bb8fbfd-vgbw9\" (UID: \"4f668c0d-a883-4283-bea4-fb5f56cd1481\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-vgbw9" Jul 12 19:27:15.020218 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:27:15.020151 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f668c0d-a883-4283-bea4-fb5f56cd1481-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-vgbw9\" (UID: \"4f668c0d-a883-4283-bea4-fb5f56cd1481\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-vgbw9" Jul 12 19:27:15.020218 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:27:15.020196 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/4f668c0d-a883-4283-bea4-fb5f56cd1481-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-vgbw9\" (UID: \"4f668c0d-a883-4283-bea4-fb5f56cd1481\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-vgbw9" Jul 12 19:27:15.020218 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:27:15.020216 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/4f668c0d-a883-4283-bea4-fb5f56cd1481-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-vgbw9\" (UID: \"4f668c0d-a883-4283-bea4-fb5f56cd1481\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-vgbw9" Jul 12 19:27:15.085810 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:27:15.085786 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678c9jcd"] Jul 12 19:27:15.087716 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:27:15.087691 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd445ad9c_2f45_4808_989a_d988ef1dbaff.slice/crio-7cbf83a41c018ab567e05e8d148287d6a56477b9803c6be5fb2bafdeb4a414b2 WatchSource:0}: Error finding container 7cbf83a41c018ab567e05e8d148287d6a56477b9803c6be5fb2bafdeb4a414b2: Status 404 returned error can't find the container with id 7cbf83a41c018ab567e05e8d148287d6a56477b9803c6be5fb2bafdeb4a414b2 Jul 12 19:27:15.121035 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:27:15.121003 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/4f668c0d-a883-4283-bea4-fb5f56cd1481-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-vgbw9\" (UID: \"4f668c0d-a883-4283-bea4-fb5f56cd1481\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-vgbw9" Jul 12 19:27:15.121147 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:27:15.121041 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xmgw\" (UniqueName: \"kubernetes.io/projected/4f668c0d-a883-4283-bea4-fb5f56cd1481-kube-api-access-2xmgw\") pod \"istiod-openshift-gateway-94bb8fbfd-vgbw9\" (UID: \"4f668c0d-a883-4283-bea4-fb5f56cd1481\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-vgbw9" Jul 12 19:27:15.121147 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:27:15.121061 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f668c0d-a883-4283-bea4-fb5f56cd1481-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-vgbw9\" (UID: \"4f668c0d-a883-4283-bea4-fb5f56cd1481\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-vgbw9" Jul 12 19:27:15.121147 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:27:15.121100 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/4f668c0d-a883-4283-bea4-fb5f56cd1481-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-vgbw9\" (UID: \"4f668c0d-a883-4283-bea4-fb5f56cd1481\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-vgbw9" Jul 12 19:27:15.121147 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:27:15.121123 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/4f668c0d-a883-4283-bea4-fb5f56cd1481-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-vgbw9\" (UID: \"4f668c0d-a883-4283-bea4-fb5f56cd1481\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-vgbw9" Jul 12 19:27:15.121367 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:27:15.121161 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4f668c0d-a883-4283-bea4-fb5f56cd1481-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-vgbw9\" (UID: \"4f668c0d-a883-4283-bea4-fb5f56cd1481\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-vgbw9" Jul 12 19:27:15.121477 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:27:15.121449 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f668c0d-a883-4283-bea4-fb5f56cd1481-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-vgbw9\" (UID: \"4f668c0d-a883-4283-bea4-fb5f56cd1481\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-vgbw9" Jul 12 19:27:15.121925 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:27:15.121899 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/4f668c0d-a883-4283-bea4-fb5f56cd1481-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-vgbw9\" (UID: \"4f668c0d-a883-4283-bea4-fb5f56cd1481\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-vgbw9" Jul 12 19:27:15.123481 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:27:15.123457 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f668c0d-a883-4283-bea4-fb5f56cd1481-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-vgbw9\" (UID: \"4f668c0d-a883-4283-bea4-fb5f56cd1481\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-vgbw9" Jul 12 19:27:15.123653 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:27:15.123635 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4f668c0d-a883-4283-bea4-fb5f56cd1481-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-vgbw9\" (UID: \"4f668c0d-a883-4283-bea4-fb5f56cd1481\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-vgbw9" Jul 12 19:27:15.123777 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:27:15.123763 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/4f668c0d-a883-4283-bea4-fb5f56cd1481-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-vgbw9\" (UID: \"4f668c0d-a883-4283-bea4-fb5f56cd1481\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-vgbw9" Jul 12 19:27:15.123833 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:27:15.123808 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/4f668c0d-a883-4283-bea4-fb5f56cd1481-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-vgbw9\" (UID: \"4f668c0d-a883-4283-bea4-fb5f56cd1481\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-vgbw9" Jul 12 19:27:15.132099 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:27:15.132071 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xmgw\" (UniqueName: \"kubernetes.io/projected/4f668c0d-a883-4283-bea4-fb5f56cd1481-kube-api-access-2xmgw\") pod \"istiod-openshift-gateway-94bb8fbfd-vgbw9\" (UID: \"4f668c0d-a883-4283-bea4-fb5f56cd1481\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-vgbw9" Jul 12 19:27:15.132795 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:27:15.132780 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f668c0d-a883-4283-bea4-fb5f56cd1481-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-vgbw9\" (UID: \"4f668c0d-a883-4283-bea4-fb5f56cd1481\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-vgbw9" Jul 12 19:27:15.221682 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:27:15.221593 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678c9jcd" event={"ID":"d445ad9c-2f45-4808-989a-d988ef1dbaff","Type":"ContainerStarted","Data":"7cbf83a41c018ab567e05e8d148287d6a56477b9803c6be5fb2bafdeb4a414b2"} Jul 12 19:27:15.312929 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:27:15.312873 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-vgbw9" Jul 12 19:27:15.459723 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:27:15.459692 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-vgbw9"] Jul 12 19:27:15.462270 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:27:15.462240 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f668c0d_a883_4283_bea4_fb5f56cd1481.slice/crio-2fa20a1e11a44385cdd7aab625afd9cfbd6c4b7c8f8ad9fbf86f3231551de953 WatchSource:0}: Error finding container 2fa20a1e11a44385cdd7aab625afd9cfbd6c4b7c8f8ad9fbf86f3231551de953: Status 404 returned error can't find the container with id 2fa20a1e11a44385cdd7aab625afd9cfbd6c4b7c8f8ad9fbf86f3231551de953 Jul 12 19:27:15.698373 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:27:15.698343 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k5h9m_8e016c2e-0c3a-49da-83d4-f526738c1cbd/ovn-acl-logging/0.log" Jul 12 19:27:15.700402 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:27:15.700380 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k5h9m_8e016c2e-0c3a-49da-83d4-f526738c1cbd/ovn-acl-logging/0.log" Jul 12 19:27:15.701972 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:27:15.701940 2565 kubelet.go:1648] "Image garbage collection succeeded" Jul 12 19:27:16.228678 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:27:16.228635 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-vgbw9" event={"ID":"4f668c0d-a883-4283-bea4-fb5f56cd1481","Type":"ContainerStarted","Data":"2fa20a1e11a44385cdd7aab625afd9cfbd6c4b7c8f8ad9fbf86f3231551de953"} Jul 12 19:27:17.509170 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:27:17.509033 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:27:17.594446 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:27:17.594409 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:27:17.679633 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:27:17.679593 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-79f8975998-xbfx6"] Jul 12 19:27:17.689598 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:27:17.689572 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-79f8975998-xbfx6" Jul 12 19:27:17.692248 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:27:17.692219 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 12 19:27:17.692420 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:27:17.692261 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 12 19:27:17.693493 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:27:17.693319 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-mslwk\"" Jul 12 19:27:17.693493 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:27:17.693365 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 12 19:27:17.693493 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:27:17.693381 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 12 19:27:17.693493 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:27:17.693465 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 12 19:27:17.697659 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:27:17.697635 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-79f8975998-xbfx6"] Jul 12 19:27:17.746213 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:27:17.746185 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8bgv\" (UniqueName: \"kubernetes.io/projected/87e72f83-5c44-408f-bfa5-24e34721c580-kube-api-access-s8bgv\") pod \"lws-controller-manager-79f8975998-xbfx6\" (UID: \"87e72f83-5c44-408f-bfa5-24e34721c580\") " pod="openshift-lws-operator/lws-controller-manager-79f8975998-xbfx6" Jul 12 19:27:17.746386 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:27:17.746288 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/87e72f83-5c44-408f-bfa5-24e34721c580-cert\") pod \"lws-controller-manager-79f8975998-xbfx6\" (UID: \"87e72f83-5c44-408f-bfa5-24e34721c580\") " pod="openshift-lws-operator/lws-controller-manager-79f8975998-xbfx6" Jul 12 19:27:17.746386 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:27:17.746329 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/87e72f83-5c44-408f-bfa5-24e34721c580-manager-config\") pod \"lws-controller-manager-79f8975998-xbfx6\" (UID: \"87e72f83-5c44-408f-bfa5-24e34721c580\") " pod="openshift-lws-operator/lws-controller-manager-79f8975998-xbfx6" Jul 12 19:27:17.746386 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:27:17.746355 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/87e72f83-5c44-408f-bfa5-24e34721c580-metrics-cert\") pod \"lws-controller-manager-79f8975998-xbfx6\" (UID: \"87e72f83-5c44-408f-bfa5-24e34721c580\") " pod="openshift-lws-operator/lws-controller-manager-79f8975998-xbfx6" Jul 12 19:27:17.846918 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:27:17.846822 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/87e72f83-5c44-408f-bfa5-24e34721c580-cert\") pod \"lws-controller-manager-79f8975998-xbfx6\" (UID: \"87e72f83-5c44-408f-bfa5-24e34721c580\") " pod="openshift-lws-operator/lws-controller-manager-79f8975998-xbfx6" Jul 12 19:27:17.846918 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:27:17.846871 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/87e72f83-5c44-408f-bfa5-24e34721c580-manager-config\") pod \"lws-controller-manager-79f8975998-xbfx6\" (UID: \"87e72f83-5c44-408f-bfa5-24e34721c580\") " pod="openshift-lws-operator/lws-controller-manager-79f8975998-xbfx6" Jul 12 19:27:17.846918 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:27:17.846917 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/87e72f83-5c44-408f-bfa5-24e34721c580-metrics-cert\") pod \"lws-controller-manager-79f8975998-xbfx6\" (UID: \"87e72f83-5c44-408f-bfa5-24e34721c580\") " pod="openshift-lws-operator/lws-controller-manager-79f8975998-xbfx6" Jul 12 19:27:17.847177 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:27:17.846989 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8bgv\" (UniqueName: \"kubernetes.io/projected/87e72f83-5c44-408f-bfa5-24e34721c580-kube-api-access-s8bgv\") pod \"lws-controller-manager-79f8975998-xbfx6\" (UID: \"87e72f83-5c44-408f-bfa5-24e34721c580\") " pod="openshift-lws-operator/lws-controller-manager-79f8975998-xbfx6" Jul 12 19:27:17.847820 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:27:17.847792 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/87e72f83-5c44-408f-bfa5-24e34721c580-manager-config\") pod \"lws-controller-manager-79f8975998-xbfx6\" (UID: \"87e72f83-5c44-408f-bfa5-24e34721c580\") " pod="openshift-lws-operator/lws-controller-manager-79f8975998-xbfx6" Jul 12 19:27:17.849901 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:27:17.849834 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/87e72f83-5c44-408f-bfa5-24e34721c580-metrics-cert\") pod \"lws-controller-manager-79f8975998-xbfx6\" (UID: \"87e72f83-5c44-408f-bfa5-24e34721c580\") " pod="openshift-lws-operator/lws-controller-manager-79f8975998-xbfx6" Jul 12 19:27:17.850430 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:27:17.850403 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/87e72f83-5c44-408f-bfa5-24e34721c580-cert\") pod \"lws-controller-manager-79f8975998-xbfx6\" (UID: \"87e72f83-5c44-408f-bfa5-24e34721c580\") " pod="openshift-lws-operator/lws-controller-manager-79f8975998-xbfx6" Jul 12 19:27:17.860060 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:27:17.860036 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8bgv\" (UniqueName: \"kubernetes.io/projected/87e72f83-5c44-408f-bfa5-24e34721c580-kube-api-access-s8bgv\") pod \"lws-controller-manager-79f8975998-xbfx6\" (UID: \"87e72f83-5c44-408f-bfa5-24e34721c580\") " pod="openshift-lws-operator/lws-controller-manager-79f8975998-xbfx6" Jul 12 19:27:17.931245 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:27:17.931204 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:27:18.003203 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:27:18.003171 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-79f8975998-xbfx6" Jul 12 19:27:18.197396 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:27:18.197365 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:27:18.765243 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:27:18.765208 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:27:18.798066 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:27:18.798021 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 12 19:27:18.798198 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:27:18.798100 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 12 19:27:18.798198 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:27:18.798134 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 12 19:27:18.802788 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:27:18.802755 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 12 19:27:18.802872 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:27:18.802819 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 12 19:27:18.922125 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:27:18.922086 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:27:18.934047 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:27:18.933872 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-79f8975998-xbfx6"] Jul 12 19:27:18.937167 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:27:18.937141 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87e72f83_5c44_408f_bfa5_24e34721c580.slice/crio-afebb8d0f6578307ad9af0aed080e0a2a5fbeed2510a806f925f74ea79841e0a WatchSource:0}: Error finding container afebb8d0f6578307ad9af0aed080e0a2a5fbeed2510a806f925f74ea79841e0a: Status 404 returned error can't find the container with id afebb8d0f6578307ad9af0aed080e0a2a5fbeed2510a806f925f74ea79841e0a Jul 12 19:27:18.939523 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:27:18.939508 2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 12 19:27:19.009235 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:27:19.009199 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:27:19.243207 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:27:19.243169 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-vgbw9" event={"ID":"4f668c0d-a883-4283-bea4-fb5f56cd1481","Type":"ContainerStarted","Data":"6fe1db50641ca54b5a77b9c381e6391586a1bd893e77ac222c47ea016eff1ce1"} Jul 12 19:27:19.243370 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:27:19.243341 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-vgbw9" Jul 12 19:27:19.244906 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:27:19.244863 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678c9jcd" event={"ID":"d445ad9c-2f45-4808-989a-d988ef1dbaff","Type":"ContainerStarted","Data":"a4160f9e6d9fc51a0a7799c1047d43d42266fb69cb5e34fa6f5a2afc24e8599a"} Jul 12 19:27:19.245271 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:27:19.245246 2565 patch_prober.go:28] interesting pod/istiod-openshift-gateway-94bb8fbfd-vgbw9 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 12 19:27:19.245400 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:27:19.245303 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-vgbw9" podUID="4f668c0d-a883-4283-bea4-fb5f56cd1481" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 12 19:27:19.246235 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:27:19.246214 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-79f8975998-xbfx6" event={"ID":"87e72f83-5c44-408f-bfa5-24e34721c580","Type":"ContainerStarted","Data":"afebb8d0f6578307ad9af0aed080e0a2a5fbeed2510a806f925f74ea79841e0a"} Jul 12 19:27:19.265360 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:27:19.265318 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-vgbw9" podStartSLOduration=1.927052796 podStartE2EDuration="5.265305281s" totalImagesPullingTime="3.338252485s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 19:27:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 19:27:19.263872232 +0000 UTC m=+303.996444759" watchObservedRunningTime="2026-07-12 19:27:19.265305281 +0000 UTC m=+303.997877753" Jul 12 19:27:19.284421 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:27:19.284379 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678c9jcd" podStartSLOduration=1.5762512910000002 podStartE2EDuration="5.284366246s" totalImagesPullingTime="3.708114955s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 19:27:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 19:27:19.283135693 +0000 UTC m=+304.015708219" watchObservedRunningTime="2026-07-12 19:27:19.284366246 +0000 UTC m=+304.016938735" Jul 12 19:27:19.927723 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:27:19.927686 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678c9jcd" Jul 12 19:27:19.933214 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:27:19.933188 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678c9jcd" Jul 12 19:27:20.251991 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:27:20.251909 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678c9jcd" Jul 12 19:27:20.252643 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:27:20.252622 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-vgbw9" Jul 12 19:27:20.253044 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:27:20.253028 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678c9jcd" Jul 12 19:27:21.258291 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:27:21.258251 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-79f8975998-xbfx6" event={"ID":"87e72f83-5c44-408f-bfa5-24e34721c580","Type":"ContainerStarted","Data":"ed69084f4de7582f8f6f27da4d8077e83dc885ad3eed25c07a61839d79ec5ac6"} Jul 12 19:27:21.273898 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:27:21.273829 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-79f8975998-xbfx6" podStartSLOduration=2.060199457 podStartE2EDuration="4.273812898s" totalImagesPullingTime="2.213613441s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 19:27:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 19:27:21.272295135 +0000 UTC m=+306.004867629" watchObservedRunningTime="2026-07-12 19:27:21.273812898 +0000 UTC m=+306.006385441" Jul 12 19:27:22.263049 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:27:22.263014 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-79f8975998-xbfx6" Jul 12 19:27:24.220783 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:27:24.220756 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-4jntx" Jul 12 19:27:33.269893 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:27:33.269859 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-79f8975998-xbfx6" Jul 12 19:28:05.691907 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:28:05.691861 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:28:05.788717 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:28:05.788687 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:28:06.404401 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:28:06.404362 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:28:06.494741 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:28:06.494711 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:28:06.569560 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:28:06.569523 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:28:07.316755 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:28:07.316723 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:28:07.523671 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:28:07.523629 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:28:07.640644 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:28:07.640609 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:28:07.690214 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:28:07.690177 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:28:35.678744 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:28:35.678704 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:28:39.287217 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:28:39.287180 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:28:39.383935 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:28:39.383901 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:28:43.673186 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:28:43.673154 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:28:44.285223 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:28:44.285192 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:28:45.916055 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:28:45.916011 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:28:47.508641 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:28:47.508599 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:28:48.306968 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:28:48.306935 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:28:50.328429 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:28:50.328396 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:28:51.682827 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:28:51.682789 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:28:51.785035 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:28:51.784999 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:28:52.890710 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:28:52.890676 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:28:53.238808 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:28:53.238720 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:28:54.881023 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:28:54.880987 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:28:54.928971 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:28:54.928944 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:29:00.557547 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:29:00.557506 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:29:00.620535 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:29:00.620500 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:29:01.938279 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:29:01.938233 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:29:03.749011 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:29:03.748979 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:29:03.863408 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:29:03.863379 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:29:04.203156 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:29:04.203123 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:29:04.306735 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:29:04.306701 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:29:06.230919 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:29:06.230867 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:29:06.337716 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:29:06.337680 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:29:10.470368 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:29:10.470331 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:29:10.568845 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:29:10.568810 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:29:12.855038 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:29:12.855002 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:29:12.953379 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:29:12.953346 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:29:21.709729 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:29:21.709690 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:29:21.807752 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:29:21.807708 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:29:24.130699 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:29:24.130664 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:29:24.188077 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:29:24.188036 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:29:28.526431 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:29:28.526399 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:29:28.623282 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:29:28.623250 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:29:31.291481 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:29:31.291448 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:29:31.350074 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:29:31.350044 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:29:39.903191 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:29:39.903110 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:29:40.002914 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:29:40.002866 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:29:42.348747 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:29:42.348691 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:29:42.401172 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:29:42.401119 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:29:44.785319 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:29:44.785284 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:29:45.137847 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:29:45.137772 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:29:49.360767 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:29:49.360731 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:29:49.488767 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:29:49.488734 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:29:49.548828 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:29:49.548794 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:29:50.185719 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:29:50.185658 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:29:50.235506 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:29:50.235473 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:29:59.021821 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:29:59.021786 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7d4d5f7c4f-d5vdz"] Jul 12 19:29:59.025150 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:29:59.025134 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7d4d5f7c4f-d5vdz" Jul 12 19:29:59.035362 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:29:59.035337 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7d4d5f7c4f-d5vdz"] Jul 12 19:29:59.071852 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:29:59.071819 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:29:59.124007 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:29:59.123968 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:29:59.138192 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:29:59.138163 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ac3c9fb2-cbee-4fc5-91da-448c7482fe6c-console-oauth-config\") pod \"console-7d4d5f7c4f-d5vdz\" (UID: \"ac3c9fb2-cbee-4fc5-91da-448c7482fe6c\") " pod="openshift-console/console-7d4d5f7c4f-d5vdz" Jul 12 19:29:59.138344 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:29:59.138204 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ac3c9fb2-cbee-4fc5-91da-448c7482fe6c-trusted-ca-bundle\") pod \"console-7d4d5f7c4f-d5vdz\" (UID: \"ac3c9fb2-cbee-4fc5-91da-448c7482fe6c\") " pod="openshift-console/console-7d4d5f7c4f-d5vdz" Jul 12 19:29:59.138344 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:29:59.138231 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ac3c9fb2-cbee-4fc5-91da-448c7482fe6c-console-config\") pod \"console-7d4d5f7c4f-d5vdz\" (UID: \"ac3c9fb2-cbee-4fc5-91da-448c7482fe6c\") " pod="openshift-console/console-7d4d5f7c4f-d5vdz" Jul 12 19:29:59.138344 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:29:59.138251 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ac3c9fb2-cbee-4fc5-91da-448c7482fe6c-oauth-serving-cert\") pod \"console-7d4d5f7c4f-d5vdz\" (UID: \"ac3c9fb2-cbee-4fc5-91da-448c7482fe6c\") " pod="openshift-console/console-7d4d5f7c4f-d5vdz" Jul 12 19:29:59.138472 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:29:59.138345 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ac3c9fb2-cbee-4fc5-91da-448c7482fe6c-console-serving-cert\") pod \"console-7d4d5f7c4f-d5vdz\" (UID: \"ac3c9fb2-cbee-4fc5-91da-448c7482fe6c\") " pod="openshift-console/console-7d4d5f7c4f-d5vdz" Jul 12 19:29:59.138472 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:29:59.138391 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-txflp\" (UniqueName: \"kubernetes.io/projected/ac3c9fb2-cbee-4fc5-91da-448c7482fe6c-kube-api-access-txflp\") pod \"console-7d4d5f7c4f-d5vdz\" (UID: \"ac3c9fb2-cbee-4fc5-91da-448c7482fe6c\") " pod="openshift-console/console-7d4d5f7c4f-d5vdz" Jul 12 19:29:59.138472 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:29:59.138444 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ac3c9fb2-cbee-4fc5-91da-448c7482fe6c-service-ca\") pod \"console-7d4d5f7c4f-d5vdz\" (UID: \"ac3c9fb2-cbee-4fc5-91da-448c7482fe6c\") " pod="openshift-console/console-7d4d5f7c4f-d5vdz" Jul 12 19:29:59.221012 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:29:59.220977 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:29:59.238871 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:29:59.238839 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ac3c9fb2-cbee-4fc5-91da-448c7482fe6c-console-oauth-config\") pod \"console-7d4d5f7c4f-d5vdz\" (UID: \"ac3c9fb2-cbee-4fc5-91da-448c7482fe6c\") " pod="openshift-console/console-7d4d5f7c4f-d5vdz" Jul 12 19:29:59.238871 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:29:59.238875 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ac3c9fb2-cbee-4fc5-91da-448c7482fe6c-trusted-ca-bundle\") pod \"console-7d4d5f7c4f-d5vdz\" (UID: \"ac3c9fb2-cbee-4fc5-91da-448c7482fe6c\") " pod="openshift-console/console-7d4d5f7c4f-d5vdz" Jul 12 19:29:59.239114 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:29:59.238933 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ac3c9fb2-cbee-4fc5-91da-448c7482fe6c-console-config\") pod \"console-7d4d5f7c4f-d5vdz\" (UID: \"ac3c9fb2-cbee-4fc5-91da-448c7482fe6c\") " pod="openshift-console/console-7d4d5f7c4f-d5vdz" Jul 12 19:29:59.239114 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:29:59.238958 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ac3c9fb2-cbee-4fc5-91da-448c7482fe6c-oauth-serving-cert\") pod \"console-7d4d5f7c4f-d5vdz\" (UID: \"ac3c9fb2-cbee-4fc5-91da-448c7482fe6c\") " pod="openshift-console/console-7d4d5f7c4f-d5vdz" Jul 12 19:29:59.239114 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:29:59.239001 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ac3c9fb2-cbee-4fc5-91da-448c7482fe6c-console-serving-cert\") pod \"console-7d4d5f7c4f-d5vdz\" (UID: \"ac3c9fb2-cbee-4fc5-91da-448c7482fe6c\") " pod="openshift-console/console-7d4d5f7c4f-d5vdz" Jul 12 19:29:59.239114 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:29:59.239044 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txflp\" (UniqueName: \"kubernetes.io/projected/ac3c9fb2-cbee-4fc5-91da-448c7482fe6c-kube-api-access-txflp\") pod \"console-7d4d5f7c4f-d5vdz\" (UID: \"ac3c9fb2-cbee-4fc5-91da-448c7482fe6c\") " pod="openshift-console/console-7d4d5f7c4f-d5vdz" Jul 12 19:29:59.239114 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:29:59.239097 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ac3c9fb2-cbee-4fc5-91da-448c7482fe6c-service-ca\") pod \"console-7d4d5f7c4f-d5vdz\" (UID: \"ac3c9fb2-cbee-4fc5-91da-448c7482fe6c\") " pod="openshift-console/console-7d4d5f7c4f-d5vdz" Jul 12 19:29:59.239782 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:29:59.239756 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ac3c9fb2-cbee-4fc5-91da-448c7482fe6c-console-config\") pod \"console-7d4d5f7c4f-d5vdz\" (UID: \"ac3c9fb2-cbee-4fc5-91da-448c7482fe6c\") " pod="openshift-console/console-7d4d5f7c4f-d5vdz" Jul 12 19:29:59.239922 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:29:59.239756 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ac3c9fb2-cbee-4fc5-91da-448c7482fe6c-oauth-serving-cert\") pod \"console-7d4d5f7c4f-d5vdz\" (UID: \"ac3c9fb2-cbee-4fc5-91da-448c7482fe6c\") " pod="openshift-console/console-7d4d5f7c4f-d5vdz" Jul 12 19:29:59.239922 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:29:59.239871 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ac3c9fb2-cbee-4fc5-91da-448c7482fe6c-service-ca\") pod \"console-7d4d5f7c4f-d5vdz\" (UID: \"ac3c9fb2-cbee-4fc5-91da-448c7482fe6c\") " pod="openshift-console/console-7d4d5f7c4f-d5vdz" Jul 12 19:29:59.240010 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:29:59.239937 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ac3c9fb2-cbee-4fc5-91da-448c7482fe6c-trusted-ca-bundle\") pod \"console-7d4d5f7c4f-d5vdz\" (UID: \"ac3c9fb2-cbee-4fc5-91da-448c7482fe6c\") " pod="openshift-console/console-7d4d5f7c4f-d5vdz" Jul 12 19:29:59.241295 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:29:59.241278 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ac3c9fb2-cbee-4fc5-91da-448c7482fe6c-console-oauth-config\") pod \"console-7d4d5f7c4f-d5vdz\" (UID: \"ac3c9fb2-cbee-4fc5-91da-448c7482fe6c\") " pod="openshift-console/console-7d4d5f7c4f-d5vdz" Jul 12 19:29:59.241521 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:29:59.241503 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ac3c9fb2-cbee-4fc5-91da-448c7482fe6c-console-serving-cert\") pod \"console-7d4d5f7c4f-d5vdz\" (UID: \"ac3c9fb2-cbee-4fc5-91da-448c7482fe6c\") " pod="openshift-console/console-7d4d5f7c4f-d5vdz" Jul 12 19:29:59.246591 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:29:59.246569 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-txflp\" (UniqueName: \"kubernetes.io/projected/ac3c9fb2-cbee-4fc5-91da-448c7482fe6c-kube-api-access-txflp\") pod \"console-7d4d5f7c4f-d5vdz\" (UID: \"ac3c9fb2-cbee-4fc5-91da-448c7482fe6c\") " pod="openshift-console/console-7d4d5f7c4f-d5vdz" Jul 12 19:29:59.335184 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:29:59.335105 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7d4d5f7c4f-d5vdz" Jul 12 19:29:59.460028 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:29:59.460003 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7d4d5f7c4f-d5vdz"] Jul 12 19:29:59.462563 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:29:59.462539 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac3c9fb2_cbee_4fc5_91da_448c7482fe6c.slice/crio-2a83af09aecb38ed588b59ff751e8e4e018606d48fcfa8535fc023891d135dd9 WatchSource:0}: Error finding container 2a83af09aecb38ed588b59ff751e8e4e018606d48fcfa8535fc023891d135dd9: Status 404 returned error can't find the container with id 2a83af09aecb38ed588b59ff751e8e4e018606d48fcfa8535fc023891d135dd9 Jul 12 19:29:59.822546 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:29:59.822509 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:29:59.896730 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:29:59.896692 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7d4d5f7c4f-d5vdz" event={"ID":"ac3c9fb2-cbee-4fc5-91da-448c7482fe6c","Type":"ContainerStarted","Data":"2b54d6b021cdbf7a704ddae120c8e104a2e99d217589352eafb1cdd0bc17789d"} Jul 12 19:29:59.896949 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:29:59.896749 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7d4d5f7c4f-d5vdz" event={"ID":"ac3c9fb2-cbee-4fc5-91da-448c7482fe6c","Type":"ContainerStarted","Data":"2a83af09aecb38ed588b59ff751e8e4e018606d48fcfa8535fc023891d135dd9"} Jul 12 19:29:59.927825 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:29:59.927770 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7d4d5f7c4f-d5vdz" podStartSLOduration=0.9277506 podStartE2EDuration="927.7506ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 19:29:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 19:29:59.926387921 +0000 UTC m=+464.658960449" watchObservedRunningTime="2026-07-12 19:29:59.9277506 +0000 UTC m=+464.660323090" Jul 12 19:29:59.929907 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:29:59.929856 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:30:05.772902 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:30:05.772855 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:30:09.335444 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:30:09.335399 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7d4d5f7c4f-d5vdz" Jul 12 19:30:09.335801 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:30:09.335500 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7d4d5f7c4f-d5vdz" Jul 12 19:30:09.339802 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:30:09.339781 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7d4d5f7c4f-d5vdz" Jul 12 19:30:09.941252 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:30:09.941226 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7d4d5f7c4f-d5vdz" Jul 12 19:30:09.982499 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:30:09.982463 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-554ff9c978-8zmg2"] Jul 12 19:30:10.079059 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:30:10.079030 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:30:10.178829 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:30:10.178798 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:30:10.279946 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:30:10.279844 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:30:17.895521 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:30:17.895491 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:30:17.993810 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:30:17.993767 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:30:18.126491 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:30:18.126460 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:30:35.007158 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:30:35.007107 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-554ff9c978-8zmg2" podUID="e1f9c14d-7d55-423b-802d-086f272ec50b" containerName="console" containerID="cri-o://bd70ea03af01f2987f13887e795f0c49c028659beb515fd5a4397af48c8c9005" gracePeriod=15 Jul 12 19:30:35.244187 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:30:35.244158 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-554ff9c978-8zmg2" Jul 12 19:30:35.262087 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:30:35.261308 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e1f9c14d-7d55-423b-802d-086f272ec50b-console-serving-cert\") pod \"e1f9c14d-7d55-423b-802d-086f272ec50b\" (UID: \"e1f9c14d-7d55-423b-802d-086f272ec50b\") " Jul 12 19:30:35.262087 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:30:35.261390 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e1f9c14d-7d55-423b-802d-086f272ec50b-service-ca\") pod \"e1f9c14d-7d55-423b-802d-086f272ec50b\" (UID: \"e1f9c14d-7d55-423b-802d-086f272ec50b\") " Jul 12 19:30:35.262087 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:30:35.261439 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e1f9c14d-7d55-423b-802d-086f272ec50b-console-config\") pod \"e1f9c14d-7d55-423b-802d-086f272ec50b\" (UID: \"e1f9c14d-7d55-423b-802d-086f272ec50b\") " Jul 12 19:30:35.262087 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:30:35.261462 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e1f9c14d-7d55-423b-802d-086f272ec50b-console-oauth-config\") pod \"e1f9c14d-7d55-423b-802d-086f272ec50b\" (UID: \"e1f9c14d-7d55-423b-802d-086f272ec50b\") " Jul 12 19:30:35.262087 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:30:35.261515 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e1f9c14d-7d55-423b-802d-086f272ec50b-trusted-ca-bundle\") pod \"e1f9c14d-7d55-423b-802d-086f272ec50b\" (UID: \"e1f9c14d-7d55-423b-802d-086f272ec50b\") " Jul 12 19:30:35.262087 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:30:35.261547 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e1f9c14d-7d55-423b-802d-086f272ec50b-oauth-serving-cert\") pod \"e1f9c14d-7d55-423b-802d-086f272ec50b\" (UID: \"e1f9c14d-7d55-423b-802d-086f272ec50b\") " Jul 12 19:30:35.262087 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:30:35.261579 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b6kht\" (UniqueName: \"kubernetes.io/projected/e1f9c14d-7d55-423b-802d-086f272ec50b-kube-api-access-b6kht\") pod \"e1f9c14d-7d55-423b-802d-086f272ec50b\" (UID: \"e1f9c14d-7d55-423b-802d-086f272ec50b\") " Jul 12 19:30:35.262538 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:30:35.262126 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1f9c14d-7d55-423b-802d-086f272ec50b-service-ca" (OuterVolumeSpecName: "service-ca") pod "e1f9c14d-7d55-423b-802d-086f272ec50b" (UID: "e1f9c14d-7d55-423b-802d-086f272ec50b"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 12 19:30:35.262538 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:30:35.262135 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1f9c14d-7d55-423b-802d-086f272ec50b-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "e1f9c14d-7d55-423b-802d-086f272ec50b" (UID: "e1f9c14d-7d55-423b-802d-086f272ec50b"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 12 19:30:35.262538 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:30:35.262368 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1f9c14d-7d55-423b-802d-086f272ec50b-console-config" (OuterVolumeSpecName: "console-config") pod "e1f9c14d-7d55-423b-802d-086f272ec50b" (UID: "e1f9c14d-7d55-423b-802d-086f272ec50b"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 12 19:30:35.262733 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:30:35.262661 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1f9c14d-7d55-423b-802d-086f272ec50b-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "e1f9c14d-7d55-423b-802d-086f272ec50b" (UID: "e1f9c14d-7d55-423b-802d-086f272ec50b"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 12 19:30:35.264429 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:30:35.264395 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e1f9c14d-7d55-423b-802d-086f272ec50b-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "e1f9c14d-7d55-423b-802d-086f272ec50b" (UID: "e1f9c14d-7d55-423b-802d-086f272ec50b"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 12 19:30:35.264548 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:30:35.264451 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e1f9c14d-7d55-423b-802d-086f272ec50b-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "e1f9c14d-7d55-423b-802d-086f272ec50b" (UID: "e1f9c14d-7d55-423b-802d-086f272ec50b"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 12 19:30:35.264949 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:30:35.264922 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1f9c14d-7d55-423b-802d-086f272ec50b-kube-api-access-b6kht" (OuterVolumeSpecName: "kube-api-access-b6kht") pod "e1f9c14d-7d55-423b-802d-086f272ec50b" (UID: "e1f9c14d-7d55-423b-802d-086f272ec50b"). InnerVolumeSpecName "kube-api-access-b6kht". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 12 19:30:35.362655 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:30:35.362619 2565 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e1f9c14d-7d55-423b-802d-086f272ec50b-console-config\") on node \"ip-10-0-139-170.ec2.internal\" DevicePath \"\"" Jul 12 19:30:35.362655 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:30:35.362648 2565 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e1f9c14d-7d55-423b-802d-086f272ec50b-console-oauth-config\") on node \"ip-10-0-139-170.ec2.internal\" DevicePath \"\"" Jul 12 19:30:35.362655 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:30:35.362658 2565 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e1f9c14d-7d55-423b-802d-086f272ec50b-trusted-ca-bundle\") on node \"ip-10-0-139-170.ec2.internal\" DevicePath \"\"" Jul 12 19:30:35.362901 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:30:35.362666 2565 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e1f9c14d-7d55-423b-802d-086f272ec50b-oauth-serving-cert\") on node \"ip-10-0-139-170.ec2.internal\" DevicePath \"\"" Jul 12 19:30:35.362901 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:30:35.362675 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b6kht\" (UniqueName: \"kubernetes.io/projected/e1f9c14d-7d55-423b-802d-086f272ec50b-kube-api-access-b6kht\") on node \"ip-10-0-139-170.ec2.internal\" DevicePath \"\"" Jul 12 19:30:35.362901 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:30:35.362684 2565 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e1f9c14d-7d55-423b-802d-086f272ec50b-console-serving-cert\") on node \"ip-10-0-139-170.ec2.internal\" DevicePath \"\"" Jul 12 19:30:35.362901 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:30:35.362692 2565 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e1f9c14d-7d55-423b-802d-086f272ec50b-service-ca\") on node \"ip-10-0-139-170.ec2.internal\" DevicePath \"\"" Jul 12 19:30:36.044854 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:30:36.044821 2565 generic.go:356] "Generic (PLEG): container finished" podID="e1f9c14d-7d55-423b-802d-086f272ec50b" containerID="bd70ea03af01f2987f13887e795f0c49c028659beb515fd5a4397af48c8c9005" exitCode=0 Jul 12 19:30:36.045242 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:30:36.044862 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-554ff9c978-8zmg2" Jul 12 19:30:36.045242 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:30:36.044905 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-554ff9c978-8zmg2" event={"ID":"e1f9c14d-7d55-423b-802d-086f272ec50b","Type":"ContainerDied","Data":"bd70ea03af01f2987f13887e795f0c49c028659beb515fd5a4397af48c8c9005"} Jul 12 19:30:36.045242 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:30:36.044956 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-554ff9c978-8zmg2" event={"ID":"e1f9c14d-7d55-423b-802d-086f272ec50b","Type":"ContainerDied","Data":"2541486d1cbead7266992d508b5db2d310f0493ab3ccc10e4cb1e3c7646ab6e8"} Jul 12 19:30:36.045242 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:30:36.044978 2565 scope.go:117] "RemoveContainer" containerID="bd70ea03af01f2987f13887e795f0c49c028659beb515fd5a4397af48c8c9005" Jul 12 19:30:36.053056 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:30:36.053039 2565 scope.go:117] "RemoveContainer" containerID="bd70ea03af01f2987f13887e795f0c49c028659beb515fd5a4397af48c8c9005" Jul 12 19:30:36.053297 ip-10-0-139-170 kubenswrapper[2565]: E0712 19:30:36.053283 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd70ea03af01f2987f13887e795f0c49c028659beb515fd5a4397af48c8c9005\": container with ID starting with bd70ea03af01f2987f13887e795f0c49c028659beb515fd5a4397af48c8c9005 not found: ID does not exist" containerID="bd70ea03af01f2987f13887e795f0c49c028659beb515fd5a4397af48c8c9005" Jul 12 19:30:36.053350 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:30:36.053304 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd70ea03af01f2987f13887e795f0c49c028659beb515fd5a4397af48c8c9005"} err="failed to get container status \"bd70ea03af01f2987f13887e795f0c49c028659beb515fd5a4397af48c8c9005\": rpc error: code = NotFound desc = could not find container \"bd70ea03af01f2987f13887e795f0c49c028659beb515fd5a4397af48c8c9005\": container with ID starting with bd70ea03af01f2987f13887e795f0c49c028659beb515fd5a4397af48c8c9005 not found: ID does not exist" Jul 12 19:30:36.077106 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:30:36.077077 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-554ff9c978-8zmg2"] Jul 12 19:30:36.090633 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:30:36.090603 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-554ff9c978-8zmg2"] Jul 12 19:30:37.856854 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:30:37.856822 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e1f9c14d-7d55-423b-802d-086f272ec50b" path="/var/lib/kubelet/pods/e1f9c14d-7d55-423b-802d-086f272ec50b/volumes" Jul 12 19:30:47.473705 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:30:47.473668 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:30:47.523395 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:30:47.523363 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:31:17.920729 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:31:17.920694 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:31:18.202640 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:31:18.202546 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:31:24.240499 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:31:24.240455 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:31:24.334578 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:31:24.334544 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:31:24.407321 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:31:24.407284 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:31:24.485866 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:31:24.485835 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:31:27.769952 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:31:27.769911 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:31:27.875533 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:31:27.875497 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:31:57.029675 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:31:57.029626 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:31:57.524334 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:31:57.524275 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:31:57.659552 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:31:57.659506 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:31:57.938971 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:31:57.938936 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:31:58.013086 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:31:58.013049 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:32:15.722983 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:32:15.722953 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k5h9m_8e016c2e-0c3a-49da-83d4-f526738c1cbd/ovn-acl-logging/0.log" Jul 12 19:32:15.723537 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:32:15.723234 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k5h9m_8e016c2e-0c3a-49da-83d4-f526738c1cbd/ovn-acl-logging/0.log" Jul 12 19:32:48.156354 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:32:48.156263 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:32:49.422183 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:32:49.422140 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:33:14.974659 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:33:14.974615 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:33:21.656737 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:33:21.656705 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:33:32.158264 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:33:32.158226 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:33:42.652852 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:33:42.652819 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:33:54.679664 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:33:54.679627 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:34:02.948699 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:34:02.948662 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:34:06.855071 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:34:06.855036 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:34:17.329997 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:34:17.329914 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-sshcb"] Jul 12 19:34:17.340583 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:34:17.340533 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e1f9c14d-7d55-423b-802d-086f272ec50b" containerName="console" Jul 12 19:34:17.340583 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:34:17.340588 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="e1f9c14d-7d55-423b-802d-086f272ec50b" containerName="console" Jul 12 19:34:17.340806 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:34:17.340713 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="e1f9c14d-7d55-423b-802d-086f272ec50b" containerName="console" Jul 12 19:34:17.344081 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:34:17.344048 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-sshcb"] Jul 12 19:34:17.344235 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:34:17.344084 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-sshcb" Jul 12 19:34:17.346621 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:34:17.346598 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 12 19:34:17.346755 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:34:17.346646 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-48spd\"" Jul 12 19:34:17.346755 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:34:17.346652 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Jul 12 19:34:17.347651 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:34:17.347619 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 12 19:34:17.362571 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:34:17.362545 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:34:17.505255 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:34:17.505224 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/41da0a81-53dc-45a9-a44e-4b3b1acdd405-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-sshcb\" (UID: \"41da0a81-53dc-45a9-a44e-4b3b1acdd405\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-sshcb" Jul 12 19:34:17.505448 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:34:17.505271 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41da0a81-53dc-45a9-a44e-4b3b1acdd405-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-sshcb\" (UID: \"41da0a81-53dc-45a9-a44e-4b3b1acdd405\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-sshcb" Jul 12 19:34:17.505448 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:34:17.505320 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/41da0a81-53dc-45a9-a44e-4b3b1acdd405-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-sshcb\" (UID: \"41da0a81-53dc-45a9-a44e-4b3b1acdd405\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-sshcb" Jul 12 19:34:17.505448 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:34:17.505345 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2d7jm\" (UniqueName: \"kubernetes.io/projected/41da0a81-53dc-45a9-a44e-4b3b1acdd405-kube-api-access-2d7jm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-sshcb\" (UID: \"41da0a81-53dc-45a9-a44e-4b3b1acdd405\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-sshcb" Jul 12 19:34:17.505448 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:34:17.505412 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/41da0a81-53dc-45a9-a44e-4b3b1acdd405-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-sshcb\" (UID: \"41da0a81-53dc-45a9-a44e-4b3b1acdd405\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-sshcb" Jul 12 19:34:17.505661 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:34:17.505459 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/41da0a81-53dc-45a9-a44e-4b3b1acdd405-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-sshcb\" (UID: \"41da0a81-53dc-45a9-a44e-4b3b1acdd405\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-sshcb" Jul 12 19:34:17.606252 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:34:17.606155 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2d7jm\" (UniqueName: \"kubernetes.io/projected/41da0a81-53dc-45a9-a44e-4b3b1acdd405-kube-api-access-2d7jm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-sshcb\" (UID: \"41da0a81-53dc-45a9-a44e-4b3b1acdd405\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-sshcb" Jul 12 19:34:17.606252 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:34:17.606202 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/41da0a81-53dc-45a9-a44e-4b3b1acdd405-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-sshcb\" (UID: \"41da0a81-53dc-45a9-a44e-4b3b1acdd405\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-sshcb" Jul 12 19:34:17.606466 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:34:17.606254 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/41da0a81-53dc-45a9-a44e-4b3b1acdd405-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-sshcb\" (UID: \"41da0a81-53dc-45a9-a44e-4b3b1acdd405\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-sshcb" Jul 12 19:34:17.606466 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:34:17.606312 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/41da0a81-53dc-45a9-a44e-4b3b1acdd405-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-sshcb\" (UID: \"41da0a81-53dc-45a9-a44e-4b3b1acdd405\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-sshcb" Jul 12 19:34:17.606466 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:34:17.606349 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41da0a81-53dc-45a9-a44e-4b3b1acdd405-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-sshcb\" (UID: \"41da0a81-53dc-45a9-a44e-4b3b1acdd405\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-sshcb" Jul 12 19:34:17.606466 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:34:17.606379 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/41da0a81-53dc-45a9-a44e-4b3b1acdd405-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-sshcb\" (UID: \"41da0a81-53dc-45a9-a44e-4b3b1acdd405\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-sshcb" Jul 12 19:34:17.606734 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:34:17.606711 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/41da0a81-53dc-45a9-a44e-4b3b1acdd405-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-sshcb\" (UID: \"41da0a81-53dc-45a9-a44e-4b3b1acdd405\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-sshcb" Jul 12 19:34:17.606798 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:34:17.606744 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41da0a81-53dc-45a9-a44e-4b3b1acdd405-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-sshcb\" (UID: \"41da0a81-53dc-45a9-a44e-4b3b1acdd405\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-sshcb" Jul 12 19:34:17.606834 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:34:17.606816 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/41da0a81-53dc-45a9-a44e-4b3b1acdd405-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-sshcb\" (UID: \"41da0a81-53dc-45a9-a44e-4b3b1acdd405\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-sshcb" Jul 12 19:34:17.608530 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:34:17.608510 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/41da0a81-53dc-45a9-a44e-4b3b1acdd405-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-sshcb\" (UID: \"41da0a81-53dc-45a9-a44e-4b3b1acdd405\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-sshcb" Jul 12 19:34:17.608809 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:34:17.608794 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/41da0a81-53dc-45a9-a44e-4b3b1acdd405-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-sshcb\" (UID: \"41da0a81-53dc-45a9-a44e-4b3b1acdd405\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-sshcb" Jul 12 19:34:17.612994 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:34:17.612976 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2d7jm\" (UniqueName: \"kubernetes.io/projected/41da0a81-53dc-45a9-a44e-4b3b1acdd405-kube-api-access-2d7jm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-sshcb\" (UID: \"41da0a81-53dc-45a9-a44e-4b3b1acdd405\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-sshcb" Jul 12 19:34:17.656088 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:34:17.656037 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-sshcb" Jul 12 19:34:17.784166 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:34:17.783963 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-sshcb"] Jul 12 19:34:17.787029 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:34:17.787000 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41da0a81_53dc_45a9_a44e_4b3b1acdd405.slice/crio-0309eaa0e7ec7c91b2b3ab0f4166c24dd8b99f6cdbf27ff5ebce8034b01e1154 WatchSource:0}: Error finding container 0309eaa0e7ec7c91b2b3ab0f4166c24dd8b99f6cdbf27ff5ebce8034b01e1154: Status 404 returned error can't find the container with id 0309eaa0e7ec7c91b2b3ab0f4166c24dd8b99f6cdbf27ff5ebce8034b01e1154 Jul 12 19:34:17.788782 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:34:17.788763 2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 12 19:34:17.982925 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:34:17.982865 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-sshcb" event={"ID":"41da0a81-53dc-45a9-a44e-4b3b1acdd405","Type":"ContainerStarted","Data":"0309eaa0e7ec7c91b2b3ab0f4166c24dd8b99f6cdbf27ff5ebce8034b01e1154"} Jul 12 19:34:18.988871 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:34:18.988837 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-sshcb" event={"ID":"41da0a81-53dc-45a9-a44e-4b3b1acdd405","Type":"ContainerStarted","Data":"26f2d2f6e4e06e4c73462dae7c882eced7ea0232f42537b114b751f4c0a90cfa"} Jul 12 19:34:19.004439 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:34:19.004391 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-sshcb" podStartSLOduration=0.987707645 podStartE2EDuration="2.004377035s" totalImagesPullingTime="1.01666939s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 19:34:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 19:34:19.003210998 +0000 UTC m=+723.735783499" watchObservedRunningTime="2026-07-12 19:34:19.004377035 +0000 UTC m=+723.736949525" Jul 12 19:34:27.451641 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:34:27.451602 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:34:27.656352 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:34:27.656314 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-sshcb" Jul 12 19:34:27.656528 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:34:27.656410 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-sshcb" Jul 12 19:34:27.668329 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:34:27.668304 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-sshcb" Jul 12 19:34:28.036911 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:34:28.036856 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-sshcb" Jul 12 19:34:47.891123 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:34:47.891084 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:34:47.989550 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:34:47.989517 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:35:24.026568 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:35:24.026528 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:35:24.127749 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:35:24.127713 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:35:25.088761 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:35:25.088729 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:35:25.190241 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:35:25.190209 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:36:19.365398 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:36:19.365360 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:36:35.235130 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:36:35.235094 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:36:35.284848 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:36:35.284813 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:36:35.904832 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:36:35.904796 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:36:37.358737 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:36:37.358706 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:37:14.903774 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:37:14.903686 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:37:15.004127 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:37:15.004095 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:37:15.111751 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:37:15.111720 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:37:15.747279 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:37:15.747244 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k5h9m_8e016c2e-0c3a-49da-83d4-f526738c1cbd/ovn-acl-logging/0.log" Jul 12 19:37:15.748195 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:37:15.748167 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k5h9m_8e016c2e-0c3a-49da-83d4-f526738c1cbd/ovn-acl-logging/0.log" Jul 12 19:37:20.864781 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:37:20.864746 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:37:20.914306 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:37:20.914270 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:37:20.989142 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:37:20.989110 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:37:21.088917 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:37:21.088868 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:37:21.202855 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:37:21.202821 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:37:26.792017 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:37:26.791977 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:37:26.854746 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:37:26.854719 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:41:21.315101 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:41:21.315065 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:42:15.771997 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:42:15.771924 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k5h9m_8e016c2e-0c3a-49da-83d4-f526738c1cbd/ovn-acl-logging/0.log" Jul 12 19:42:15.772819 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:42:15.772800 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k5h9m_8e016c2e-0c3a-49da-83d4-f526738c1cbd/ovn-acl-logging/0.log" Jul 12 19:44:22.226398 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:44:22.226349 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:44:29.436962 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:44:29.436877 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:47:15.795440 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:47:15.795414 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k5h9m_8e016c2e-0c3a-49da-83d4-f526738c1cbd/ovn-acl-logging/0.log" Jul 12 19:47:15.798106 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:47:15.796804 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k5h9m_8e016c2e-0c3a-49da-83d4-f526738c1cbd/ovn-acl-logging/0.log" Jul 12 19:48:11.213326 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:48:11.213251 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:48:19.738745 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:48:19.738680 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:48:27.990727 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:48:27.990692 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:48:33.083065 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:48:33.083027 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:48:36.018086 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:48:36.018049 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:48:36.109389 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:48:36.109356 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:48:36.457633 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:48:36.457592 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:48:42.023072 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:48:42.023038 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:48:42.077293 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:48:42.077256 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:48:42.185300 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:48:42.185269 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-a4a89c5e-openshift-default-586d7bbd5f-gw9vc"] Jul 12 19:48:42.188650 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:48:42.188628 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a4a89c5e-openshift-default-586d7bbd5f-gw9vc" Jul 12 19:48:42.191011 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:48:42.190988 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-a4a89c5e-openshift-default-dockercfg-cd4rp\"" Jul 12 19:48:42.209806 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:48:42.209786 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-a4a89c5e-openshift-default-586d7bbd5f-gw9vc"] Jul 12 19:48:42.212868 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:48:42.212844 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:48:42.267711 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:48:42.267683 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee192e78-c59a-49e1-8c75-e312994f86e7-credential-socket\") pod \"e2e-ait-a4a89c5e-openshift-default-586d7bbd5f-gw9vc\" (UID: \"ee192e78-c59a-49e1-8c75-e312994f86e7\") " pod="openshift-ingress/e2e-ait-a4a89c5e-openshift-default-586d7bbd5f-gw9vc" Jul 12 19:48:42.267843 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:48:42.267717 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee192e78-c59a-49e1-8c75-e312994f86e7-istio-token\") pod \"e2e-ait-a4a89c5e-openshift-default-586d7bbd5f-gw9vc\" (UID: \"ee192e78-c59a-49e1-8c75-e312994f86e7\") " pod="openshift-ingress/e2e-ait-a4a89c5e-openshift-default-586d7bbd5f-gw9vc" Jul 12 19:48:42.267843 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:48:42.267738 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee192e78-c59a-49e1-8c75-e312994f86e7-istio-podinfo\") pod \"e2e-ait-a4a89c5e-openshift-default-586d7bbd5f-gw9vc\" (UID: \"ee192e78-c59a-49e1-8c75-e312994f86e7\") " pod="openshift-ingress/e2e-ait-a4a89c5e-openshift-default-586d7bbd5f-gw9vc" Jul 12 19:48:42.267968 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:48:42.267851 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee192e78-c59a-49e1-8c75-e312994f86e7-istio-envoy\") pod \"e2e-ait-a4a89c5e-openshift-default-586d7bbd5f-gw9vc\" (UID: \"ee192e78-c59a-49e1-8c75-e312994f86e7\") " pod="openshift-ingress/e2e-ait-a4a89c5e-openshift-default-586d7bbd5f-gw9vc" Jul 12 19:48:42.267968 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:48:42.267875 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee192e78-c59a-49e1-8c75-e312994f86e7-istio-data\") pod \"e2e-ait-a4a89c5e-openshift-default-586d7bbd5f-gw9vc\" (UID: \"ee192e78-c59a-49e1-8c75-e312994f86e7\") " pod="openshift-ingress/e2e-ait-a4a89c5e-openshift-default-586d7bbd5f-gw9vc" Jul 12 19:48:42.267968 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:48:42.267913 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee192e78-c59a-49e1-8c75-e312994f86e7-workload-certs\") pod \"e2e-ait-a4a89c5e-openshift-default-586d7bbd5f-gw9vc\" (UID: \"ee192e78-c59a-49e1-8c75-e312994f86e7\") " pod="openshift-ingress/e2e-ait-a4a89c5e-openshift-default-586d7bbd5f-gw9vc" Jul 12 19:48:42.267968 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:48:42.267938 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee192e78-c59a-49e1-8c75-e312994f86e7-istiod-ca-cert\") pod \"e2e-ait-a4a89c5e-openshift-default-586d7bbd5f-gw9vc\" (UID: \"ee192e78-c59a-49e1-8c75-e312994f86e7\") " pod="openshift-ingress/e2e-ait-a4a89c5e-openshift-default-586d7bbd5f-gw9vc" Jul 12 19:48:42.268148 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:48:42.267970 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nd65z\" (UniqueName: \"kubernetes.io/projected/ee192e78-c59a-49e1-8c75-e312994f86e7-kube-api-access-nd65z\") pod \"e2e-ait-a4a89c5e-openshift-default-586d7bbd5f-gw9vc\" (UID: \"ee192e78-c59a-49e1-8c75-e312994f86e7\") " pod="openshift-ingress/e2e-ait-a4a89c5e-openshift-default-586d7bbd5f-gw9vc" Jul 12 19:48:42.268148 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:48:42.267996 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee192e78-c59a-49e1-8c75-e312994f86e7-workload-socket\") pod \"e2e-ait-a4a89c5e-openshift-default-586d7bbd5f-gw9vc\" (UID: \"ee192e78-c59a-49e1-8c75-e312994f86e7\") " pod="openshift-ingress/e2e-ait-a4a89c5e-openshift-default-586d7bbd5f-gw9vc" Jul 12 19:48:42.273819 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:48:42.273752 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:48:42.369310 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:48:42.369278 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee192e78-c59a-49e1-8c75-e312994f86e7-istiod-ca-cert\") pod \"e2e-ait-a4a89c5e-openshift-default-586d7bbd5f-gw9vc\" (UID: \"ee192e78-c59a-49e1-8c75-e312994f86e7\") " pod="openshift-ingress/e2e-ait-a4a89c5e-openshift-default-586d7bbd5f-gw9vc" Jul 12 19:48:42.369310 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:48:42.369318 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nd65z\" (UniqueName: \"kubernetes.io/projected/ee192e78-c59a-49e1-8c75-e312994f86e7-kube-api-access-nd65z\") pod \"e2e-ait-a4a89c5e-openshift-default-586d7bbd5f-gw9vc\" (UID: \"ee192e78-c59a-49e1-8c75-e312994f86e7\") " pod="openshift-ingress/e2e-ait-a4a89c5e-openshift-default-586d7bbd5f-gw9vc" Jul 12 19:48:42.369558 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:48:42.369345 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee192e78-c59a-49e1-8c75-e312994f86e7-workload-socket\") pod \"e2e-ait-a4a89c5e-openshift-default-586d7bbd5f-gw9vc\" (UID: \"ee192e78-c59a-49e1-8c75-e312994f86e7\") " pod="openshift-ingress/e2e-ait-a4a89c5e-openshift-default-586d7bbd5f-gw9vc" Jul 12 19:48:42.369558 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:48:42.369371 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee192e78-c59a-49e1-8c75-e312994f86e7-credential-socket\") pod \"e2e-ait-a4a89c5e-openshift-default-586d7bbd5f-gw9vc\" (UID: \"ee192e78-c59a-49e1-8c75-e312994f86e7\") " pod="openshift-ingress/e2e-ait-a4a89c5e-openshift-default-586d7bbd5f-gw9vc" Jul 12 19:48:42.369558 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:48:42.369387 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee192e78-c59a-49e1-8c75-e312994f86e7-istio-token\") pod \"e2e-ait-a4a89c5e-openshift-default-586d7bbd5f-gw9vc\" (UID: \"ee192e78-c59a-49e1-8c75-e312994f86e7\") " pod="openshift-ingress/e2e-ait-a4a89c5e-openshift-default-586d7bbd5f-gw9vc" Jul 12 19:48:42.369558 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:48:42.369409 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee192e78-c59a-49e1-8c75-e312994f86e7-istio-podinfo\") pod \"e2e-ait-a4a89c5e-openshift-default-586d7bbd5f-gw9vc\" (UID: \"ee192e78-c59a-49e1-8c75-e312994f86e7\") " pod="openshift-ingress/e2e-ait-a4a89c5e-openshift-default-586d7bbd5f-gw9vc" Jul 12 19:48:42.369558 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:48:42.369474 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee192e78-c59a-49e1-8c75-e312994f86e7-istio-envoy\") pod \"e2e-ait-a4a89c5e-openshift-default-586d7bbd5f-gw9vc\" (UID: \"ee192e78-c59a-49e1-8c75-e312994f86e7\") " pod="openshift-ingress/e2e-ait-a4a89c5e-openshift-default-586d7bbd5f-gw9vc" Jul 12 19:48:42.369558 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:48:42.369498 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee192e78-c59a-49e1-8c75-e312994f86e7-istio-data\") pod \"e2e-ait-a4a89c5e-openshift-default-586d7bbd5f-gw9vc\" (UID: \"ee192e78-c59a-49e1-8c75-e312994f86e7\") " pod="openshift-ingress/e2e-ait-a4a89c5e-openshift-default-586d7bbd5f-gw9vc" Jul 12 19:48:42.369558 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:48:42.369523 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee192e78-c59a-49e1-8c75-e312994f86e7-workload-certs\") pod \"e2e-ait-a4a89c5e-openshift-default-586d7bbd5f-gw9vc\" (UID: \"ee192e78-c59a-49e1-8c75-e312994f86e7\") " pod="openshift-ingress/e2e-ait-a4a89c5e-openshift-default-586d7bbd5f-gw9vc" Jul 12 19:48:42.369932 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:48:42.369819 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee192e78-c59a-49e1-8c75-e312994f86e7-credential-socket\") pod \"e2e-ait-a4a89c5e-openshift-default-586d7bbd5f-gw9vc\" (UID: \"ee192e78-c59a-49e1-8c75-e312994f86e7\") " pod="openshift-ingress/e2e-ait-a4a89c5e-openshift-default-586d7bbd5f-gw9vc" Jul 12 19:48:42.369932 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:48:42.369863 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee192e78-c59a-49e1-8c75-e312994f86e7-workload-socket\") pod \"e2e-ait-a4a89c5e-openshift-default-586d7bbd5f-gw9vc\" (UID: \"ee192e78-c59a-49e1-8c75-e312994f86e7\") " pod="openshift-ingress/e2e-ait-a4a89c5e-openshift-default-586d7bbd5f-gw9vc" Jul 12 19:48:42.370052 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:48:42.369939 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee192e78-c59a-49e1-8c75-e312994f86e7-workload-certs\") pod \"e2e-ait-a4a89c5e-openshift-default-586d7bbd5f-gw9vc\" (UID: \"ee192e78-c59a-49e1-8c75-e312994f86e7\") " pod="openshift-ingress/e2e-ait-a4a89c5e-openshift-default-586d7bbd5f-gw9vc" Jul 12 19:48:42.370052 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:48:42.370041 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee192e78-c59a-49e1-8c75-e312994f86e7-istio-data\") pod \"e2e-ait-a4a89c5e-openshift-default-586d7bbd5f-gw9vc\" (UID: \"ee192e78-c59a-49e1-8c75-e312994f86e7\") " pod="openshift-ingress/e2e-ait-a4a89c5e-openshift-default-586d7bbd5f-gw9vc" Jul 12 19:48:42.370170 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:48:42.370148 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee192e78-c59a-49e1-8c75-e312994f86e7-istiod-ca-cert\") pod \"e2e-ait-a4a89c5e-openshift-default-586d7bbd5f-gw9vc\" (UID: \"ee192e78-c59a-49e1-8c75-e312994f86e7\") " pod="openshift-ingress/e2e-ait-a4a89c5e-openshift-default-586d7bbd5f-gw9vc" Jul 12 19:48:42.371692 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:48:42.371675 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee192e78-c59a-49e1-8c75-e312994f86e7-istio-envoy\") pod \"e2e-ait-a4a89c5e-openshift-default-586d7bbd5f-gw9vc\" (UID: \"ee192e78-c59a-49e1-8c75-e312994f86e7\") " pod="openshift-ingress/e2e-ait-a4a89c5e-openshift-default-586d7bbd5f-gw9vc" Jul 12 19:48:42.371988 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:48:42.371971 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee192e78-c59a-49e1-8c75-e312994f86e7-istio-podinfo\") pod \"e2e-ait-a4a89c5e-openshift-default-586d7bbd5f-gw9vc\" (UID: \"ee192e78-c59a-49e1-8c75-e312994f86e7\") " pod="openshift-ingress/e2e-ait-a4a89c5e-openshift-default-586d7bbd5f-gw9vc" Jul 12 19:48:42.377124 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:48:42.377090 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee192e78-c59a-49e1-8c75-e312994f86e7-istio-token\") pod \"e2e-ait-a4a89c5e-openshift-default-586d7bbd5f-gw9vc\" (UID: \"ee192e78-c59a-49e1-8c75-e312994f86e7\") " pod="openshift-ingress/e2e-ait-a4a89c5e-openshift-default-586d7bbd5f-gw9vc" Jul 12 19:48:42.377249 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:48:42.377169 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nd65z\" (UniqueName: \"kubernetes.io/projected/ee192e78-c59a-49e1-8c75-e312994f86e7-kube-api-access-nd65z\") pod \"e2e-ait-a4a89c5e-openshift-default-586d7bbd5f-gw9vc\" (UID: \"ee192e78-c59a-49e1-8c75-e312994f86e7\") " pod="openshift-ingress/e2e-ait-a4a89c5e-openshift-default-586d7bbd5f-gw9vc" Jul 12 19:48:42.498757 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:48:42.498725 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a4a89c5e-openshift-default-586d7bbd5f-gw9vc" Jul 12 19:48:42.632008 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:48:42.631982 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-a4a89c5e-openshift-default-586d7bbd5f-gw9vc"] Jul 12 19:48:42.633836 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:48:42.633809 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee192e78_c59a_49e1_8c75_e312994f86e7.slice/crio-9134a09ea1b5c71de1fc071613fbb9e8738e75f5f4267d1b48e2e3f0a8fff723 WatchSource:0}: Error finding container 9134a09ea1b5c71de1fc071613fbb9e8738e75f5f4267d1b48e2e3f0a8fff723: Status 404 returned error can't find the container with id 9134a09ea1b5c71de1fc071613fbb9e8738e75f5f4267d1b48e2e3f0a8fff723 Jul 12 19:48:42.635872 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:48:42.635838 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 12 19:48:42.636006 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:48:42.635925 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 12 19:48:42.636006 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:48:42.635956 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 12 19:48:43.430155 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:48:43.430115 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:48:43.514291 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:48:43.514242 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:48:43.572988 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:48:43.572942 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:48:43.610606 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:48:43.610569 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a4a89c5e-openshift-default-586d7bbd5f-gw9vc" event={"ID":"ee192e78-c59a-49e1-8c75-e312994f86e7","Type":"ContainerStarted","Data":"a789befff65e363c684831b63bff28872513fef248b434dd5c878c9d73800be6"} Jul 12 19:48:43.610804 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:48:43.610649 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a4a89c5e-openshift-default-586d7bbd5f-gw9vc" event={"ID":"ee192e78-c59a-49e1-8c75-e312994f86e7","Type":"ContainerStarted","Data":"9134a09ea1b5c71de1fc071613fbb9e8738e75f5f4267d1b48e2e3f0a8fff723"} Jul 12 19:48:43.632099 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:48:43.632046 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-a4a89c5e-openshift-default-586d7bbd5f-gw9vc" podStartSLOduration=1.63202826 podStartE2EDuration="1.63202826s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 19:48:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 19:48:43.631185872 +0000 UTC m=+1588.363758363" watchObservedRunningTime="2026-07-12 19:48:43.63202826 +0000 UTC m=+1588.364600799" Jul 12 19:48:43.946152 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:48:43.946095 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:48:44.057264 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:48:44.057230 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:48:44.158555 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:48:44.158516 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:48:44.499484 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:48:44.499449 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-a4a89c5e-openshift-default-586d7bbd5f-gw9vc" Jul 12 19:48:44.504228 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:48:44.504206 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-a4a89c5e-openshift-default-586d7bbd5f-gw9vc" Jul 12 19:48:44.615029 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:48:44.615004 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-a4a89c5e-openshift-default-586d7bbd5f-gw9vc" Jul 12 19:48:44.615941 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:48:44.615923 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-a4a89c5e-openshift-default-586d7bbd5f-gw9vc" Jul 12 19:48:45.711745 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:48:45.711711 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:48:48.464671 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:48:48.464604 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:48:48.607770 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:48:48.607739 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-a4a89c5e-openshift-default-586d7bbd5f-gw9vc"] Jul 12 19:48:48.607979 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:48:48.607931 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-a4a89c5e-openshift-default-586d7bbd5f-gw9vc" podUID="ee192e78-c59a-49e1-8c75-e312994f86e7" containerName="istio-proxy" containerID="cri-o://a789befff65e363c684831b63bff28872513fef248b434dd5c878c9d73800be6" gracePeriod=30 Jul 12 19:48:48.630237 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:48:48.630207 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:48:48.728545 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:48:48.728448 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:48:48.780355 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:48:48.780320 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:48:52.351365 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:48:52.351329 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:48:54.300619 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:48:54.300584 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:48:54.362751 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:48:54.362719 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:48:54.496095 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:48:54.496061 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:48:54.596351 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:48:54.596322 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:48:54.841479 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:48:54.841422 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a4a89c5e-openshift-default-586d7bbd5f-gw9vc" Jul 12 19:48:54.993199 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:48:54.993165 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee192e78-c59a-49e1-8c75-e312994f86e7-istiod-ca-cert\") pod \"ee192e78-c59a-49e1-8c75-e312994f86e7\" (UID: \"ee192e78-c59a-49e1-8c75-e312994f86e7\") " Jul 12 19:48:54.993393 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:48:54.993220 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee192e78-c59a-49e1-8c75-e312994f86e7-credential-socket\") pod \"ee192e78-c59a-49e1-8c75-e312994f86e7\" (UID: \"ee192e78-c59a-49e1-8c75-e312994f86e7\") " Jul 12 19:48:54.993393 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:48:54.993267 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee192e78-c59a-49e1-8c75-e312994f86e7-istio-token\") pod \"ee192e78-c59a-49e1-8c75-e312994f86e7\" (UID: \"ee192e78-c59a-49e1-8c75-e312994f86e7\") " Jul 12 19:48:54.993393 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:48:54.993286 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee192e78-c59a-49e1-8c75-e312994f86e7-istio-envoy\") pod \"ee192e78-c59a-49e1-8c75-e312994f86e7\" (UID: \"ee192e78-c59a-49e1-8c75-e312994f86e7\") " Jul 12 19:48:54.993393 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:48:54.993308 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee192e78-c59a-49e1-8c75-e312994f86e7-workload-socket\") pod \"ee192e78-c59a-49e1-8c75-e312994f86e7\" (UID: \"ee192e78-c59a-49e1-8c75-e312994f86e7\") " Jul 12 19:48:54.993393 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:48:54.993366 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee192e78-c59a-49e1-8c75-e312994f86e7-istio-podinfo\") pod \"ee192e78-c59a-49e1-8c75-e312994f86e7\" (UID: \"ee192e78-c59a-49e1-8c75-e312994f86e7\") " Jul 12 19:48:54.993626 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:48:54.993408 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nd65z\" (UniqueName: \"kubernetes.io/projected/ee192e78-c59a-49e1-8c75-e312994f86e7-kube-api-access-nd65z\") pod \"ee192e78-c59a-49e1-8c75-e312994f86e7\" (UID: \"ee192e78-c59a-49e1-8c75-e312994f86e7\") " Jul 12 19:48:54.993626 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:48:54.993465 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee192e78-c59a-49e1-8c75-e312994f86e7-workload-certs\") pod \"ee192e78-c59a-49e1-8c75-e312994f86e7\" (UID: \"ee192e78-c59a-49e1-8c75-e312994f86e7\") " Jul 12 19:48:54.993626 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:48:54.993508 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee192e78-c59a-49e1-8c75-e312994f86e7-istio-data\") pod \"ee192e78-c59a-49e1-8c75-e312994f86e7\" (UID: \"ee192e78-c59a-49e1-8c75-e312994f86e7\") " Jul 12 19:48:54.993761 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:48:54.993584 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee192e78-c59a-49e1-8c75-e312994f86e7-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "ee192e78-c59a-49e1-8c75-e312994f86e7" (UID: "ee192e78-c59a-49e1-8c75-e312994f86e7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 12 19:48:54.994163 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:48:54.994059 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee192e78-c59a-49e1-8c75-e312994f86e7-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "ee192e78-c59a-49e1-8c75-e312994f86e7" (UID: "ee192e78-c59a-49e1-8c75-e312994f86e7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 19:48:54.994297 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:48:54.994069 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee192e78-c59a-49e1-8c75-e312994f86e7-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "ee192e78-c59a-49e1-8c75-e312994f86e7" (UID: "ee192e78-c59a-49e1-8c75-e312994f86e7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 19:48:54.994378 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:48:54.994258 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee192e78-c59a-49e1-8c75-e312994f86e7-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "ee192e78-c59a-49e1-8c75-e312994f86e7" (UID: "ee192e78-c59a-49e1-8c75-e312994f86e7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 19:48:54.995071 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:48:54.995049 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee192e78-c59a-49e1-8c75-e312994f86e7-istio-data" (OuterVolumeSpecName: "istio-data") pod "ee192e78-c59a-49e1-8c75-e312994f86e7" (UID: "ee192e78-c59a-49e1-8c75-e312994f86e7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 19:48:54.996647 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:48:54.996619 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee192e78-c59a-49e1-8c75-e312994f86e7-istio-token" (OuterVolumeSpecName: "istio-token") pod "ee192e78-c59a-49e1-8c75-e312994f86e7" (UID: "ee192e78-c59a-49e1-8c75-e312994f86e7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 12 19:48:54.996743 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:48:54.996649 2565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee192e78-c59a-49e1-8c75-e312994f86e7-credential-socket\") on node \"ip-10-0-139-170.ec2.internal\" DevicePath \"\"" Jul 12 19:48:54.996743 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:48:54.996661 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee192e78-c59a-49e1-8c75-e312994f86e7-kube-api-access-nd65z" (OuterVolumeSpecName: "kube-api-access-nd65z") pod "ee192e78-c59a-49e1-8c75-e312994f86e7" (UID: "ee192e78-c59a-49e1-8c75-e312994f86e7"). InnerVolumeSpecName "kube-api-access-nd65z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 12 19:48:54.996743 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:48:54.996679 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ee192e78-c59a-49e1-8c75-e312994f86e7-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "ee192e78-c59a-49e1-8c75-e312994f86e7" (UID: "ee192e78-c59a-49e1-8c75-e312994f86e7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 12 19:48:54.996743 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:48:54.996668 2565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee192e78-c59a-49e1-8c75-e312994f86e7-workload-socket\") on node \"ip-10-0-139-170.ec2.internal\" DevicePath \"\"" Jul 12 19:48:54.996743 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:48:54.996711 2565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee192e78-c59a-49e1-8c75-e312994f86e7-workload-certs\") on node \"ip-10-0-139-170.ec2.internal\" DevicePath \"\"" Jul 12 19:48:54.996743 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:48:54.996727 2565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee192e78-c59a-49e1-8c75-e312994f86e7-istio-data\") on node \"ip-10-0-139-170.ec2.internal\" DevicePath \"\"" Jul 12 19:48:54.996988 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:48:54.996747 2565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee192e78-c59a-49e1-8c75-e312994f86e7-istiod-ca-cert\") on node \"ip-10-0-139-170.ec2.internal\" DevicePath \"\"" Jul 12 19:48:54.996988 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:48:54.996944 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee192e78-c59a-49e1-8c75-e312994f86e7-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "ee192e78-c59a-49e1-8c75-e312994f86e7" (UID: "ee192e78-c59a-49e1-8c75-e312994f86e7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 19:48:55.098032 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:48:55.097993 2565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee192e78-c59a-49e1-8c75-e312994f86e7-istio-podinfo\") on node \"ip-10-0-139-170.ec2.internal\" DevicePath \"\"" Jul 12 19:48:55.098032 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:48:55.098027 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nd65z\" (UniqueName: \"kubernetes.io/projected/ee192e78-c59a-49e1-8c75-e312994f86e7-kube-api-access-nd65z\") on node \"ip-10-0-139-170.ec2.internal\" DevicePath \"\"" Jul 12 19:48:55.098032 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:48:55.098037 2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee192e78-c59a-49e1-8c75-e312994f86e7-istio-token\") on node \"ip-10-0-139-170.ec2.internal\" DevicePath \"\"" Jul 12 19:48:55.098250 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:48:55.098045 2565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee192e78-c59a-49e1-8c75-e312994f86e7-istio-envoy\") on node \"ip-10-0-139-170.ec2.internal\" DevicePath \"\"" Jul 12 19:48:55.665907 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:48:55.665860 2565 generic.go:356] "Generic (PLEG): container finished" podID="ee192e78-c59a-49e1-8c75-e312994f86e7" containerID="a789befff65e363c684831b63bff28872513fef248b434dd5c878c9d73800be6" exitCode=0 Jul 12 19:48:55.666286 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:48:55.665947 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a4a89c5e-openshift-default-586d7bbd5f-gw9vc" Jul 12 19:48:55.666286 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:48:55.665953 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a4a89c5e-openshift-default-586d7bbd5f-gw9vc" event={"ID":"ee192e78-c59a-49e1-8c75-e312994f86e7","Type":"ContainerDied","Data":"a789befff65e363c684831b63bff28872513fef248b434dd5c878c9d73800be6"} Jul 12 19:48:55.666286 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:48:55.666034 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a4a89c5e-openshift-default-586d7bbd5f-gw9vc" event={"ID":"ee192e78-c59a-49e1-8c75-e312994f86e7","Type":"ContainerDied","Data":"9134a09ea1b5c71de1fc071613fbb9e8738e75f5f4267d1b48e2e3f0a8fff723"} Jul 12 19:48:55.666286 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:48:55.666056 2565 scope.go:117] "RemoveContainer" containerID="a789befff65e363c684831b63bff28872513fef248b434dd5c878c9d73800be6" Jul 12 19:48:55.675482 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:48:55.675463 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:48:55.676113 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:48:55.676097 2565 scope.go:117] "RemoveContainer" containerID="a789befff65e363c684831b63bff28872513fef248b434dd5c878c9d73800be6" Jul 12 19:48:55.676345 ip-10-0-139-170 kubenswrapper[2565]: E0712 19:48:55.676327 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a789befff65e363c684831b63bff28872513fef248b434dd5c878c9d73800be6\": container with ID starting with a789befff65e363c684831b63bff28872513fef248b434dd5c878c9d73800be6 not found: ID does not exist" containerID="a789befff65e363c684831b63bff28872513fef248b434dd5c878c9d73800be6" Jul 12 19:48:55.676396 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:48:55.676352 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a789befff65e363c684831b63bff28872513fef248b434dd5c878c9d73800be6"} err="failed to get container status \"a789befff65e363c684831b63bff28872513fef248b434dd5c878c9d73800be6\": rpc error: code = NotFound desc = could not find container \"a789befff65e363c684831b63bff28872513fef248b434dd5c878c9d73800be6\": container with ID starting with a789befff65e363c684831b63bff28872513fef248b434dd5c878c9d73800be6 not found: ID does not exist" Jul 12 19:48:55.694470 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:48:55.694444 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-a4a89c5e-openshift-default-586d7bbd5f-gw9vc"] Jul 12 19:48:55.696733 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:48:55.696712 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-a4a89c5e-openshift-default-586d7bbd5f-gw9vc"] Jul 12 19:48:55.770507 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:48:55.770470 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:48:55.827357 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:48:55.827327 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:48:55.857386 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:48:55.857356 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ee192e78-c59a-49e1-8c75-e312994f86e7" path="/var/lib/kubelet/pods/ee192e78-c59a-49e1-8c75-e312994f86e7/volumes" Jul 12 19:48:56.137781 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:48:56.137692 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:48:56.238045 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:48:56.238010 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:48:56.338098 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:48:56.338065 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:48:57.745362 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:48:57.745332 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:49:00.743118 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:49:00.743087 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:49:01.028467 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:49:01.028375 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:49:07.035019 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:49:07.034982 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:49:07.134510 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:49:07.134471 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:49:08.717985 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:49:08.717951 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:49:12.839274 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:49:12.839236 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:49:12.936768 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:49:12.936737 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:49:13.001897 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:49:13.001844 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-771d3436-openshift-default-59bbb9f575-cv6fz"] Jul 12 19:49:13.002332 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:49:13.002308 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee192e78-c59a-49e1-8c75-e312994f86e7" containerName="istio-proxy" Jul 12 19:49:13.002332 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:49:13.002328 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee192e78-c59a-49e1-8c75-e312994f86e7" containerName="istio-proxy" Jul 12 19:49:13.002464 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:49:13.002403 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="ee192e78-c59a-49e1-8c75-e312994f86e7" containerName="istio-proxy" Jul 12 19:49:13.006892 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:49:13.006854 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-771d3436-openshift-default-59bbb9f575-cv6fz" Jul 12 19:49:13.009151 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:49:13.009132 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-771d3436-openshift-default-dockercfg-8td8n\"" Jul 12 19:49:13.016560 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:49:13.016538 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-771d3436-openshift-default-59bbb9f575-cv6fz"] Jul 12 19:49:13.037702 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:49:13.037677 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:49:13.054105 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:49:13.054067 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f23df018-de81-45de-b833-8995c16337d4-workload-certs\") pod \"e2e-derive-771d3436-openshift-default-59bbb9f575-cv6fz\" (UID: \"f23df018-de81-45de-b833-8995c16337d4\") " pod="openshift-ingress/e2e-derive-771d3436-openshift-default-59bbb9f575-cv6fz" Jul 12 19:49:13.054255 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:49:13.054109 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f23df018-de81-45de-b833-8995c16337d4-credential-socket\") pod \"e2e-derive-771d3436-openshift-default-59bbb9f575-cv6fz\" (UID: \"f23df018-de81-45de-b833-8995c16337d4\") " pod="openshift-ingress/e2e-derive-771d3436-openshift-default-59bbb9f575-cv6fz" Jul 12 19:49:13.054255 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:49:13.054133 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m29wl\" (UniqueName: \"kubernetes.io/projected/f23df018-de81-45de-b833-8995c16337d4-kube-api-access-m29wl\") pod \"e2e-derive-771d3436-openshift-default-59bbb9f575-cv6fz\" (UID: \"f23df018-de81-45de-b833-8995c16337d4\") " pod="openshift-ingress/e2e-derive-771d3436-openshift-default-59bbb9f575-cv6fz" Jul 12 19:49:13.054255 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:49:13.054195 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f23df018-de81-45de-b833-8995c16337d4-istio-podinfo\") pod \"e2e-derive-771d3436-openshift-default-59bbb9f575-cv6fz\" (UID: \"f23df018-de81-45de-b833-8995c16337d4\") " pod="openshift-ingress/e2e-derive-771d3436-openshift-default-59bbb9f575-cv6fz" Jul 12 19:49:13.054436 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:49:13.054257 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f23df018-de81-45de-b833-8995c16337d4-istiod-ca-cert\") pod \"e2e-derive-771d3436-openshift-default-59bbb9f575-cv6fz\" (UID: \"f23df018-de81-45de-b833-8995c16337d4\") " pod="openshift-ingress/e2e-derive-771d3436-openshift-default-59bbb9f575-cv6fz" Jul 12 19:49:13.054436 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:49:13.054290 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f23df018-de81-45de-b833-8995c16337d4-workload-socket\") pod \"e2e-derive-771d3436-openshift-default-59bbb9f575-cv6fz\" (UID: \"f23df018-de81-45de-b833-8995c16337d4\") " pod="openshift-ingress/e2e-derive-771d3436-openshift-default-59bbb9f575-cv6fz" Jul 12 19:49:13.054436 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:49:13.054307 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f23df018-de81-45de-b833-8995c16337d4-istio-envoy\") pod \"e2e-derive-771d3436-openshift-default-59bbb9f575-cv6fz\" (UID: \"f23df018-de81-45de-b833-8995c16337d4\") " pod="openshift-ingress/e2e-derive-771d3436-openshift-default-59bbb9f575-cv6fz" Jul 12 19:49:13.054436 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:49:13.054327 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f23df018-de81-45de-b833-8995c16337d4-istio-data\") pod \"e2e-derive-771d3436-openshift-default-59bbb9f575-cv6fz\" (UID: \"f23df018-de81-45de-b833-8995c16337d4\") " pod="openshift-ingress/e2e-derive-771d3436-openshift-default-59bbb9f575-cv6fz" Jul 12 19:49:13.054436 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:49:13.054374 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f23df018-de81-45de-b833-8995c16337d4-istio-token\") pod \"e2e-derive-771d3436-openshift-default-59bbb9f575-cv6fz\" (UID: \"f23df018-de81-45de-b833-8995c16337d4\") " pod="openshift-ingress/e2e-derive-771d3436-openshift-default-59bbb9f575-cv6fz" Jul 12 19:49:13.092313 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:49:13.092221 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:49:13.155031 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:49:13.154992 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f23df018-de81-45de-b833-8995c16337d4-istio-podinfo\") pod \"e2e-derive-771d3436-openshift-default-59bbb9f575-cv6fz\" (UID: \"f23df018-de81-45de-b833-8995c16337d4\") " pod="openshift-ingress/e2e-derive-771d3436-openshift-default-59bbb9f575-cv6fz" Jul 12 19:49:13.155031 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:49:13.155033 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f23df018-de81-45de-b833-8995c16337d4-istiod-ca-cert\") pod \"e2e-derive-771d3436-openshift-default-59bbb9f575-cv6fz\" (UID: \"f23df018-de81-45de-b833-8995c16337d4\") " pod="openshift-ingress/e2e-derive-771d3436-openshift-default-59bbb9f575-cv6fz" Jul 12 19:49:13.155299 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:49:13.155064 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f23df018-de81-45de-b833-8995c16337d4-workload-socket\") pod \"e2e-derive-771d3436-openshift-default-59bbb9f575-cv6fz\" (UID: \"f23df018-de81-45de-b833-8995c16337d4\") " pod="openshift-ingress/e2e-derive-771d3436-openshift-default-59bbb9f575-cv6fz" Jul 12 19:49:13.155299 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:49:13.155088 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f23df018-de81-45de-b833-8995c16337d4-istio-envoy\") pod \"e2e-derive-771d3436-openshift-default-59bbb9f575-cv6fz\" (UID: \"f23df018-de81-45de-b833-8995c16337d4\") " pod="openshift-ingress/e2e-derive-771d3436-openshift-default-59bbb9f575-cv6fz" Jul 12 19:49:13.155299 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:49:13.155108 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f23df018-de81-45de-b833-8995c16337d4-istio-data\") pod \"e2e-derive-771d3436-openshift-default-59bbb9f575-cv6fz\" (UID: \"f23df018-de81-45de-b833-8995c16337d4\") " pod="openshift-ingress/e2e-derive-771d3436-openshift-default-59bbb9f575-cv6fz" Jul 12 19:49:13.155299 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:49:13.155145 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f23df018-de81-45de-b833-8995c16337d4-istio-token\") pod \"e2e-derive-771d3436-openshift-default-59bbb9f575-cv6fz\" (UID: \"f23df018-de81-45de-b833-8995c16337d4\") " pod="openshift-ingress/e2e-derive-771d3436-openshift-default-59bbb9f575-cv6fz" Jul 12 19:49:13.155299 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:49:13.155214 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f23df018-de81-45de-b833-8995c16337d4-workload-certs\") pod \"e2e-derive-771d3436-openshift-default-59bbb9f575-cv6fz\" (UID: \"f23df018-de81-45de-b833-8995c16337d4\") " pod="openshift-ingress/e2e-derive-771d3436-openshift-default-59bbb9f575-cv6fz" Jul 12 19:49:13.155299 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:49:13.155254 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f23df018-de81-45de-b833-8995c16337d4-credential-socket\") pod \"e2e-derive-771d3436-openshift-default-59bbb9f575-cv6fz\" (UID: \"f23df018-de81-45de-b833-8995c16337d4\") " pod="openshift-ingress/e2e-derive-771d3436-openshift-default-59bbb9f575-cv6fz" Jul 12 19:49:13.155581 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:49:13.155560 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f23df018-de81-45de-b833-8995c16337d4-workload-socket\") pod \"e2e-derive-771d3436-openshift-default-59bbb9f575-cv6fz\" (UID: \"f23df018-de81-45de-b833-8995c16337d4\") " pod="openshift-ingress/e2e-derive-771d3436-openshift-default-59bbb9f575-cv6fz" Jul 12 19:49:13.155653 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:49:13.155626 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f23df018-de81-45de-b833-8995c16337d4-credential-socket\") pod \"e2e-derive-771d3436-openshift-default-59bbb9f575-cv6fz\" (UID: \"f23df018-de81-45de-b833-8995c16337d4\") " pod="openshift-ingress/e2e-derive-771d3436-openshift-default-59bbb9f575-cv6fz" Jul 12 19:49:13.155777 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:49:13.155645 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m29wl\" (UniqueName: \"kubernetes.io/projected/f23df018-de81-45de-b833-8995c16337d4-kube-api-access-m29wl\") pod \"e2e-derive-771d3436-openshift-default-59bbb9f575-cv6fz\" (UID: \"f23df018-de81-45de-b833-8995c16337d4\") " pod="openshift-ingress/e2e-derive-771d3436-openshift-default-59bbb9f575-cv6fz" Jul 12 19:49:13.155777 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:49:13.155703 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f23df018-de81-45de-b833-8995c16337d4-workload-certs\") pod \"e2e-derive-771d3436-openshift-default-59bbb9f575-cv6fz\" (UID: \"f23df018-de81-45de-b833-8995c16337d4\") " pod="openshift-ingress/e2e-derive-771d3436-openshift-default-59bbb9f575-cv6fz" Jul 12 19:49:13.155777 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:49:13.155735 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f23df018-de81-45de-b833-8995c16337d4-istio-data\") pod \"e2e-derive-771d3436-openshift-default-59bbb9f575-cv6fz\" (UID: \"f23df018-de81-45de-b833-8995c16337d4\") " pod="openshift-ingress/e2e-derive-771d3436-openshift-default-59bbb9f575-cv6fz" Jul 12 19:49:13.155966 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:49:13.155918 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f23df018-de81-45de-b833-8995c16337d4-istiod-ca-cert\") pod \"e2e-derive-771d3436-openshift-default-59bbb9f575-cv6fz\" (UID: \"f23df018-de81-45de-b833-8995c16337d4\") " pod="openshift-ingress/e2e-derive-771d3436-openshift-default-59bbb9f575-cv6fz" Jul 12 19:49:13.157532 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:49:13.157509 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f23df018-de81-45de-b833-8995c16337d4-istio-envoy\") pod \"e2e-derive-771d3436-openshift-default-59bbb9f575-cv6fz\" (UID: \"f23df018-de81-45de-b833-8995c16337d4\") " pod="openshift-ingress/e2e-derive-771d3436-openshift-default-59bbb9f575-cv6fz" Jul 12 19:49:13.157800 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:49:13.157782 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f23df018-de81-45de-b833-8995c16337d4-istio-podinfo\") pod \"e2e-derive-771d3436-openshift-default-59bbb9f575-cv6fz\" (UID: \"f23df018-de81-45de-b833-8995c16337d4\") " pod="openshift-ingress/e2e-derive-771d3436-openshift-default-59bbb9f575-cv6fz" Jul 12 19:49:13.162749 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:49:13.162725 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f23df018-de81-45de-b833-8995c16337d4-istio-token\") pod \"e2e-derive-771d3436-openshift-default-59bbb9f575-cv6fz\" (UID: \"f23df018-de81-45de-b833-8995c16337d4\") " pod="openshift-ingress/e2e-derive-771d3436-openshift-default-59bbb9f575-cv6fz" Jul 12 19:49:13.162965 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:49:13.162947 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m29wl\" (UniqueName: \"kubernetes.io/projected/f23df018-de81-45de-b833-8995c16337d4-kube-api-access-m29wl\") pod \"e2e-derive-771d3436-openshift-default-59bbb9f575-cv6fz\" (UID: \"f23df018-de81-45de-b833-8995c16337d4\") " pod="openshift-ingress/e2e-derive-771d3436-openshift-default-59bbb9f575-cv6fz" Jul 12 19:49:13.318777 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:49:13.318738 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-771d3436-openshift-default-59bbb9f575-cv6fz" Jul 12 19:49:13.450802 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:49:13.450776 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-771d3436-openshift-default-59bbb9f575-cv6fz"] Jul 12 19:49:13.453286 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:49:13.453250 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf23df018_de81_45de_b833_8995c16337d4.slice/crio-5bccc505c718cc5b3a6aac96eb01e0eff6ccad97101e5ec620cb96f6b7798e87 WatchSource:0}: Error finding container 5bccc505c718cc5b3a6aac96eb01e0eff6ccad97101e5ec620cb96f6b7798e87: Status 404 returned error can't find the container with id 5bccc505c718cc5b3a6aac96eb01e0eff6ccad97101e5ec620cb96f6b7798e87 Jul 12 19:49:13.455396 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:49:13.455368 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 12 19:49:13.455458 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:49:13.455447 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 12 19:49:13.455499 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:49:13.455476 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 12 19:49:13.741235 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:49:13.741195 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-771d3436-openshift-default-59bbb9f575-cv6fz" event={"ID":"f23df018-de81-45de-b833-8995c16337d4","Type":"ContainerStarted","Data":"ccf55f548f286113bbbb8a5b8f17cb3d360ccb2a1d9e971cd6122d70ae6adc7b"} Jul 12 19:49:13.741433 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:49:13.741254 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-771d3436-openshift-default-59bbb9f575-cv6fz" event={"ID":"f23df018-de81-45de-b833-8995c16337d4","Type":"ContainerStarted","Data":"5bccc505c718cc5b3a6aac96eb01e0eff6ccad97101e5ec620cb96f6b7798e87"} Jul 12 19:49:13.759215 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:49:13.759159 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-771d3436-openshift-default-59bbb9f575-cv6fz" podStartSLOduration=1.7591459710000001 podStartE2EDuration="1.759145971s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 19:49:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 19:49:13.757872812 +0000 UTC m=+1618.490445301" watchObservedRunningTime="2026-07-12 19:49:13.759145971 +0000 UTC m=+1618.491718459" Jul 12 19:49:14.250193 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:49:14.250151 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:49:14.319425 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:49:14.319394 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-771d3436-openshift-default-59bbb9f575-cv6fz" Jul 12 19:49:14.324276 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:49:14.324250 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-771d3436-openshift-default-59bbb9f575-cv6fz" Jul 12 19:49:14.391593 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:49:14.391557 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:49:14.493039 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:49:14.493006 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:49:14.594604 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:49:14.594518 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:49:14.745779 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:49:14.745747 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-771d3436-openshift-default-59bbb9f575-cv6fz" Jul 12 19:49:14.746597 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:49:14.746578 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-771d3436-openshift-default-59bbb9f575-cv6fz" Jul 12 19:49:14.985338 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:49:14.985303 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:49:15.086531 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:49:15.086495 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:49:16.603270 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:49:16.603234 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:49:18.649736 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:49:18.649702 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:49:18.806438 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:49:18.806405 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-771d3436-openshift-default-59bbb9f575-cv6fz"] Jul 12 19:49:18.806619 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:49:18.806575 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-771d3436-openshift-default-59bbb9f575-cv6fz" podUID="f23df018-de81-45de-b833-8995c16337d4" containerName="istio-proxy" containerID="cri-o://ccf55f548f286113bbbb8a5b8f17cb3d360ccb2a1d9e971cd6122d70ae6adc7b" gracePeriod=30 Jul 12 19:49:18.833595 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:49:18.833561 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:49:18.930749 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:49:18.930676 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:49:19.030615 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:49:19.030579 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:49:20.483598 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:49:20.483563 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:49:24.052062 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:49:24.052034 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-771d3436-openshift-default-59bbb9f575-cv6fz" Jul 12 19:49:24.156109 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:49:24.156084 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f23df018-de81-45de-b833-8995c16337d4-istio-envoy\") pod \"f23df018-de81-45de-b833-8995c16337d4\" (UID: \"f23df018-de81-45de-b833-8995c16337d4\") " Jul 12 19:49:24.156260 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:49:24.156144 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m29wl\" (UniqueName: \"kubernetes.io/projected/f23df018-de81-45de-b833-8995c16337d4-kube-api-access-m29wl\") pod \"f23df018-de81-45de-b833-8995c16337d4\" (UID: \"f23df018-de81-45de-b833-8995c16337d4\") " Jul 12 19:49:24.156260 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:49:24.156181 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f23df018-de81-45de-b833-8995c16337d4-istio-podinfo\") pod \"f23df018-de81-45de-b833-8995c16337d4\" (UID: \"f23df018-de81-45de-b833-8995c16337d4\") " Jul 12 19:49:24.156260 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:49:24.156237 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f23df018-de81-45de-b833-8995c16337d4-istio-data\") pod \"f23df018-de81-45de-b833-8995c16337d4\" (UID: \"f23df018-de81-45de-b833-8995c16337d4\") " Jul 12 19:49:24.156402 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:49:24.156265 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f23df018-de81-45de-b833-8995c16337d4-workload-certs\") pod \"f23df018-de81-45de-b833-8995c16337d4\" (UID: \"f23df018-de81-45de-b833-8995c16337d4\") " Jul 12 19:49:24.156402 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:49:24.156328 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f23df018-de81-45de-b833-8995c16337d4-workload-socket\") pod \"f23df018-de81-45de-b833-8995c16337d4\" (UID: \"f23df018-de81-45de-b833-8995c16337d4\") " Jul 12 19:49:24.156402 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:49:24.156362 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f23df018-de81-45de-b833-8995c16337d4-istio-token\") pod \"f23df018-de81-45de-b833-8995c16337d4\" (UID: \"f23df018-de81-45de-b833-8995c16337d4\") " Jul 12 19:49:24.156402 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:49:24.156392 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f23df018-de81-45de-b833-8995c16337d4-istiod-ca-cert\") pod \"f23df018-de81-45de-b833-8995c16337d4\" (UID: \"f23df018-de81-45de-b833-8995c16337d4\") " Jul 12 19:49:24.156592 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:49:24.156432 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f23df018-de81-45de-b833-8995c16337d4-credential-socket\") pod \"f23df018-de81-45de-b833-8995c16337d4\" (UID: \"f23df018-de81-45de-b833-8995c16337d4\") " Jul 12 19:49:24.156592 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:49:24.156520 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f23df018-de81-45de-b833-8995c16337d4-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "f23df018-de81-45de-b833-8995c16337d4" (UID: "f23df018-de81-45de-b833-8995c16337d4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 19:49:24.156880 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:49:24.156529 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f23df018-de81-45de-b833-8995c16337d4-istio-data" (OuterVolumeSpecName: "istio-data") pod "f23df018-de81-45de-b833-8995c16337d4" (UID: "f23df018-de81-45de-b833-8995c16337d4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 19:49:24.156880 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:49:24.156701 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f23df018-de81-45de-b833-8995c16337d4-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "f23df018-de81-45de-b833-8995c16337d4" (UID: "f23df018-de81-45de-b833-8995c16337d4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 19:49:24.156880 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:49:24.156782 2565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f23df018-de81-45de-b833-8995c16337d4-istio-data\") on node \"ip-10-0-139-170.ec2.internal\" DevicePath \"\"" Jul 12 19:49:24.156880 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:49:24.156802 2565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f23df018-de81-45de-b833-8995c16337d4-workload-certs\") on node \"ip-10-0-139-170.ec2.internal\" DevicePath \"\"" Jul 12 19:49:24.156880 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:49:24.156815 2565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f23df018-de81-45de-b833-8995c16337d4-workload-socket\") on node \"ip-10-0-139-170.ec2.internal\" DevicePath \"\"" Jul 12 19:49:24.156880 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:49:24.156836 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f23df018-de81-45de-b833-8995c16337d4-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "f23df018-de81-45de-b833-8995c16337d4" (UID: "f23df018-de81-45de-b833-8995c16337d4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 19:49:24.157218 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:49:24.156898 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f23df018-de81-45de-b833-8995c16337d4-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "f23df018-de81-45de-b833-8995c16337d4" (UID: "f23df018-de81-45de-b833-8995c16337d4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 12 19:49:24.158372 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:49:24.158339 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f23df018-de81-45de-b833-8995c16337d4-kube-api-access-m29wl" (OuterVolumeSpecName: "kube-api-access-m29wl") pod "f23df018-de81-45de-b833-8995c16337d4" (UID: "f23df018-de81-45de-b833-8995c16337d4"). InnerVolumeSpecName "kube-api-access-m29wl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 12 19:49:24.158372 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:49:24.158348 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f23df018-de81-45de-b833-8995c16337d4-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "f23df018-de81-45de-b833-8995c16337d4" (UID: "f23df018-de81-45de-b833-8995c16337d4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 19:49:24.158538 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:49:24.158408 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f23df018-de81-45de-b833-8995c16337d4-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "f23df018-de81-45de-b833-8995c16337d4" (UID: "f23df018-de81-45de-b833-8995c16337d4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 12 19:49:24.158589 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:49:24.158549 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f23df018-de81-45de-b833-8995c16337d4-istio-token" (OuterVolumeSpecName: "istio-token") pod "f23df018-de81-45de-b833-8995c16337d4" (UID: "f23df018-de81-45de-b833-8995c16337d4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 12 19:49:24.257795 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:49:24.257756 2565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f23df018-de81-45de-b833-8995c16337d4-credential-socket\") on node \"ip-10-0-139-170.ec2.internal\" DevicePath \"\"" Jul 12 19:49:24.257795 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:49:24.257792 2565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f23df018-de81-45de-b833-8995c16337d4-istio-envoy\") on node \"ip-10-0-139-170.ec2.internal\" DevicePath \"\"" Jul 12 19:49:24.258031 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:49:24.257806 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m29wl\" (UniqueName: \"kubernetes.io/projected/f23df018-de81-45de-b833-8995c16337d4-kube-api-access-m29wl\") on node \"ip-10-0-139-170.ec2.internal\" DevicePath \"\"" Jul 12 19:49:24.258031 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:49:24.257816 2565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f23df018-de81-45de-b833-8995c16337d4-istio-podinfo\") on node \"ip-10-0-139-170.ec2.internal\" DevicePath \"\"" Jul 12 19:49:24.258031 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:49:24.257825 2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f23df018-de81-45de-b833-8995c16337d4-istio-token\") on node \"ip-10-0-139-170.ec2.internal\" DevicePath \"\"" Jul 12 19:49:24.258031 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:49:24.257833 2565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f23df018-de81-45de-b833-8995c16337d4-istiod-ca-cert\") on node \"ip-10-0-139-170.ec2.internal\" DevicePath \"\"" Jul 12 19:49:24.523871 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:49:24.523774 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:49:24.580677 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:49:24.580645 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:49:24.796151 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:49:24.796058 2565 generic.go:356] "Generic (PLEG): container finished" podID="f23df018-de81-45de-b833-8995c16337d4" containerID="ccf55f548f286113bbbb8a5b8f17cb3d360ccb2a1d9e971cd6122d70ae6adc7b" exitCode=0 Jul 12 19:49:24.796151 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:49:24.796111 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-771d3436-openshift-default-59bbb9f575-cv6fz" Jul 12 19:49:24.796151 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:49:24.796140 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-771d3436-openshift-default-59bbb9f575-cv6fz" event={"ID":"f23df018-de81-45de-b833-8995c16337d4","Type":"ContainerDied","Data":"ccf55f548f286113bbbb8a5b8f17cb3d360ccb2a1d9e971cd6122d70ae6adc7b"} Jul 12 19:49:24.796422 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:49:24.796218 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-771d3436-openshift-default-59bbb9f575-cv6fz" event={"ID":"f23df018-de81-45de-b833-8995c16337d4","Type":"ContainerDied","Data":"5bccc505c718cc5b3a6aac96eb01e0eff6ccad97101e5ec620cb96f6b7798e87"} Jul 12 19:49:24.796422 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:49:24.796247 2565 scope.go:117] "RemoveContainer" containerID="ccf55f548f286113bbbb8a5b8f17cb3d360ccb2a1d9e971cd6122d70ae6adc7b" Jul 12 19:49:24.804794 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:49:24.804775 2565 scope.go:117] "RemoveContainer" containerID="ccf55f548f286113bbbb8a5b8f17cb3d360ccb2a1d9e971cd6122d70ae6adc7b" Jul 12 19:49:24.805053 ip-10-0-139-170 kubenswrapper[2565]: E0712 19:49:24.805034 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ccf55f548f286113bbbb8a5b8f17cb3d360ccb2a1d9e971cd6122d70ae6adc7b\": container with ID starting with ccf55f548f286113bbbb8a5b8f17cb3d360ccb2a1d9e971cd6122d70ae6adc7b not found: ID does not exist" containerID="ccf55f548f286113bbbb8a5b8f17cb3d360ccb2a1d9e971cd6122d70ae6adc7b" Jul 12 19:49:24.805119 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:49:24.805064 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ccf55f548f286113bbbb8a5b8f17cb3d360ccb2a1d9e971cd6122d70ae6adc7b"} err="failed to get container status \"ccf55f548f286113bbbb8a5b8f17cb3d360ccb2a1d9e971cd6122d70ae6adc7b\": rpc error: code = NotFound desc = could not find container \"ccf55f548f286113bbbb8a5b8f17cb3d360ccb2a1d9e971cd6122d70ae6adc7b\": container with ID starting with ccf55f548f286113bbbb8a5b8f17cb3d360ccb2a1d9e971cd6122d70ae6adc7b not found: ID does not exist" Jul 12 19:49:24.816172 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:49:24.816150 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-771d3436-openshift-default-59bbb9f575-cv6fz"] Jul 12 19:49:24.820027 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:49:24.820006 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-771d3436-openshift-default-59bbb9f575-cv6fz"] Jul 12 19:49:25.020133 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:49:25.020102 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:49:25.145112 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:49:25.145072 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:49:25.857619 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:49:25.857585 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f23df018-de81-45de-b833-8995c16337d4" path="/var/lib/kubelet/pods/f23df018-de81-45de-b833-8995c16337d4/volumes" Jul 12 19:49:26.492207 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:49:26.492176 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:49:32.080794 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:49:32.080747 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:49:32.142573 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:49:32.142539 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:49:32.219896 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:49:32.219855 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:49:32.319104 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:49:32.319071 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:49:32.419495 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:49:32.419463 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:49:32.819519 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:49:32.819443 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:49:46.653080 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:49:46.653047 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:49:46.700561 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:49:46.700532 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:49:58.654555 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:49:58.654517 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:50:04.283317 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:50:04.283280 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:50:04.405671 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:50:04.405640 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:50:04.517868 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:50:04.517838 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:50:20.328330 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:50:20.328296 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:50:25.871859 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:50:25.871808 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:50:26.307403 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:50:26.307337 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:50:26.378179 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:50:26.378145 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:50:26.462687 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:50:26.462642 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:50:26.561209 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:50:26.561122 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:50:37.030199 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:50:37.030165 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:50:37.790548 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:50:37.790503 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:50:43.356063 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:50:43.356029 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:50:43.415309 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:50:43.415272 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:50:43.490710 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:50:43.490669 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:50:43.589452 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:50:43.589423 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:50:43.690074 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:50:43.690045 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:50:44.765070 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:50:44.765041 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:50:50.491631 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:50:50.491594 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:50:50.551856 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:50:50.551821 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:50:50.611704 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:50:50.611674 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:50:50.714850 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:50:50.714818 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:50:50.813626 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:50:50.813539 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:50:51.491778 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:50:51.491730 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:50:51.591781 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:50:51.591746 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:50:51.692020 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:50:51.691959 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:50:53.226434 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:50:53.226394 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:50:59.198480 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:50:59.198422 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:50:59.296943 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:50:59.296913 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:50:59.843475 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:50:59.843437 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:51:05.569738 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:51:05.569695 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:51:05.669494 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:51:05.669461 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:51:05.795992 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:51:05.795954 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:51:06.069791 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:51:06.069753 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-41e17b7d-openshift-default-7cc77754f5-j9dlb"] Jul 12 19:51:06.070102 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:51:06.070088 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f23df018-de81-45de-b833-8995c16337d4" containerName="istio-proxy" Jul 12 19:51:06.070152 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:51:06.070104 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="f23df018-de81-45de-b833-8995c16337d4" containerName="istio-proxy" Jul 12 19:51:06.070200 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:51:06.070191 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="f23df018-de81-45de-b833-8995c16337d4" containerName="istio-proxy" Jul 12 19:51:06.073099 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:51:06.073082 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-41e17b7d-openshift-default-7cc77754f5-j9dlb" Jul 12 19:51:06.075352 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:51:06.075325 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-41e17b7d-openshift-default-dockercfg-rj6m6\"" Jul 12 19:51:06.093412 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:51:06.093385 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-41e17b7d-openshift-default-7cc77754f5-j9dlb"] Jul 12 19:51:06.095592 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:51:06.095566 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:51:06.146692 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:51:06.146666 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b626e47-45d3-4865-a8b6-759d71d2046f-istio-data\") pod \"e2e-mt-41e17b7d-openshift-default-7cc77754f5-j9dlb\" (UID: \"6b626e47-45d3-4865-a8b6-759d71d2046f\") " pod="openshift-ingress/e2e-mt-41e17b7d-openshift-default-7cc77754f5-j9dlb" Jul 12 19:51:06.146833 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:51:06.146697 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b626e47-45d3-4865-a8b6-759d71d2046f-istio-envoy\") pod \"e2e-mt-41e17b7d-openshift-default-7cc77754f5-j9dlb\" (UID: \"6b626e47-45d3-4865-a8b6-759d71d2046f\") " pod="openshift-ingress/e2e-mt-41e17b7d-openshift-default-7cc77754f5-j9dlb" Jul 12 19:51:06.146833 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:51:06.146721 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4gp6g\" (UniqueName: \"kubernetes.io/projected/6b626e47-45d3-4865-a8b6-759d71d2046f-kube-api-access-4gp6g\") pod \"e2e-mt-41e17b7d-openshift-default-7cc77754f5-j9dlb\" (UID: \"6b626e47-45d3-4865-a8b6-759d71d2046f\") " pod="openshift-ingress/e2e-mt-41e17b7d-openshift-default-7cc77754f5-j9dlb" Jul 12 19:51:06.146833 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:51:06.146787 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b626e47-45d3-4865-a8b6-759d71d2046f-workload-socket\") pod \"e2e-mt-41e17b7d-openshift-default-7cc77754f5-j9dlb\" (UID: \"6b626e47-45d3-4865-a8b6-759d71d2046f\") " pod="openshift-ingress/e2e-mt-41e17b7d-openshift-default-7cc77754f5-j9dlb" Jul 12 19:51:06.146833 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:51:06.146808 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b626e47-45d3-4865-a8b6-759d71d2046f-istiod-ca-cert\") pod \"e2e-mt-41e17b7d-openshift-default-7cc77754f5-j9dlb\" (UID: \"6b626e47-45d3-4865-a8b6-759d71d2046f\") " pod="openshift-ingress/e2e-mt-41e17b7d-openshift-default-7cc77754f5-j9dlb" Jul 12 19:51:06.146833 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:51:06.146833 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b626e47-45d3-4865-a8b6-759d71d2046f-workload-certs\") pod \"e2e-mt-41e17b7d-openshift-default-7cc77754f5-j9dlb\" (UID: \"6b626e47-45d3-4865-a8b6-759d71d2046f\") " pod="openshift-ingress/e2e-mt-41e17b7d-openshift-default-7cc77754f5-j9dlb" Jul 12 19:51:06.147075 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:51:06.146867 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b626e47-45d3-4865-a8b6-759d71d2046f-istio-podinfo\") pod \"e2e-mt-41e17b7d-openshift-default-7cc77754f5-j9dlb\" (UID: \"6b626e47-45d3-4865-a8b6-759d71d2046f\") " pod="openshift-ingress/e2e-mt-41e17b7d-openshift-default-7cc77754f5-j9dlb" Jul 12 19:51:06.147075 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:51:06.146926 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b626e47-45d3-4865-a8b6-759d71d2046f-credential-socket\") pod \"e2e-mt-41e17b7d-openshift-default-7cc77754f5-j9dlb\" (UID: \"6b626e47-45d3-4865-a8b6-759d71d2046f\") " pod="openshift-ingress/e2e-mt-41e17b7d-openshift-default-7cc77754f5-j9dlb" Jul 12 19:51:06.147075 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:51:06.146947 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b626e47-45d3-4865-a8b6-759d71d2046f-istio-token\") pod \"e2e-mt-41e17b7d-openshift-default-7cc77754f5-j9dlb\" (UID: \"6b626e47-45d3-4865-a8b6-759d71d2046f\") " pod="openshift-ingress/e2e-mt-41e17b7d-openshift-default-7cc77754f5-j9dlb" Jul 12 19:51:06.191999 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:51:06.191970 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:51:06.247632 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:51:06.247603 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b626e47-45d3-4865-a8b6-759d71d2046f-istio-podinfo\") pod \"e2e-mt-41e17b7d-openshift-default-7cc77754f5-j9dlb\" (UID: \"6b626e47-45d3-4865-a8b6-759d71d2046f\") " pod="openshift-ingress/e2e-mt-41e17b7d-openshift-default-7cc77754f5-j9dlb" Jul 12 19:51:06.247823 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:51:06.247647 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b626e47-45d3-4865-a8b6-759d71d2046f-credential-socket\") pod \"e2e-mt-41e17b7d-openshift-default-7cc77754f5-j9dlb\" (UID: \"6b626e47-45d3-4865-a8b6-759d71d2046f\") " pod="openshift-ingress/e2e-mt-41e17b7d-openshift-default-7cc77754f5-j9dlb" Jul 12 19:51:06.247823 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:51:06.247677 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b626e47-45d3-4865-a8b6-759d71d2046f-istio-token\") pod \"e2e-mt-41e17b7d-openshift-default-7cc77754f5-j9dlb\" (UID: \"6b626e47-45d3-4865-a8b6-759d71d2046f\") " pod="openshift-ingress/e2e-mt-41e17b7d-openshift-default-7cc77754f5-j9dlb" Jul 12 19:51:06.247823 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:51:06.247719 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b626e47-45d3-4865-a8b6-759d71d2046f-istio-data\") pod \"e2e-mt-41e17b7d-openshift-default-7cc77754f5-j9dlb\" (UID: \"6b626e47-45d3-4865-a8b6-759d71d2046f\") " pod="openshift-ingress/e2e-mt-41e17b7d-openshift-default-7cc77754f5-j9dlb" Jul 12 19:51:06.247823 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:51:06.247743 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b626e47-45d3-4865-a8b6-759d71d2046f-istio-envoy\") pod \"e2e-mt-41e17b7d-openshift-default-7cc77754f5-j9dlb\" (UID: \"6b626e47-45d3-4865-a8b6-759d71d2046f\") " pod="openshift-ingress/e2e-mt-41e17b7d-openshift-default-7cc77754f5-j9dlb" Jul 12 19:51:06.247823 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:51:06.247767 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4gp6g\" (UniqueName: \"kubernetes.io/projected/6b626e47-45d3-4865-a8b6-759d71d2046f-kube-api-access-4gp6g\") pod \"e2e-mt-41e17b7d-openshift-default-7cc77754f5-j9dlb\" (UID: \"6b626e47-45d3-4865-a8b6-759d71d2046f\") " pod="openshift-ingress/e2e-mt-41e17b7d-openshift-default-7cc77754f5-j9dlb" Jul 12 19:51:06.248114 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:51:06.247840 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b626e47-45d3-4865-a8b6-759d71d2046f-workload-socket\") pod \"e2e-mt-41e17b7d-openshift-default-7cc77754f5-j9dlb\" (UID: \"6b626e47-45d3-4865-a8b6-759d71d2046f\") " pod="openshift-ingress/e2e-mt-41e17b7d-openshift-default-7cc77754f5-j9dlb" Jul 12 19:51:06.248114 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:51:06.247863 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b626e47-45d3-4865-a8b6-759d71d2046f-istiod-ca-cert\") pod \"e2e-mt-41e17b7d-openshift-default-7cc77754f5-j9dlb\" (UID: \"6b626e47-45d3-4865-a8b6-759d71d2046f\") " pod="openshift-ingress/e2e-mt-41e17b7d-openshift-default-7cc77754f5-j9dlb" Jul 12 19:51:06.248114 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:51:06.247926 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b626e47-45d3-4865-a8b6-759d71d2046f-workload-certs\") pod \"e2e-mt-41e17b7d-openshift-default-7cc77754f5-j9dlb\" (UID: \"6b626e47-45d3-4865-a8b6-759d71d2046f\") " pod="openshift-ingress/e2e-mt-41e17b7d-openshift-default-7cc77754f5-j9dlb" Jul 12 19:51:06.248272 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:51:06.248152 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b626e47-45d3-4865-a8b6-759d71d2046f-istio-data\") pod \"e2e-mt-41e17b7d-openshift-default-7cc77754f5-j9dlb\" (UID: \"6b626e47-45d3-4865-a8b6-759d71d2046f\") " pod="openshift-ingress/e2e-mt-41e17b7d-openshift-default-7cc77754f5-j9dlb" Jul 12 19:51:06.248272 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:51:06.248247 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b626e47-45d3-4865-a8b6-759d71d2046f-workload-certs\") pod \"e2e-mt-41e17b7d-openshift-default-7cc77754f5-j9dlb\" (UID: \"6b626e47-45d3-4865-a8b6-759d71d2046f\") " pod="openshift-ingress/e2e-mt-41e17b7d-openshift-default-7cc77754f5-j9dlb" Jul 12 19:51:06.248387 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:51:06.248314 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b626e47-45d3-4865-a8b6-759d71d2046f-credential-socket\") pod \"e2e-mt-41e17b7d-openshift-default-7cc77754f5-j9dlb\" (UID: \"6b626e47-45d3-4865-a8b6-759d71d2046f\") " pod="openshift-ingress/e2e-mt-41e17b7d-openshift-default-7cc77754f5-j9dlb" Jul 12 19:51:06.248387 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:51:06.248336 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b626e47-45d3-4865-a8b6-759d71d2046f-workload-socket\") pod \"e2e-mt-41e17b7d-openshift-default-7cc77754f5-j9dlb\" (UID: \"6b626e47-45d3-4865-a8b6-759d71d2046f\") " pod="openshift-ingress/e2e-mt-41e17b7d-openshift-default-7cc77754f5-j9dlb" Jul 12 19:51:06.248632 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:51:06.248612 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b626e47-45d3-4865-a8b6-759d71d2046f-istiod-ca-cert\") pod \"e2e-mt-41e17b7d-openshift-default-7cc77754f5-j9dlb\" (UID: \"6b626e47-45d3-4865-a8b6-759d71d2046f\") " pod="openshift-ingress/e2e-mt-41e17b7d-openshift-default-7cc77754f5-j9dlb" Jul 12 19:51:06.250068 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:51:06.250043 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b626e47-45d3-4865-a8b6-759d71d2046f-istio-envoy\") pod \"e2e-mt-41e17b7d-openshift-default-7cc77754f5-j9dlb\" (UID: \"6b626e47-45d3-4865-a8b6-759d71d2046f\") " pod="openshift-ingress/e2e-mt-41e17b7d-openshift-default-7cc77754f5-j9dlb" Jul 12 19:51:06.250173 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:51:06.250160 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b626e47-45d3-4865-a8b6-759d71d2046f-istio-podinfo\") pod \"e2e-mt-41e17b7d-openshift-default-7cc77754f5-j9dlb\" (UID: \"6b626e47-45d3-4865-a8b6-759d71d2046f\") " pod="openshift-ingress/e2e-mt-41e17b7d-openshift-default-7cc77754f5-j9dlb" Jul 12 19:51:06.255605 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:51:06.255577 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b626e47-45d3-4865-a8b6-759d71d2046f-istio-token\") pod \"e2e-mt-41e17b7d-openshift-default-7cc77754f5-j9dlb\" (UID: \"6b626e47-45d3-4865-a8b6-759d71d2046f\") " pod="openshift-ingress/e2e-mt-41e17b7d-openshift-default-7cc77754f5-j9dlb" Jul 12 19:51:06.255712 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:51:06.255653 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4gp6g\" (UniqueName: \"kubernetes.io/projected/6b626e47-45d3-4865-a8b6-759d71d2046f-kube-api-access-4gp6g\") pod \"e2e-mt-41e17b7d-openshift-default-7cc77754f5-j9dlb\" (UID: \"6b626e47-45d3-4865-a8b6-759d71d2046f\") " pod="openshift-ingress/e2e-mt-41e17b7d-openshift-default-7cc77754f5-j9dlb" Jul 12 19:51:06.383032 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:51:06.382938 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-41e17b7d-openshift-default-7cc77754f5-j9dlb" Jul 12 19:51:06.512481 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:51:06.512451 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-41e17b7d-openshift-default-7cc77754f5-j9dlb"] Jul 12 19:51:06.515087 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:51:06.515059 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b626e47_45d3_4865_a8b6_759d71d2046f.slice/crio-b6de350f01680b0346aa1519f1d80f83770827d4a4518e92897f5f428a33d919 WatchSource:0}: Error finding container b6de350f01680b0346aa1519f1d80f83770827d4a4518e92897f5f428a33d919: Status 404 returned error can't find the container with id b6de350f01680b0346aa1519f1d80f83770827d4a4518e92897f5f428a33d919 Jul 12 19:51:06.517522 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:51:06.517488 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 12 19:51:06.517599 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:51:06.517573 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 12 19:51:06.517643 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:51:06.517600 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 12 19:51:07.216741 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:51:07.216706 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-41e17b7d-openshift-default-7cc77754f5-j9dlb" event={"ID":"6b626e47-45d3-4865-a8b6-759d71d2046f","Type":"ContainerStarted","Data":"abae1ba2dbe58590c3607320f7c119047f4771bea5b574d967440785934571fa"} Jul 12 19:51:07.217151 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:51:07.216756 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-41e17b7d-openshift-default-7cc77754f5-j9dlb" event={"ID":"6b626e47-45d3-4865-a8b6-759d71d2046f","Type":"ContainerStarted","Data":"b6de350f01680b0346aa1519f1d80f83770827d4a4518e92897f5f428a33d919"} Jul 12 19:51:07.236327 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:51:07.236278 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-41e17b7d-openshift-default-7cc77754f5-j9dlb" podStartSLOduration=1.236262308 podStartE2EDuration="1.236262308s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 19:51:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 19:51:07.234244477 +0000 UTC m=+1731.966816967" watchObservedRunningTime="2026-07-12 19:51:07.236262308 +0000 UTC m=+1731.968834797" Jul 12 19:51:07.383352 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:51:07.383311 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-41e17b7d-openshift-default-7cc77754f5-j9dlb" Jul 12 19:51:07.388633 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:51:07.388605 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-41e17b7d-openshift-default-7cc77754f5-j9dlb" Jul 12 19:51:08.221350 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:51:08.221318 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-41e17b7d-openshift-default-7cc77754f5-j9dlb" Jul 12 19:51:08.222398 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:51:08.222381 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-41e17b7d-openshift-default-7cc77754f5-j9dlb" Jul 12 19:51:09.229275 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:51:09.229241 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:51:09.297164 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:51:09.297119 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:51:09.359540 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:51:09.359503 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:51:09.498181 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:51:09.498097 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:51:09.799672 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:51:09.799586 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:51:09.900400 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:51:09.900368 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:51:13.731285 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:51:13.731248 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:51:14.393721 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:51:14.393685 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:51:14.491499 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:51:14.491459 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:51:14.819606 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:51:14.819516 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:51:20.519820 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:51:20.519781 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:51:20.577480 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:51:20.577441 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:51:20.904187 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:51:20.904151 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:51:21.218983 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:51:21.218900 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:51:21.316420 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:51:21.316386 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:51:23.223315 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:51:23.223279 2565 patch_prober.go:28] interesting pod/e2e-mt-41e17b7d-openshift-default-7cc77754f5-j9dlb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" start-of-body= Jul 12 19:51:23.223727 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:51:23.223343 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-41e17b7d-openshift-default-7cc77754f5-j9dlb" podUID="6b626e47-45d3-4865-a8b6-759d71d2046f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" Jul 12 19:51:23.295840 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:51:23.295806 2565 generic.go:356] "Generic (PLEG): container finished" podID="6b626e47-45d3-4865-a8b6-759d71d2046f" containerID="abae1ba2dbe58590c3607320f7c119047f4771bea5b574d967440785934571fa" exitCode=0 Jul 12 19:51:23.296034 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:51:23.295877 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-41e17b7d-openshift-default-7cc77754f5-j9dlb" event={"ID":"6b626e47-45d3-4865-a8b6-759d71d2046f","Type":"ContainerDied","Data":"abae1ba2dbe58590c3607320f7c119047f4771bea5b574d967440785934571fa"} Jul 12 19:51:23.296285 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:51:23.296272 2565 scope.go:117] "RemoveContainer" containerID="abae1ba2dbe58590c3607320f7c119047f4771bea5b574d967440785934571fa" Jul 12 19:51:23.297211 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:51:23.297185 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 12 19:51:23.297312 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:51:23.297246 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 12 19:51:23.297312 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:51:23.297276 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 12 19:51:24.301789 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:51:24.301756 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-41e17b7d-openshift-default-7cc77754f5-j9dlb" event={"ID":"6b626e47-45d3-4865-a8b6-759d71d2046f","Type":"ContainerStarted","Data":"d7fe9c4aaff4dbd7b836162ee0a4a7f2da491da929a977cefb907b1b65a4d94f"} Jul 12 19:51:24.383344 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:51:24.383313 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-41e17b7d-openshift-default-7cc77754f5-j9dlb" Jul 12 19:51:24.383575 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:51:24.383551 2565 patch_prober.go:28] interesting pod/e2e-mt-41e17b7d-openshift-default-7cc77754f5-j9dlb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" start-of-body= Jul 12 19:51:24.383642 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:51:24.383609 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-41e17b7d-openshift-default-7cc77754f5-j9dlb" podUID="6b626e47-45d3-4865-a8b6-759d71d2046f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" Jul 12 19:51:25.383720 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:51:25.383684 2565 patch_prober.go:28] interesting pod/e2e-mt-41e17b7d-openshift-default-7cc77754f5-j9dlb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" start-of-body= Jul 12 19:51:25.384171 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:51:25.383739 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-41e17b7d-openshift-default-7cc77754f5-j9dlb" podUID="6b626e47-45d3-4865-a8b6-759d71d2046f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" Jul 12 19:51:26.383927 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:51:26.383868 2565 patch_prober.go:28] interesting pod/e2e-mt-41e17b7d-openshift-default-7cc77754f5-j9dlb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" start-of-body= Jul 12 19:51:26.384304 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:51:26.383953 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-41e17b7d-openshift-default-7cc77754f5-j9dlb" podUID="6b626e47-45d3-4865-a8b6-759d71d2046f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" Jul 12 19:51:26.974801 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:51:26.974763 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:51:27.087762 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:51:27.084590 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:51:27.195445 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:51:27.195411 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-41e17b7d-openshift-default-7cc77754f5-j9dlb"] Jul 12 19:51:27.195632 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:51:27.195607 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-41e17b7d-openshift-default-7cc77754f5-j9dlb" podUID="6b626e47-45d3-4865-a8b6-759d71d2046f" containerName="istio-proxy" containerID="cri-o://d7fe9c4aaff4dbd7b836162ee0a4a7f2da491da929a977cefb907b1b65a4d94f" gracePeriod=30 Jul 12 19:51:27.214444 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:51:27.214414 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:51:27.471635 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:51:27.471595 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:51:27.578709 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:51:27.578676 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:51:27.670871 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:51:27.670838 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:51:27.771288 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:51:27.771197 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:51:33.142661 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:51:33.142640 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-41e17b7d-openshift-default-7cc77754f5-j9dlb" Jul 12 19:51:33.288092 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:51:33.287999 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b626e47-45d3-4865-a8b6-759d71d2046f-workload-socket\") pod \"6b626e47-45d3-4865-a8b6-759d71d2046f\" (UID: \"6b626e47-45d3-4865-a8b6-759d71d2046f\") " Jul 12 19:51:33.288092 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:51:33.288051 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b626e47-45d3-4865-a8b6-759d71d2046f-istio-data\") pod \"6b626e47-45d3-4865-a8b6-759d71d2046f\" (UID: \"6b626e47-45d3-4865-a8b6-759d71d2046f\") " Jul 12 19:51:33.288092 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:51:33.288081 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b626e47-45d3-4865-a8b6-759d71d2046f-istiod-ca-cert\") pod \"6b626e47-45d3-4865-a8b6-759d71d2046f\" (UID: \"6b626e47-45d3-4865-a8b6-759d71d2046f\") " Jul 12 19:51:33.288355 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:51:33.288119 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b626e47-45d3-4865-a8b6-759d71d2046f-istio-token\") pod \"6b626e47-45d3-4865-a8b6-759d71d2046f\" (UID: \"6b626e47-45d3-4865-a8b6-759d71d2046f\") " Jul 12 19:51:33.288355 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:51:33.288151 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b626e47-45d3-4865-a8b6-759d71d2046f-istio-envoy\") pod \"6b626e47-45d3-4865-a8b6-759d71d2046f\" (UID: \"6b626e47-45d3-4865-a8b6-759d71d2046f\") " Jul 12 19:51:33.288355 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:51:33.288173 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4gp6g\" (UniqueName: \"kubernetes.io/projected/6b626e47-45d3-4865-a8b6-759d71d2046f-kube-api-access-4gp6g\") pod \"6b626e47-45d3-4865-a8b6-759d71d2046f\" (UID: \"6b626e47-45d3-4865-a8b6-759d71d2046f\") " Jul 12 19:51:33.288355 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:51:33.288219 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b626e47-45d3-4865-a8b6-759d71d2046f-credential-socket\") pod \"6b626e47-45d3-4865-a8b6-759d71d2046f\" (UID: \"6b626e47-45d3-4865-a8b6-759d71d2046f\") " Jul 12 19:51:33.288355 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:51:33.288262 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b626e47-45d3-4865-a8b6-759d71d2046f-workload-certs\") pod \"6b626e47-45d3-4865-a8b6-759d71d2046f\" (UID: \"6b626e47-45d3-4865-a8b6-759d71d2046f\") " Jul 12 19:51:33.288355 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:51:33.288277 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b626e47-45d3-4865-a8b6-759d71d2046f-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "6b626e47-45d3-4865-a8b6-759d71d2046f" (UID: "6b626e47-45d3-4865-a8b6-759d71d2046f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 19:51:33.288355 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:51:33.288286 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b626e47-45d3-4865-a8b6-759d71d2046f-istio-podinfo\") pod \"6b626e47-45d3-4865-a8b6-759d71d2046f\" (UID: \"6b626e47-45d3-4865-a8b6-759d71d2046f\") " Jul 12 19:51:33.288721 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:51:33.288511 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b626e47-45d3-4865-a8b6-759d71d2046f-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "6b626e47-45d3-4865-a8b6-759d71d2046f" (UID: "6b626e47-45d3-4865-a8b6-759d71d2046f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 12 19:51:33.288721 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:51:33.288560 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b626e47-45d3-4865-a8b6-759d71d2046f-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "6b626e47-45d3-4865-a8b6-759d71d2046f" (UID: "6b626e47-45d3-4865-a8b6-759d71d2046f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 19:51:33.288721 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:51:33.288699 2565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b626e47-45d3-4865-a8b6-759d71d2046f-credential-socket\") on node \"ip-10-0-139-170.ec2.internal\" DevicePath \"\"" Jul 12 19:51:33.288721 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:51:33.288720 2565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b626e47-45d3-4865-a8b6-759d71d2046f-workload-socket\") on node \"ip-10-0-139-170.ec2.internal\" DevicePath \"\"" Jul 12 19:51:33.288928 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:51:33.288735 2565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b626e47-45d3-4865-a8b6-759d71d2046f-istiod-ca-cert\") on node \"ip-10-0-139-170.ec2.internal\" DevicePath \"\"" Jul 12 19:51:33.288928 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:51:33.288740 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b626e47-45d3-4865-a8b6-759d71d2046f-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "6b626e47-45d3-4865-a8b6-759d71d2046f" (UID: "6b626e47-45d3-4865-a8b6-759d71d2046f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 19:51:33.289130 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:51:33.289109 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b626e47-45d3-4865-a8b6-759d71d2046f-istio-data" (OuterVolumeSpecName: "istio-data") pod "6b626e47-45d3-4865-a8b6-759d71d2046f" (UID: "6b626e47-45d3-4865-a8b6-759d71d2046f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 19:51:33.290436 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:51:33.290397 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b626e47-45d3-4865-a8b6-759d71d2046f-istio-token" (OuterVolumeSpecName: "istio-token") pod "6b626e47-45d3-4865-a8b6-759d71d2046f" (UID: "6b626e47-45d3-4865-a8b6-759d71d2046f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 12 19:51:33.290638 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:51:33.290618 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6b626e47-45d3-4865-a8b6-759d71d2046f-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "6b626e47-45d3-4865-a8b6-759d71d2046f" (UID: "6b626e47-45d3-4865-a8b6-759d71d2046f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 12 19:51:33.290720 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:51:33.290698 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b626e47-45d3-4865-a8b6-759d71d2046f-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "6b626e47-45d3-4865-a8b6-759d71d2046f" (UID: "6b626e47-45d3-4865-a8b6-759d71d2046f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 19:51:33.290849 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:51:33.290830 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b626e47-45d3-4865-a8b6-759d71d2046f-kube-api-access-4gp6g" (OuterVolumeSpecName: "kube-api-access-4gp6g") pod "6b626e47-45d3-4865-a8b6-759d71d2046f" (UID: "6b626e47-45d3-4865-a8b6-759d71d2046f"). InnerVolumeSpecName "kube-api-access-4gp6g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 12 19:51:33.340343 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:51:33.340310 2565 generic.go:356] "Generic (PLEG): container finished" podID="6b626e47-45d3-4865-a8b6-759d71d2046f" containerID="d7fe9c4aaff4dbd7b836162ee0a4a7f2da491da929a977cefb907b1b65a4d94f" exitCode=0 Jul 12 19:51:33.340463 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:51:33.340352 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-41e17b7d-openshift-default-7cc77754f5-j9dlb" event={"ID":"6b626e47-45d3-4865-a8b6-759d71d2046f","Type":"ContainerDied","Data":"d7fe9c4aaff4dbd7b836162ee0a4a7f2da491da929a977cefb907b1b65a4d94f"} Jul 12 19:51:33.340463 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:51:33.340361 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-41e17b7d-openshift-default-7cc77754f5-j9dlb" Jul 12 19:51:33.340463 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:51:33.340395 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-41e17b7d-openshift-default-7cc77754f5-j9dlb" event={"ID":"6b626e47-45d3-4865-a8b6-759d71d2046f","Type":"ContainerDied","Data":"b6de350f01680b0346aa1519f1d80f83770827d4a4518e92897f5f428a33d919"} Jul 12 19:51:33.340463 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:51:33.340417 2565 scope.go:117] "RemoveContainer" containerID="d7fe9c4aaff4dbd7b836162ee0a4a7f2da491da929a977cefb907b1b65a4d94f" Jul 12 19:51:33.356634 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:51:33.356611 2565 scope.go:117] "RemoveContainer" containerID="abae1ba2dbe58590c3607320f7c119047f4771bea5b574d967440785934571fa" Jul 12 19:51:33.363877 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:51:33.363856 2565 scope.go:117] "RemoveContainer" containerID="d7fe9c4aaff4dbd7b836162ee0a4a7f2da491da929a977cefb907b1b65a4d94f" Jul 12 19:51:33.364199 ip-10-0-139-170 kubenswrapper[2565]: E0712 19:51:33.364181 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7fe9c4aaff4dbd7b836162ee0a4a7f2da491da929a977cefb907b1b65a4d94f\": container with ID starting with d7fe9c4aaff4dbd7b836162ee0a4a7f2da491da929a977cefb907b1b65a4d94f not found: ID does not exist" containerID="d7fe9c4aaff4dbd7b836162ee0a4a7f2da491da929a977cefb907b1b65a4d94f" Jul 12 19:51:33.364280 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:51:33.364207 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7fe9c4aaff4dbd7b836162ee0a4a7f2da491da929a977cefb907b1b65a4d94f"} err="failed to get container status \"d7fe9c4aaff4dbd7b836162ee0a4a7f2da491da929a977cefb907b1b65a4d94f\": rpc error: code = NotFound desc = could not find container \"d7fe9c4aaff4dbd7b836162ee0a4a7f2da491da929a977cefb907b1b65a4d94f\": container with ID starting with d7fe9c4aaff4dbd7b836162ee0a4a7f2da491da929a977cefb907b1b65a4d94f not found: ID does not exist" Jul 12 19:51:33.364280 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:51:33.364224 2565 scope.go:117] "RemoveContainer" containerID="abae1ba2dbe58590c3607320f7c119047f4771bea5b574d967440785934571fa" Jul 12 19:51:33.364456 ip-10-0-139-170 kubenswrapper[2565]: E0712 19:51:33.364439 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"abae1ba2dbe58590c3607320f7c119047f4771bea5b574d967440785934571fa\": container with ID starting with abae1ba2dbe58590c3607320f7c119047f4771bea5b574d967440785934571fa not found: ID does not exist" containerID="abae1ba2dbe58590c3607320f7c119047f4771bea5b574d967440785934571fa" Jul 12 19:51:33.364491 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:51:33.364461 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"abae1ba2dbe58590c3607320f7c119047f4771bea5b574d967440785934571fa"} err="failed to get container status \"abae1ba2dbe58590c3607320f7c119047f4771bea5b574d967440785934571fa\": rpc error: code = NotFound desc = could not find container \"abae1ba2dbe58590c3607320f7c119047f4771bea5b574d967440785934571fa\": container with ID starting with abae1ba2dbe58590c3607320f7c119047f4771bea5b574d967440785934571fa not found: ID does not exist" Jul 12 19:51:33.367674 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:51:33.367653 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-41e17b7d-openshift-default-7cc77754f5-j9dlb"] Jul 12 19:51:33.371791 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:51:33.371770 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-41e17b7d-openshift-default-7cc77754f5-j9dlb"] Jul 12 19:51:33.381524 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:51:33.381501 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:51:33.389772 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:51:33.389750 2565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b626e47-45d3-4865-a8b6-759d71d2046f-workload-certs\") on node \"ip-10-0-139-170.ec2.internal\" DevicePath \"\"" Jul 12 19:51:33.389772 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:51:33.389770 2565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b626e47-45d3-4865-a8b6-759d71d2046f-istio-podinfo\") on node \"ip-10-0-139-170.ec2.internal\" DevicePath \"\"" Jul 12 19:51:33.389903 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:51:33.389778 2565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b626e47-45d3-4865-a8b6-759d71d2046f-istio-data\") on node \"ip-10-0-139-170.ec2.internal\" DevicePath \"\"" Jul 12 19:51:33.389903 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:51:33.389787 2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b626e47-45d3-4865-a8b6-759d71d2046f-istio-token\") on node \"ip-10-0-139-170.ec2.internal\" DevicePath \"\"" Jul 12 19:51:33.389903 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:51:33.389794 2565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b626e47-45d3-4865-a8b6-759d71d2046f-istio-envoy\") on node \"ip-10-0-139-170.ec2.internal\" DevicePath \"\"" Jul 12 19:51:33.389903 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:51:33.389802 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4gp6g\" (UniqueName: \"kubernetes.io/projected/6b626e47-45d3-4865-a8b6-759d71d2046f-kube-api-access-4gp6g\") on node \"ip-10-0-139-170.ec2.internal\" DevicePath \"\"" Jul 12 19:51:33.436511 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:51:33.436484 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:51:33.537944 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:51:33.537914 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:51:33.638284 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:51:33.638194 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:51:33.738315 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:51:33.738280 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:51:33.857480 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:51:33.857448 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6b626e47-45d3-4865-a8b6-759d71d2046f" path="/var/lib/kubelet/pods/6b626e47-45d3-4865-a8b6-759d71d2046f/volumes" Jul 12 19:51:39.407436 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:51:39.407402 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:51:39.508905 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:51:39.508843 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:51:45.713841 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:51:45.713809 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:51:45.991527 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:51:45.991456 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-2cdc69a3-openshift-default-6b5b684f59-n7vxw"] Jul 12 19:51:45.991793 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:51:45.991776 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6b626e47-45d3-4865-a8b6-759d71d2046f" containerName="istio-proxy" Jul 12 19:51:45.991793 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:51:45.991791 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="6b626e47-45d3-4865-a8b6-759d71d2046f" containerName="istio-proxy" Jul 12 19:51:45.991969 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:51:45.991802 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6b626e47-45d3-4865-a8b6-759d71d2046f" containerName="istio-proxy" Jul 12 19:51:45.991969 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:51:45.991807 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="6b626e47-45d3-4865-a8b6-759d71d2046f" containerName="istio-proxy" Jul 12 19:51:45.991969 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:51:45.991864 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="6b626e47-45d3-4865-a8b6-759d71d2046f" containerName="istio-proxy" Jul 12 19:51:45.994685 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:51:45.994668 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-2cdc69a3-openshift-default-6b5b684f59-n7vxw" Jul 12 19:51:45.997130 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:51:45.997112 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-2cdc69a3-openshift-default-dockercfg-nk4qj\"" Jul 12 19:51:46.005551 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:51:46.005531 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-2cdc69a3-openshift-default-6b5b684f59-n7vxw"] Jul 12 19:51:46.026532 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:51:46.026503 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:51:46.097661 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:51:46.097631 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4a9e3e3-5f28-488a-ba71-75cae4779719-istio-podinfo\") pod \"e2e-mt-2cdc69a3-openshift-default-6b5b684f59-n7vxw\" (UID: \"a4a9e3e3-5f28-488a-ba71-75cae4779719\") " pod="openshift-ingress/e2e-mt-2cdc69a3-openshift-default-6b5b684f59-n7vxw" Jul 12 19:51:46.097846 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:51:46.097696 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6gw2\" (UniqueName: \"kubernetes.io/projected/a4a9e3e3-5f28-488a-ba71-75cae4779719-kube-api-access-w6gw2\") pod \"e2e-mt-2cdc69a3-openshift-default-6b5b684f59-n7vxw\" (UID: \"a4a9e3e3-5f28-488a-ba71-75cae4779719\") " pod="openshift-ingress/e2e-mt-2cdc69a3-openshift-default-6b5b684f59-n7vxw" Jul 12 19:51:46.097846 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:51:46.097809 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4a9e3e3-5f28-488a-ba71-75cae4779719-credential-socket\") pod \"e2e-mt-2cdc69a3-openshift-default-6b5b684f59-n7vxw\" (UID: \"a4a9e3e3-5f28-488a-ba71-75cae4779719\") " pod="openshift-ingress/e2e-mt-2cdc69a3-openshift-default-6b5b684f59-n7vxw" Jul 12 19:51:46.097997 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:51:46.097867 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4a9e3e3-5f28-488a-ba71-75cae4779719-workload-socket\") pod \"e2e-mt-2cdc69a3-openshift-default-6b5b684f59-n7vxw\" (UID: \"a4a9e3e3-5f28-488a-ba71-75cae4779719\") " pod="openshift-ingress/e2e-mt-2cdc69a3-openshift-default-6b5b684f59-n7vxw" Jul 12 19:51:46.097997 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:51:46.097915 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4a9e3e3-5f28-488a-ba71-75cae4779719-istio-token\") pod \"e2e-mt-2cdc69a3-openshift-default-6b5b684f59-n7vxw\" (UID: \"a4a9e3e3-5f28-488a-ba71-75cae4779719\") " pod="openshift-ingress/e2e-mt-2cdc69a3-openshift-default-6b5b684f59-n7vxw" Jul 12 19:51:46.097997 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:51:46.097942 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4a9e3e3-5f28-488a-ba71-75cae4779719-workload-certs\") pod \"e2e-mt-2cdc69a3-openshift-default-6b5b684f59-n7vxw\" (UID: \"a4a9e3e3-5f28-488a-ba71-75cae4779719\") " pod="openshift-ingress/e2e-mt-2cdc69a3-openshift-default-6b5b684f59-n7vxw" Jul 12 19:51:46.097997 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:51:46.097963 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4a9e3e3-5f28-488a-ba71-75cae4779719-istio-data\") pod \"e2e-mt-2cdc69a3-openshift-default-6b5b684f59-n7vxw\" (UID: \"a4a9e3e3-5f28-488a-ba71-75cae4779719\") " pod="openshift-ingress/e2e-mt-2cdc69a3-openshift-default-6b5b684f59-n7vxw" Jul 12 19:51:46.097997 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:51:46.097989 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4a9e3e3-5f28-488a-ba71-75cae4779719-istio-envoy\") pod \"e2e-mt-2cdc69a3-openshift-default-6b5b684f59-n7vxw\" (UID: \"a4a9e3e3-5f28-488a-ba71-75cae4779719\") " pod="openshift-ingress/e2e-mt-2cdc69a3-openshift-default-6b5b684f59-n7vxw" Jul 12 19:51:46.098182 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:51:46.098016 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4a9e3e3-5f28-488a-ba71-75cae4779719-istiod-ca-cert\") pod \"e2e-mt-2cdc69a3-openshift-default-6b5b684f59-n7vxw\" (UID: \"a4a9e3e3-5f28-488a-ba71-75cae4779719\") " pod="openshift-ingress/e2e-mt-2cdc69a3-openshift-default-6b5b684f59-n7vxw" Jul 12 19:51:46.198832 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:51:46.198792 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4a9e3e3-5f28-488a-ba71-75cae4779719-credential-socket\") pod \"e2e-mt-2cdc69a3-openshift-default-6b5b684f59-n7vxw\" (UID: \"a4a9e3e3-5f28-488a-ba71-75cae4779719\") " pod="openshift-ingress/e2e-mt-2cdc69a3-openshift-default-6b5b684f59-n7vxw" Jul 12 19:51:46.199034 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:51:46.198851 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4a9e3e3-5f28-488a-ba71-75cae4779719-workload-socket\") pod \"e2e-mt-2cdc69a3-openshift-default-6b5b684f59-n7vxw\" (UID: \"a4a9e3e3-5f28-488a-ba71-75cae4779719\") " pod="openshift-ingress/e2e-mt-2cdc69a3-openshift-default-6b5b684f59-n7vxw" Jul 12 19:51:46.199034 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:51:46.198875 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4a9e3e3-5f28-488a-ba71-75cae4779719-istio-token\") pod \"e2e-mt-2cdc69a3-openshift-default-6b5b684f59-n7vxw\" (UID: \"a4a9e3e3-5f28-488a-ba71-75cae4779719\") " pod="openshift-ingress/e2e-mt-2cdc69a3-openshift-default-6b5b684f59-n7vxw" Jul 12 19:51:46.199034 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:51:46.198923 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4a9e3e3-5f28-488a-ba71-75cae4779719-workload-certs\") pod \"e2e-mt-2cdc69a3-openshift-default-6b5b684f59-n7vxw\" (UID: \"a4a9e3e3-5f28-488a-ba71-75cae4779719\") " pod="openshift-ingress/e2e-mt-2cdc69a3-openshift-default-6b5b684f59-n7vxw" Jul 12 19:51:46.199034 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:51:46.198943 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4a9e3e3-5f28-488a-ba71-75cae4779719-istio-data\") pod \"e2e-mt-2cdc69a3-openshift-default-6b5b684f59-n7vxw\" (UID: \"a4a9e3e3-5f28-488a-ba71-75cae4779719\") " pod="openshift-ingress/e2e-mt-2cdc69a3-openshift-default-6b5b684f59-n7vxw" Jul 12 19:51:46.199034 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:51:46.198966 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4a9e3e3-5f28-488a-ba71-75cae4779719-istio-envoy\") pod \"e2e-mt-2cdc69a3-openshift-default-6b5b684f59-n7vxw\" (UID: \"a4a9e3e3-5f28-488a-ba71-75cae4779719\") " pod="openshift-ingress/e2e-mt-2cdc69a3-openshift-default-6b5b684f59-n7vxw" Jul 12 19:51:46.199034 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:51:46.198987 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4a9e3e3-5f28-488a-ba71-75cae4779719-istiod-ca-cert\") pod \"e2e-mt-2cdc69a3-openshift-default-6b5b684f59-n7vxw\" (UID: \"a4a9e3e3-5f28-488a-ba71-75cae4779719\") " pod="openshift-ingress/e2e-mt-2cdc69a3-openshift-default-6b5b684f59-n7vxw" Jul 12 19:51:46.199326 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:51:46.199039 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4a9e3e3-5f28-488a-ba71-75cae4779719-istio-podinfo\") pod \"e2e-mt-2cdc69a3-openshift-default-6b5b684f59-n7vxw\" (UID: \"a4a9e3e3-5f28-488a-ba71-75cae4779719\") " pod="openshift-ingress/e2e-mt-2cdc69a3-openshift-default-6b5b684f59-n7vxw" Jul 12 19:51:46.199326 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:51:46.199140 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6gw2\" (UniqueName: \"kubernetes.io/projected/a4a9e3e3-5f28-488a-ba71-75cae4779719-kube-api-access-w6gw2\") pod \"e2e-mt-2cdc69a3-openshift-default-6b5b684f59-n7vxw\" (UID: \"a4a9e3e3-5f28-488a-ba71-75cae4779719\") " pod="openshift-ingress/e2e-mt-2cdc69a3-openshift-default-6b5b684f59-n7vxw" Jul 12 19:51:46.199326 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:51:46.199231 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4a9e3e3-5f28-488a-ba71-75cae4779719-credential-socket\") pod \"e2e-mt-2cdc69a3-openshift-default-6b5b684f59-n7vxw\" (UID: \"a4a9e3e3-5f28-488a-ba71-75cae4779719\") " pod="openshift-ingress/e2e-mt-2cdc69a3-openshift-default-6b5b684f59-n7vxw" Jul 12 19:51:46.199326 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:51:46.199273 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4a9e3e3-5f28-488a-ba71-75cae4779719-workload-socket\") pod \"e2e-mt-2cdc69a3-openshift-default-6b5b684f59-n7vxw\" (UID: \"a4a9e3e3-5f28-488a-ba71-75cae4779719\") " pod="openshift-ingress/e2e-mt-2cdc69a3-openshift-default-6b5b684f59-n7vxw" Jul 12 19:51:46.199497 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:51:46.199347 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4a9e3e3-5f28-488a-ba71-75cae4779719-istio-data\") pod \"e2e-mt-2cdc69a3-openshift-default-6b5b684f59-n7vxw\" (UID: \"a4a9e3e3-5f28-488a-ba71-75cae4779719\") " pod="openshift-ingress/e2e-mt-2cdc69a3-openshift-default-6b5b684f59-n7vxw" Jul 12 19:51:46.199628 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:51:46.199606 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4a9e3e3-5f28-488a-ba71-75cae4779719-workload-certs\") pod \"e2e-mt-2cdc69a3-openshift-default-6b5b684f59-n7vxw\" (UID: \"a4a9e3e3-5f28-488a-ba71-75cae4779719\") " pod="openshift-ingress/e2e-mt-2cdc69a3-openshift-default-6b5b684f59-n7vxw" Jul 12 19:51:46.199740 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:51:46.199721 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4a9e3e3-5f28-488a-ba71-75cae4779719-istiod-ca-cert\") pod \"e2e-mt-2cdc69a3-openshift-default-6b5b684f59-n7vxw\" (UID: \"a4a9e3e3-5f28-488a-ba71-75cae4779719\") " pod="openshift-ingress/e2e-mt-2cdc69a3-openshift-default-6b5b684f59-n7vxw" Jul 12 19:51:46.201190 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:51:46.201170 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4a9e3e3-5f28-488a-ba71-75cae4779719-istio-envoy\") pod \"e2e-mt-2cdc69a3-openshift-default-6b5b684f59-n7vxw\" (UID: \"a4a9e3e3-5f28-488a-ba71-75cae4779719\") " pod="openshift-ingress/e2e-mt-2cdc69a3-openshift-default-6b5b684f59-n7vxw" Jul 12 19:51:46.201405 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:51:46.201374 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4a9e3e3-5f28-488a-ba71-75cae4779719-istio-podinfo\") pod \"e2e-mt-2cdc69a3-openshift-default-6b5b684f59-n7vxw\" (UID: \"a4a9e3e3-5f28-488a-ba71-75cae4779719\") " pod="openshift-ingress/e2e-mt-2cdc69a3-openshift-default-6b5b684f59-n7vxw" Jul 12 19:51:46.207240 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:51:46.207213 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4a9e3e3-5f28-488a-ba71-75cae4779719-istio-token\") pod \"e2e-mt-2cdc69a3-openshift-default-6b5b684f59-n7vxw\" (UID: \"a4a9e3e3-5f28-488a-ba71-75cae4779719\") " pod="openshift-ingress/e2e-mt-2cdc69a3-openshift-default-6b5b684f59-n7vxw" Jul 12 19:51:46.207370 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:51:46.207272 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6gw2\" (UniqueName: \"kubernetes.io/projected/a4a9e3e3-5f28-488a-ba71-75cae4779719-kube-api-access-w6gw2\") pod \"e2e-mt-2cdc69a3-openshift-default-6b5b684f59-n7vxw\" (UID: \"a4a9e3e3-5f28-488a-ba71-75cae4779719\") " pod="openshift-ingress/e2e-mt-2cdc69a3-openshift-default-6b5b684f59-n7vxw" Jul 12 19:51:46.237222 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:51:46.237188 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:51:46.304950 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:51:46.304860 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-2cdc69a3-openshift-default-6b5b684f59-n7vxw" Jul 12 19:51:46.436686 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:51:46.436664 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-2cdc69a3-openshift-default-6b5b684f59-n7vxw"] Jul 12 19:51:46.439247 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:51:46.439220 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4a9e3e3_5f28_488a_ba71_75cae4779719.slice/crio-28bac98fea595e86166c4e9ec43e4e5451246f07330c2f929b0ad29df8585a7b WatchSource:0}: Error finding container 28bac98fea595e86166c4e9ec43e4e5451246f07330c2f929b0ad29df8585a7b: Status 404 returned error can't find the container with id 28bac98fea595e86166c4e9ec43e4e5451246f07330c2f929b0ad29df8585a7b Jul 12 19:51:46.441346 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:51:46.441313 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 12 19:51:46.441440 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:51:46.441388 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 12 19:51:46.441440 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:51:46.441433 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 12 19:51:47.403442 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:51:47.403402 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2cdc69a3-openshift-default-6b5b684f59-n7vxw" event={"ID":"a4a9e3e3-5f28-488a-ba71-75cae4779719","Type":"ContainerStarted","Data":"adedb4ea28be6c8fd81be48610d38e18a079127d3a02785bef38640eec4f043e"} Jul 12 19:51:47.403845 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:51:47.403506 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2cdc69a3-openshift-default-6b5b684f59-n7vxw" event={"ID":"a4a9e3e3-5f28-488a-ba71-75cae4779719","Type":"ContainerStarted","Data":"28bac98fea595e86166c4e9ec43e4e5451246f07330c2f929b0ad29df8585a7b"} Jul 12 19:51:47.423212 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:51:47.423164 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-2cdc69a3-openshift-default-6b5b684f59-n7vxw" podStartSLOduration=2.423147551 podStartE2EDuration="2.423147551s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 19:51:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 19:51:47.422354419 +0000 UTC m=+1772.154926908" watchObservedRunningTime="2026-07-12 19:51:47.423147551 +0000 UTC m=+1772.155720041" Jul 12 19:51:48.305969 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:51:48.305929 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-2cdc69a3-openshift-default-6b5b684f59-n7vxw" Jul 12 19:51:48.310519 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:51:48.310490 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-2cdc69a3-openshift-default-6b5b684f59-n7vxw" Jul 12 19:51:48.408358 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:51:48.408324 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-2cdc69a3-openshift-default-6b5b684f59-n7vxw" Jul 12 19:51:48.409374 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:51:48.409355 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-2cdc69a3-openshift-default-6b5b684f59-n7vxw" Jul 12 19:51:49.383599 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:51:49.382422 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:51:49.483787 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:51:49.483753 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:51:49.580515 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:51:49.580466 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:51:50.069654 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:51:50.069620 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:51:50.145670 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:51:50.145639 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:51:50.246500 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:51:50.246468 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:51:53.872057 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:51:53.872006 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:52:01.099156 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:52:01.099116 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:52:01.200518 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:52:01.200482 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:52:03.410064 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:52:03.410029 2565 patch_prober.go:28] interesting pod/e2e-mt-2cdc69a3-openshift-default-6b5b684f59-n7vxw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" start-of-body= Jul 12 19:52:03.410452 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:52:03.410097 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-2cdc69a3-openshift-default-6b5b684f59-n7vxw" podUID="a4a9e3e3-5f28-488a-ba71-75cae4779719" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" Jul 12 19:52:03.473856 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:52:03.473821 2565 generic.go:356] "Generic (PLEG): container finished" podID="a4a9e3e3-5f28-488a-ba71-75cae4779719" containerID="adedb4ea28be6c8fd81be48610d38e18a079127d3a02785bef38640eec4f043e" exitCode=0 Jul 12 19:52:03.474069 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:52:03.473921 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2cdc69a3-openshift-default-6b5b684f59-n7vxw" event={"ID":"a4a9e3e3-5f28-488a-ba71-75cae4779719","Type":"ContainerDied","Data":"adedb4ea28be6c8fd81be48610d38e18a079127d3a02785bef38640eec4f043e"} Jul 12 19:52:03.474364 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:52:03.474346 2565 scope.go:117] "RemoveContainer" containerID="adedb4ea28be6c8fd81be48610d38e18a079127d3a02785bef38640eec4f043e" Jul 12 19:52:03.475324 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:52:03.475294 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 12 19:52:03.475380 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:52:03.475365 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 12 19:52:03.475413 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:52:03.475399 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 12 19:52:04.482174 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:52:04.482137 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2cdc69a3-openshift-default-6b5b684f59-n7vxw" event={"ID":"a4a9e3e3-5f28-488a-ba71-75cae4779719","Type":"ContainerStarted","Data":"cfbcc13fbbe90c6fd7a68d105e6be8ffe9461568a1a2ef10ea766a90fbb6efa3"} Jul 12 19:52:05.305269 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:52:05.305230 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-2cdc69a3-openshift-default-6b5b684f59-n7vxw" Jul 12 19:52:05.305505 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:52:05.305478 2565 patch_prober.go:28] interesting pod/e2e-mt-2cdc69a3-openshift-default-6b5b684f59-n7vxw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" start-of-body= Jul 12 19:52:05.305591 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:52:05.305542 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-2cdc69a3-openshift-default-6b5b684f59-n7vxw" podUID="a4a9e3e3-5f28-488a-ba71-75cae4779719" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" Jul 12 19:52:06.305619 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:52:06.305579 2565 patch_prober.go:28] interesting pod/e2e-mt-2cdc69a3-openshift-default-6b5b684f59-n7vxw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" start-of-body= Jul 12 19:52:06.306033 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:52:06.305654 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-2cdc69a3-openshift-default-6b5b684f59-n7vxw" podUID="a4a9e3e3-5f28-488a-ba71-75cae4779719" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" Jul 12 19:52:07.305436 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:52:07.305402 2565 patch_prober.go:28] interesting pod/e2e-mt-2cdc69a3-openshift-default-6b5b684f59-n7vxw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" start-of-body= Jul 12 19:52:07.305651 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:52:07.305477 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-2cdc69a3-openshift-default-6b5b684f59-n7vxw" podUID="a4a9e3e3-5f28-488a-ba71-75cae4779719" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" Jul 12 19:52:08.238358 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:52:08.238279 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-2cdc69a3/e2e-lifecycle-model-2cdc69a3-kserve-764c8cd779-v9g2s"] Jul 12 19:52:08.238706 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:52:08.238691 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="6b626e47-45d3-4865-a8b6-759d71d2046f" containerName="istio-proxy" Jul 12 19:52:08.240693 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:52:08.240678 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-2cdc69a3/e2e-lifecycle-model-2cdc69a3-kserve-764c8cd779-v9g2s" Jul 12 19:52:08.244472 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:52:08.244447 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-2cdc69a3\"/\"kube-root-ca.crt\"" Jul 12 19:52:08.244472 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:52:08.244463 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-2cdc69a3\"/\"e2e-lifecycle-model-2cdc69a3-kserve-self-signed-certs\"" Jul 12 19:52:08.244682 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:52:08.244451 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-2cdc69a3\"/\"openshift-service-ca.crt\"" Jul 12 19:52:08.244682 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:52:08.244453 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-2cdc69a3\"/\"default-dockercfg-fq797\"" Jul 12 19:52:08.249680 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:52:08.249656 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-2cdc69a3/e2e-lifecycle-model-2cdc69a3-kserve-764c8cd779-v9g2s"] Jul 12 19:52:08.297959 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:52:08.297921 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bd993cb5-abfc-4016-acdd-21a501ac526b-dshm\") pod \"e2e-lifecycle-model-2cdc69a3-kserve-764c8cd779-v9g2s\" (UID: \"bd993cb5-abfc-4016-acdd-21a501ac526b\") " pod="ai-tenant-e2e-mt-2cdc69a3/e2e-lifecycle-model-2cdc69a3-kserve-764c8cd779-v9g2s" Jul 12 19:52:08.298128 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:52:08.297968 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxzgv\" (UniqueName: \"kubernetes.io/projected/bd993cb5-abfc-4016-acdd-21a501ac526b-kube-api-access-rxzgv\") pod \"e2e-lifecycle-model-2cdc69a3-kserve-764c8cd779-v9g2s\" (UID: \"bd993cb5-abfc-4016-acdd-21a501ac526b\") " pod="ai-tenant-e2e-mt-2cdc69a3/e2e-lifecycle-model-2cdc69a3-kserve-764c8cd779-v9g2s" Jul 12 19:52:08.298128 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:52:08.298038 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bd993cb5-abfc-4016-acdd-21a501ac526b-model-cache\") pod \"e2e-lifecycle-model-2cdc69a3-kserve-764c8cd779-v9g2s\" (UID: \"bd993cb5-abfc-4016-acdd-21a501ac526b\") " pod="ai-tenant-e2e-mt-2cdc69a3/e2e-lifecycle-model-2cdc69a3-kserve-764c8cd779-v9g2s" Jul 12 19:52:08.298128 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:52:08.298070 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bd993cb5-abfc-4016-acdd-21a501ac526b-tmp-dir\") pod \"e2e-lifecycle-model-2cdc69a3-kserve-764c8cd779-v9g2s\" (UID: \"bd993cb5-abfc-4016-acdd-21a501ac526b\") " pod="ai-tenant-e2e-mt-2cdc69a3/e2e-lifecycle-model-2cdc69a3-kserve-764c8cd779-v9g2s" Jul 12 19:52:08.298128 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:52:08.298108 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bd993cb5-abfc-4016-acdd-21a501ac526b-home\") pod \"e2e-lifecycle-model-2cdc69a3-kserve-764c8cd779-v9g2s\" (UID: \"bd993cb5-abfc-4016-acdd-21a501ac526b\") " pod="ai-tenant-e2e-mt-2cdc69a3/e2e-lifecycle-model-2cdc69a3-kserve-764c8cd779-v9g2s" Jul 12 19:52:08.298266 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:52:08.298219 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bd993cb5-abfc-4016-acdd-21a501ac526b-tls-certs\") pod \"e2e-lifecycle-model-2cdc69a3-kserve-764c8cd779-v9g2s\" (UID: \"bd993cb5-abfc-4016-acdd-21a501ac526b\") " pod="ai-tenant-e2e-mt-2cdc69a3/e2e-lifecycle-model-2cdc69a3-kserve-764c8cd779-v9g2s" Jul 12 19:52:08.306028 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:52:08.306001 2565 patch_prober.go:28] interesting pod/e2e-mt-2cdc69a3-openshift-default-6b5b684f59-n7vxw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" start-of-body= Jul 12 19:52:08.306347 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:52:08.306055 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-2cdc69a3-openshift-default-6b5b684f59-n7vxw" podUID="a4a9e3e3-5f28-488a-ba71-75cae4779719" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" Jul 12 19:52:08.398945 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:52:08.398902 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bd993cb5-abfc-4016-acdd-21a501ac526b-dshm\") pod \"e2e-lifecycle-model-2cdc69a3-kserve-764c8cd779-v9g2s\" (UID: \"bd993cb5-abfc-4016-acdd-21a501ac526b\") " pod="ai-tenant-e2e-mt-2cdc69a3/e2e-lifecycle-model-2cdc69a3-kserve-764c8cd779-v9g2s" Jul 12 19:52:08.399120 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:52:08.398959 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxzgv\" (UniqueName: \"kubernetes.io/projected/bd993cb5-abfc-4016-acdd-21a501ac526b-kube-api-access-rxzgv\") pod \"e2e-lifecycle-model-2cdc69a3-kserve-764c8cd779-v9g2s\" (UID: \"bd993cb5-abfc-4016-acdd-21a501ac526b\") " pod="ai-tenant-e2e-mt-2cdc69a3/e2e-lifecycle-model-2cdc69a3-kserve-764c8cd779-v9g2s" Jul 12 19:52:08.399120 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:52:08.399097 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bd993cb5-abfc-4016-acdd-21a501ac526b-model-cache\") pod \"e2e-lifecycle-model-2cdc69a3-kserve-764c8cd779-v9g2s\" (UID: \"bd993cb5-abfc-4016-acdd-21a501ac526b\") " pod="ai-tenant-e2e-mt-2cdc69a3/e2e-lifecycle-model-2cdc69a3-kserve-764c8cd779-v9g2s" Jul 12 19:52:08.399244 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:52:08.399136 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bd993cb5-abfc-4016-acdd-21a501ac526b-tmp-dir\") pod \"e2e-lifecycle-model-2cdc69a3-kserve-764c8cd779-v9g2s\" (UID: \"bd993cb5-abfc-4016-acdd-21a501ac526b\") " pod="ai-tenant-e2e-mt-2cdc69a3/e2e-lifecycle-model-2cdc69a3-kserve-764c8cd779-v9g2s" Jul 12 19:52:08.399244 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:52:08.399202 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bd993cb5-abfc-4016-acdd-21a501ac526b-home\") pod \"e2e-lifecycle-model-2cdc69a3-kserve-764c8cd779-v9g2s\" (UID: \"bd993cb5-abfc-4016-acdd-21a501ac526b\") " pod="ai-tenant-e2e-mt-2cdc69a3/e2e-lifecycle-model-2cdc69a3-kserve-764c8cd779-v9g2s" Jul 12 19:52:08.399345 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:52:08.399258 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bd993cb5-abfc-4016-acdd-21a501ac526b-tls-certs\") pod \"e2e-lifecycle-model-2cdc69a3-kserve-764c8cd779-v9g2s\" (UID: \"bd993cb5-abfc-4016-acdd-21a501ac526b\") " pod="ai-tenant-e2e-mt-2cdc69a3/e2e-lifecycle-model-2cdc69a3-kserve-764c8cd779-v9g2s" Jul 12 19:52:08.399668 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:52:08.399616 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bd993cb5-abfc-4016-acdd-21a501ac526b-home\") pod \"e2e-lifecycle-model-2cdc69a3-kserve-764c8cd779-v9g2s\" (UID: \"bd993cb5-abfc-4016-acdd-21a501ac526b\") " pod="ai-tenant-e2e-mt-2cdc69a3/e2e-lifecycle-model-2cdc69a3-kserve-764c8cd779-v9g2s" Jul 12 19:52:08.399799 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:52:08.399663 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bd993cb5-abfc-4016-acdd-21a501ac526b-model-cache\") pod \"e2e-lifecycle-model-2cdc69a3-kserve-764c8cd779-v9g2s\" (UID: \"bd993cb5-abfc-4016-acdd-21a501ac526b\") " pod="ai-tenant-e2e-mt-2cdc69a3/e2e-lifecycle-model-2cdc69a3-kserve-764c8cd779-v9g2s" Jul 12 19:52:08.399876 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:52:08.399862 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bd993cb5-abfc-4016-acdd-21a501ac526b-tmp-dir\") pod \"e2e-lifecycle-model-2cdc69a3-kserve-764c8cd779-v9g2s\" (UID: \"bd993cb5-abfc-4016-acdd-21a501ac526b\") " pod="ai-tenant-e2e-mt-2cdc69a3/e2e-lifecycle-model-2cdc69a3-kserve-764c8cd779-v9g2s" Jul 12 19:52:08.401205 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:52:08.401175 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bd993cb5-abfc-4016-acdd-21a501ac526b-dshm\") pod \"e2e-lifecycle-model-2cdc69a3-kserve-764c8cd779-v9g2s\" (UID: \"bd993cb5-abfc-4016-acdd-21a501ac526b\") " pod="ai-tenant-e2e-mt-2cdc69a3/e2e-lifecycle-model-2cdc69a3-kserve-764c8cd779-v9g2s" Jul 12 19:52:08.402144 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:52:08.402120 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bd993cb5-abfc-4016-acdd-21a501ac526b-tls-certs\") pod \"e2e-lifecycle-model-2cdc69a3-kserve-764c8cd779-v9g2s\" (UID: \"bd993cb5-abfc-4016-acdd-21a501ac526b\") " pod="ai-tenant-e2e-mt-2cdc69a3/e2e-lifecycle-model-2cdc69a3-kserve-764c8cd779-v9g2s" Jul 12 19:52:08.407739 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:52:08.407714 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxzgv\" (UniqueName: \"kubernetes.io/projected/bd993cb5-abfc-4016-acdd-21a501ac526b-kube-api-access-rxzgv\") pod \"e2e-lifecycle-model-2cdc69a3-kserve-764c8cd779-v9g2s\" (UID: \"bd993cb5-abfc-4016-acdd-21a501ac526b\") " pod="ai-tenant-e2e-mt-2cdc69a3/e2e-lifecycle-model-2cdc69a3-kserve-764c8cd779-v9g2s" Jul 12 19:52:08.418525 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:52:08.418496 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:52:08.551106 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:52:08.551019 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-2cdc69a3/e2e-lifecycle-model-2cdc69a3-kserve-764c8cd779-v9g2s" Jul 12 19:52:08.689804 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:52:08.689731 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-2cdc69a3/e2e-lifecycle-model-2cdc69a3-kserve-764c8cd779-v9g2s"] Jul 12 19:52:08.692265 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:52:08.692237 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbd993cb5_abfc_4016_acdd_21a501ac526b.slice/crio-8ea6e12677fff6ca49fb2e9b625535f5bba8303676321f5e5ccaf4c96e0057cb WatchSource:0}: Error finding container 8ea6e12677fff6ca49fb2e9b625535f5bba8303676321f5e5ccaf4c96e0057cb: Status 404 returned error can't find the container with id 8ea6e12677fff6ca49fb2e9b625535f5bba8303676321f5e5ccaf4c96e0057cb Jul 12 19:52:09.309622 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:52:09.309593 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-2cdc69a3-openshift-default-6b5b684f59-n7vxw" Jul 12 19:52:09.310092 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:52:09.309866 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-2cdc69a3-openshift-default-6b5b684f59-n7vxw" Jul 12 19:52:09.310581 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:52:09.310566 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-2cdc69a3-openshift-default-6b5b684f59-n7vxw" Jul 12 19:52:09.505178 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:52:09.505140 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-2cdc69a3/e2e-lifecycle-model-2cdc69a3-kserve-764c8cd779-v9g2s" event={"ID":"bd993cb5-abfc-4016-acdd-21a501ac526b","Type":"ContainerStarted","Data":"7955d2f8429f796692192ef4082efd68db872c565bc82dbed2e9073d95d837a3"} Jul 12 19:52:09.505178 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:52:09.505179 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-2cdc69a3/e2e-lifecycle-model-2cdc69a3-kserve-764c8cd779-v9g2s" event={"ID":"bd993cb5-abfc-4016-acdd-21a501ac526b","Type":"ContainerStarted","Data":"8ea6e12677fff6ca49fb2e9b625535f5bba8303676321f5e5ccaf4c96e0057cb"} Jul 12 19:52:09.521795 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:52:09.521745 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-2cdc69a3/e2e-lifecycle-model-2cdc69a3-kserve-764c8cd779-v9g2s" podStartSLOduration=1.521731127 podStartE2EDuration="1.521731127s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 19:52:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 19:52:09.520854811 +0000 UTC m=+1794.253427302" watchObservedRunningTime="2026-07-12 19:52:09.521731127 +0000 UTC m=+1794.254303633" Jul 12 19:52:11.452975 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:52:11.452923 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:52:11.554109 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:52:11.554077 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:52:15.819495 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:52:15.819466 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k5h9m_8e016c2e-0c3a-49da-83d4-f526738c1cbd/ovn-acl-logging/0.log" Jul 12 19:52:15.820898 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:52:15.820865 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k5h9m_8e016c2e-0c3a-49da-83d4-f526738c1cbd/ovn-acl-logging/0.log" Jul 12 19:52:18.551801 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:52:18.551772 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-2cdc69a3/e2e-lifecycle-model-2cdc69a3-kserve-764c8cd779-v9g2s" Jul 12 19:52:18.552215 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:52:18.551819 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-2cdc69a3/e2e-lifecycle-model-2cdc69a3-kserve-764c8cd779-v9g2s" Jul 12 19:52:18.563739 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:52:18.563718 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-2cdc69a3/e2e-lifecycle-model-2cdc69a3-kserve-764c8cd779-v9g2s" Jul 12 19:52:19.560680 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:52:19.560652 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-2cdc69a3/e2e-lifecycle-model-2cdc69a3-kserve-764c8cd779-v9g2s" Jul 12 19:52:41.531526 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:52:41.531484 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:52:41.916532 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:52:41.916494 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:52:47.587041 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:52:47.587005 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:52:52.570960 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:52:52.570922 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-2cdc69a3/e2e-lifecycle-model-2cdc69a3-kserve-764c8cd779-v9g2s"] Jul 12 19:52:52.613671 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:52:52.613639 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:53:07.591139 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:53:07.591095 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-2cdc69a3/e2e-lifecycle-model-2cdc69a3-kserve-764c8cd779-v9g2s" podUID="bd993cb5-abfc-4016-acdd-21a501ac526b" containerName="main" containerID="cri-o://7955d2f8429f796692192ef4082efd68db872c565bc82dbed2e9073d95d837a3" gracePeriod=45 Jul 12 19:53:07.763337 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:53:07.763310 2565 generic.go:356] "Generic (PLEG): container finished" podID="bd993cb5-abfc-4016-acdd-21a501ac526b" containerID="7955d2f8429f796692192ef4082efd68db872c565bc82dbed2e9073d95d837a3" exitCode=0 Jul 12 19:53:07.763491 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:53:07.763376 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-2cdc69a3/e2e-lifecycle-model-2cdc69a3-kserve-764c8cd779-v9g2s" event={"ID":"bd993cb5-abfc-4016-acdd-21a501ac526b","Type":"ContainerDied","Data":"7955d2f8429f796692192ef4082efd68db872c565bc82dbed2e9073d95d837a3"} Jul 12 19:53:07.836318 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:53:07.836295 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-2cdc69a3/e2e-lifecycle-model-2cdc69a3-kserve-764c8cd779-v9g2s" Jul 12 19:53:07.917414 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:53:07.917386 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bd993cb5-abfc-4016-acdd-21a501ac526b-model-cache\") pod \"bd993cb5-abfc-4016-acdd-21a501ac526b\" (UID: \"bd993cb5-abfc-4016-acdd-21a501ac526b\") " Jul 12 19:53:07.917414 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:53:07.917414 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bd993cb5-abfc-4016-acdd-21a501ac526b-home\") pod \"bd993cb5-abfc-4016-acdd-21a501ac526b\" (UID: \"bd993cb5-abfc-4016-acdd-21a501ac526b\") " Jul 12 19:53:07.917651 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:53:07.917440 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rxzgv\" (UniqueName: \"kubernetes.io/projected/bd993cb5-abfc-4016-acdd-21a501ac526b-kube-api-access-rxzgv\") pod \"bd993cb5-abfc-4016-acdd-21a501ac526b\" (UID: \"bd993cb5-abfc-4016-acdd-21a501ac526b\") " Jul 12 19:53:07.917651 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:53:07.917513 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bd993cb5-abfc-4016-acdd-21a501ac526b-tmp-dir\") pod \"bd993cb5-abfc-4016-acdd-21a501ac526b\" (UID: \"bd993cb5-abfc-4016-acdd-21a501ac526b\") " Jul 12 19:53:07.917651 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:53:07.917548 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bd993cb5-abfc-4016-acdd-21a501ac526b-tls-certs\") pod \"bd993cb5-abfc-4016-acdd-21a501ac526b\" (UID: \"bd993cb5-abfc-4016-acdd-21a501ac526b\") " Jul 12 19:53:07.917651 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:53:07.917575 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bd993cb5-abfc-4016-acdd-21a501ac526b-dshm\") pod \"bd993cb5-abfc-4016-acdd-21a501ac526b\" (UID: \"bd993cb5-abfc-4016-acdd-21a501ac526b\") " Jul 12 19:53:07.917931 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:53:07.917702 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd993cb5-abfc-4016-acdd-21a501ac526b-model-cache" (OuterVolumeSpecName: "model-cache") pod "bd993cb5-abfc-4016-acdd-21a501ac526b" (UID: "bd993cb5-abfc-4016-acdd-21a501ac526b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 19:53:07.917931 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:53:07.917717 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd993cb5-abfc-4016-acdd-21a501ac526b-home" (OuterVolumeSpecName: "home") pod "bd993cb5-abfc-4016-acdd-21a501ac526b" (UID: "bd993cb5-abfc-4016-acdd-21a501ac526b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 19:53:07.917931 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:53:07.917777 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd993cb5-abfc-4016-acdd-21a501ac526b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "bd993cb5-abfc-4016-acdd-21a501ac526b" (UID: "bd993cb5-abfc-4016-acdd-21a501ac526b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 19:53:07.917931 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:53:07.917824 2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bd993cb5-abfc-4016-acdd-21a501ac526b-model-cache\") on node \"ip-10-0-139-170.ec2.internal\" DevicePath \"\"" Jul 12 19:53:07.917931 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:53:07.917846 2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bd993cb5-abfc-4016-acdd-21a501ac526b-home\") on node \"ip-10-0-139-170.ec2.internal\" DevicePath \"\"" Jul 12 19:53:07.919672 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:53:07.919649 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd993cb5-abfc-4016-acdd-21a501ac526b-dshm" (OuterVolumeSpecName: "dshm") pod "bd993cb5-abfc-4016-acdd-21a501ac526b" (UID: "bd993cb5-abfc-4016-acdd-21a501ac526b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 19:53:07.919744 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:53:07.919707 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd993cb5-abfc-4016-acdd-21a501ac526b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "bd993cb5-abfc-4016-acdd-21a501ac526b" (UID: "bd993cb5-abfc-4016-acdd-21a501ac526b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 12 19:53:07.919781 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:53:07.919748 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd993cb5-abfc-4016-acdd-21a501ac526b-kube-api-access-rxzgv" (OuterVolumeSpecName: "kube-api-access-rxzgv") pod "bd993cb5-abfc-4016-acdd-21a501ac526b" (UID: "bd993cb5-abfc-4016-acdd-21a501ac526b"). InnerVolumeSpecName "kube-api-access-rxzgv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 12 19:53:08.018323 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:53:08.018282 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rxzgv\" (UniqueName: \"kubernetes.io/projected/bd993cb5-abfc-4016-acdd-21a501ac526b-kube-api-access-rxzgv\") on node \"ip-10-0-139-170.ec2.internal\" DevicePath \"\"" Jul 12 19:53:08.018323 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:53:08.018311 2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bd993cb5-abfc-4016-acdd-21a501ac526b-tmp-dir\") on node \"ip-10-0-139-170.ec2.internal\" DevicePath \"\"" Jul 12 19:53:08.018323 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:53:08.018327 2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bd993cb5-abfc-4016-acdd-21a501ac526b-tls-certs\") on node \"ip-10-0-139-170.ec2.internal\" DevicePath \"\"" Jul 12 19:53:08.018550 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:53:08.018337 2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bd993cb5-abfc-4016-acdd-21a501ac526b-dshm\") on node \"ip-10-0-139-170.ec2.internal\" DevicePath \"\"" Jul 12 19:53:08.770113 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:53:08.770079 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-2cdc69a3/e2e-lifecycle-model-2cdc69a3-kserve-764c8cd779-v9g2s" event={"ID":"bd993cb5-abfc-4016-acdd-21a501ac526b","Type":"ContainerDied","Data":"8ea6e12677fff6ca49fb2e9b625535f5bba8303676321f5e5ccaf4c96e0057cb"} Jul 12 19:53:08.770113 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:53:08.770101 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-2cdc69a3/e2e-lifecycle-model-2cdc69a3-kserve-764c8cd779-v9g2s" Jul 12 19:53:08.770620 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:53:08.770125 2565 scope.go:117] "RemoveContainer" containerID="7955d2f8429f796692192ef4082efd68db872c565bc82dbed2e9073d95d837a3" Jul 12 19:53:08.789414 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:53:08.789387 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-2cdc69a3/e2e-lifecycle-model-2cdc69a3-kserve-764c8cd779-v9g2s"] Jul 12 19:53:08.792428 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:53:08.792404 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-2cdc69a3/e2e-lifecycle-model-2cdc69a3-kserve-764c8cd779-v9g2s"] Jul 12 19:53:09.857669 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:53:09.857638 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bd993cb5-abfc-4016-acdd-21a501ac526b" path="/var/lib/kubelet/pods/bd993cb5-abfc-4016-acdd-21a501ac526b/volumes" Jul 12 19:53:37.830931 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:53:37.830878 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:53:38.808532 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:53:38.808495 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:53:43.478486 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:53:43.478448 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:53:57.443848 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:53:57.443809 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:53:57.544139 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:53:57.544100 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:53:57.695467 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:53:57.695379 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-2cdc69a3-openshift-default-6b5b684f59-n7vxw"] Jul 12 19:53:57.695821 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:53:57.695770 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-2cdc69a3-openshift-default-6b5b684f59-n7vxw" podUID="a4a9e3e3-5f28-488a-ba71-75cae4779719" containerName="istio-proxy" containerID="cri-o://cfbcc13fbbe90c6fd7a68d105e6be8ffe9461568a1a2ef10ea766a90fbb6efa3" gracePeriod=30 Jul 12 19:53:57.719747 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:53:57.719709 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:53:57.946188 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:53:57.946084 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:53:58.977029 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:53:58.976992 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:53:59.077780 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:53:59.077745 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:53:59.177938 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:53:59.177906 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:53:59.850990 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:53:59.850951 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:53:59.952483 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:53:59.952453 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:54:00.008340 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:54:00.008304 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:54:01.374216 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:54:01.374180 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:54:02.950246 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:54:02.950215 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-2cdc69a3-openshift-default-6b5b684f59-n7vxw" Jul 12 19:54:02.994247 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:54:02.994158 2565 generic.go:356] "Generic (PLEG): container finished" podID="a4a9e3e3-5f28-488a-ba71-75cae4779719" containerID="cfbcc13fbbe90c6fd7a68d105e6be8ffe9461568a1a2ef10ea766a90fbb6efa3" exitCode=0 Jul 12 19:54:02.994247 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:54:02.994220 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-2cdc69a3-openshift-default-6b5b684f59-n7vxw" Jul 12 19:54:02.994247 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:54:02.994219 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2cdc69a3-openshift-default-6b5b684f59-n7vxw" event={"ID":"a4a9e3e3-5f28-488a-ba71-75cae4779719","Type":"ContainerDied","Data":"cfbcc13fbbe90c6fd7a68d105e6be8ffe9461568a1a2ef10ea766a90fbb6efa3"} Jul 12 19:54:02.994476 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:54:02.994282 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2cdc69a3-openshift-default-6b5b684f59-n7vxw" event={"ID":"a4a9e3e3-5f28-488a-ba71-75cae4779719","Type":"ContainerDied","Data":"28bac98fea595e86166c4e9ec43e4e5451246f07330c2f929b0ad29df8585a7b"} Jul 12 19:54:02.994476 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:54:02.994311 2565 scope.go:117] "RemoveContainer" containerID="cfbcc13fbbe90c6fd7a68d105e6be8ffe9461568a1a2ef10ea766a90fbb6efa3" Jul 12 19:54:03.002363 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:54:03.002346 2565 scope.go:117] "RemoveContainer" containerID="adedb4ea28be6c8fd81be48610d38e18a079127d3a02785bef38640eec4f043e" Jul 12 19:54:03.010096 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:54:03.010076 2565 scope.go:117] "RemoveContainer" containerID="cfbcc13fbbe90c6fd7a68d105e6be8ffe9461568a1a2ef10ea766a90fbb6efa3" Jul 12 19:54:03.010346 ip-10-0-139-170 kubenswrapper[2565]: E0712 19:54:03.010326 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cfbcc13fbbe90c6fd7a68d105e6be8ffe9461568a1a2ef10ea766a90fbb6efa3\": container with ID starting with cfbcc13fbbe90c6fd7a68d105e6be8ffe9461568a1a2ef10ea766a90fbb6efa3 not found: ID does not exist" containerID="cfbcc13fbbe90c6fd7a68d105e6be8ffe9461568a1a2ef10ea766a90fbb6efa3" Jul 12 19:54:03.010395 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:54:03.010356 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cfbcc13fbbe90c6fd7a68d105e6be8ffe9461568a1a2ef10ea766a90fbb6efa3"} err="failed to get container status \"cfbcc13fbbe90c6fd7a68d105e6be8ffe9461568a1a2ef10ea766a90fbb6efa3\": rpc error: code = NotFound desc = could not find container \"cfbcc13fbbe90c6fd7a68d105e6be8ffe9461568a1a2ef10ea766a90fbb6efa3\": container with ID starting with cfbcc13fbbe90c6fd7a68d105e6be8ffe9461568a1a2ef10ea766a90fbb6efa3 not found: ID does not exist" Jul 12 19:54:03.010395 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:54:03.010381 2565 scope.go:117] "RemoveContainer" containerID="adedb4ea28be6c8fd81be48610d38e18a079127d3a02785bef38640eec4f043e" Jul 12 19:54:03.010604 ip-10-0-139-170 kubenswrapper[2565]: E0712 19:54:03.010589 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"adedb4ea28be6c8fd81be48610d38e18a079127d3a02785bef38640eec4f043e\": container with ID starting with adedb4ea28be6c8fd81be48610d38e18a079127d3a02785bef38640eec4f043e not found: ID does not exist" containerID="adedb4ea28be6c8fd81be48610d38e18a079127d3a02785bef38640eec4f043e" Jul 12 19:54:03.010640 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:54:03.010614 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"adedb4ea28be6c8fd81be48610d38e18a079127d3a02785bef38640eec4f043e"} err="failed to get container status \"adedb4ea28be6c8fd81be48610d38e18a079127d3a02785bef38640eec4f043e\": rpc error: code = NotFound desc = could not find container \"adedb4ea28be6c8fd81be48610d38e18a079127d3a02785bef38640eec4f043e\": container with ID starting with adedb4ea28be6c8fd81be48610d38e18a079127d3a02785bef38640eec4f043e not found: ID does not exist" Jul 12 19:54:03.111585 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:54:03.111548 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4a9e3e3-5f28-488a-ba71-75cae4779719-workload-socket\") pod \"a4a9e3e3-5f28-488a-ba71-75cae4779719\" (UID: \"a4a9e3e3-5f28-488a-ba71-75cae4779719\") " Jul 12 19:54:03.111585 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:54:03.111581 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4a9e3e3-5f28-488a-ba71-75cae4779719-istio-envoy\") pod \"a4a9e3e3-5f28-488a-ba71-75cae4779719\" (UID: \"a4a9e3e3-5f28-488a-ba71-75cae4779719\") " Jul 12 19:54:03.111811 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:54:03.111609 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4a9e3e3-5f28-488a-ba71-75cae4779719-istio-token\") pod \"a4a9e3e3-5f28-488a-ba71-75cae4779719\" (UID: \"a4a9e3e3-5f28-488a-ba71-75cae4779719\") " Jul 12 19:54:03.111811 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:54:03.111649 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w6gw2\" (UniqueName: \"kubernetes.io/projected/a4a9e3e3-5f28-488a-ba71-75cae4779719-kube-api-access-w6gw2\") pod \"a4a9e3e3-5f28-488a-ba71-75cae4779719\" (UID: \"a4a9e3e3-5f28-488a-ba71-75cae4779719\") " Jul 12 19:54:03.111811 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:54:03.111671 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4a9e3e3-5f28-488a-ba71-75cae4779719-credential-socket\") pod \"a4a9e3e3-5f28-488a-ba71-75cae4779719\" (UID: \"a4a9e3e3-5f28-488a-ba71-75cae4779719\") " Jul 12 19:54:03.111811 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:54:03.111718 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4a9e3e3-5f28-488a-ba71-75cae4779719-istiod-ca-cert\") pod \"a4a9e3e3-5f28-488a-ba71-75cae4779719\" (UID: \"a4a9e3e3-5f28-488a-ba71-75cae4779719\") " Jul 12 19:54:03.111811 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:54:03.111757 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4a9e3e3-5f28-488a-ba71-75cae4779719-workload-certs\") pod \"a4a9e3e3-5f28-488a-ba71-75cae4779719\" (UID: \"a4a9e3e3-5f28-488a-ba71-75cae4779719\") " Jul 12 19:54:03.111811 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:54:03.111782 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4a9e3e3-5f28-488a-ba71-75cae4779719-istio-data\") pod \"a4a9e3e3-5f28-488a-ba71-75cae4779719\" (UID: \"a4a9e3e3-5f28-488a-ba71-75cae4779719\") " Jul 12 19:54:03.111811 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:54:03.111811 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4a9e3e3-5f28-488a-ba71-75cae4779719-istio-podinfo\") pod \"a4a9e3e3-5f28-488a-ba71-75cae4779719\" (UID: \"a4a9e3e3-5f28-488a-ba71-75cae4779719\") " Jul 12 19:54:03.112189 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:54:03.111853 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4a9e3e3-5f28-488a-ba71-75cae4779719-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "a4a9e3e3-5f28-488a-ba71-75cae4779719" (UID: "a4a9e3e3-5f28-488a-ba71-75cae4779719"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 19:54:03.112189 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:54:03.112048 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4a9e3e3-5f28-488a-ba71-75cae4779719-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "a4a9e3e3-5f28-488a-ba71-75cae4779719" (UID: "a4a9e3e3-5f28-488a-ba71-75cae4779719"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 19:54:03.112189 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:54:03.112083 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4a9e3e3-5f28-488a-ba71-75cae4779719-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "a4a9e3e3-5f28-488a-ba71-75cae4779719" (UID: "a4a9e3e3-5f28-488a-ba71-75cae4779719"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 19:54:03.112189 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:54:03.112161 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4a9e3e3-5f28-488a-ba71-75cae4779719-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "a4a9e3e3-5f28-488a-ba71-75cae4779719" (UID: "a4a9e3e3-5f28-488a-ba71-75cae4779719"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 12 19:54:03.112375 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:54:03.112257 2565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4a9e3e3-5f28-488a-ba71-75cae4779719-workload-certs\") on node \"ip-10-0-139-170.ec2.internal\" DevicePath \"\"" Jul 12 19:54:03.112375 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:54:03.112276 2565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4a9e3e3-5f28-488a-ba71-75cae4779719-workload-socket\") on node \"ip-10-0-139-170.ec2.internal\" DevicePath \"\"" Jul 12 19:54:03.112375 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:54:03.112291 2565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4a9e3e3-5f28-488a-ba71-75cae4779719-credential-socket\") on node \"ip-10-0-139-170.ec2.internal\" DevicePath \"\"" Jul 12 19:54:03.113056 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:54:03.113032 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4a9e3e3-5f28-488a-ba71-75cae4779719-istio-data" (OuterVolumeSpecName: "istio-data") pod "a4a9e3e3-5f28-488a-ba71-75cae4779719" (UID: "a4a9e3e3-5f28-488a-ba71-75cae4779719"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 19:54:03.113851 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:54:03.113830 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4a9e3e3-5f28-488a-ba71-75cae4779719-kube-api-access-w6gw2" (OuterVolumeSpecName: "kube-api-access-w6gw2") pod "a4a9e3e3-5f28-488a-ba71-75cae4779719" (UID: "a4a9e3e3-5f28-488a-ba71-75cae4779719"). InnerVolumeSpecName "kube-api-access-w6gw2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 12 19:54:03.114108 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:54:03.114090 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a4a9e3e3-5f28-488a-ba71-75cae4779719-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "a4a9e3e3-5f28-488a-ba71-75cae4779719" (UID: "a4a9e3e3-5f28-488a-ba71-75cae4779719"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 12 19:54:03.114251 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:54:03.114231 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4a9e3e3-5f28-488a-ba71-75cae4779719-istio-token" (OuterVolumeSpecName: "istio-token") pod "a4a9e3e3-5f28-488a-ba71-75cae4779719" (UID: "a4a9e3e3-5f28-488a-ba71-75cae4779719"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 12 19:54:03.114300 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:54:03.114247 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4a9e3e3-5f28-488a-ba71-75cae4779719-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "a4a9e3e3-5f28-488a-ba71-75cae4779719" (UID: "a4a9e3e3-5f28-488a-ba71-75cae4779719"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 19:54:03.213010 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:54:03.212975 2565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4a9e3e3-5f28-488a-ba71-75cae4779719-istiod-ca-cert\") on node \"ip-10-0-139-170.ec2.internal\" DevicePath \"\"" Jul 12 19:54:03.213010 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:54:03.213003 2565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4a9e3e3-5f28-488a-ba71-75cae4779719-istio-data\") on node \"ip-10-0-139-170.ec2.internal\" DevicePath \"\"" Jul 12 19:54:03.213010 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:54:03.213013 2565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4a9e3e3-5f28-488a-ba71-75cae4779719-istio-podinfo\") on node \"ip-10-0-139-170.ec2.internal\" DevicePath \"\"" Jul 12 19:54:03.213010 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:54:03.213021 2565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4a9e3e3-5f28-488a-ba71-75cae4779719-istio-envoy\") on node \"ip-10-0-139-170.ec2.internal\" DevicePath \"\"" Jul 12 19:54:03.213270 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:54:03.213029 2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4a9e3e3-5f28-488a-ba71-75cae4779719-istio-token\") on node \"ip-10-0-139-170.ec2.internal\" DevicePath \"\"" Jul 12 19:54:03.213270 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:54:03.213037 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w6gw2\" (UniqueName: \"kubernetes.io/projected/a4a9e3e3-5f28-488a-ba71-75cae4779719-kube-api-access-w6gw2\") on node \"ip-10-0-139-170.ec2.internal\" DevicePath \"\"" Jul 12 19:54:03.319043 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:54:03.319011 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-2cdc69a3-openshift-default-6b5b684f59-n7vxw"] Jul 12 19:54:03.322795 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:54:03.322767 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-2cdc69a3-openshift-default-6b5b684f59-n7vxw"] Jul 12 19:54:03.856931 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:54:03.856896 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a4a9e3e3-5f28-488a-ba71-75cae4779719" path="/var/lib/kubelet/pods/a4a9e3e3-5f28-488a-ba71-75cae4779719/volumes" Jul 12 19:54:07.107413 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:54:07.107382 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:54:07.171020 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:54:07.170988 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:54:07.748284 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:54:07.748242 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:54:13.443224 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:54:13.443189 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:54:13.494939 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:54:13.494902 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:54:13.623287 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:54:13.623251 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:54:13.727192 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:54:13.727101 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:54:24.152400 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:54:24.152365 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:54:24.462847 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:54:24.462761 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:54:45.403531 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:54:45.403495 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:54:50.940115 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:54:50.940074 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:54:50.994250 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:54:50.994211 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:54:51.270785 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:54:51.270680 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:54:51.549137 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:54:51.549060 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-515e44-openshift-default-6b9579f57c-th7p8"] Jul 12 19:54:51.549408 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:54:51.549396 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a4a9e3e3-5f28-488a-ba71-75cae4779719" containerName="istio-proxy" Jul 12 19:54:51.549451 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:54:51.549411 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="a4a9e3e3-5f28-488a-ba71-75cae4779719" containerName="istio-proxy" Jul 12 19:54:51.549451 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:54:51.549418 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a4a9e3e3-5f28-488a-ba71-75cae4779719" containerName="istio-proxy" Jul 12 19:54:51.549451 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:54:51.549424 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="a4a9e3e3-5f28-488a-ba71-75cae4779719" containerName="istio-proxy" Jul 12 19:54:51.549451 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:54:51.549436 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd993cb5-abfc-4016-acdd-21a501ac526b" containerName="main" Jul 12 19:54:51.549451 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:54:51.549441 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd993cb5-abfc-4016-acdd-21a501ac526b" containerName="main" Jul 12 19:54:51.549599 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:54:51.549492 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="a4a9e3e3-5f28-488a-ba71-75cae4779719" containerName="istio-proxy" Jul 12 19:54:51.549599 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:54:51.549499 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="bd993cb5-abfc-4016-acdd-21a501ac526b" containerName="main" Jul 12 19:54:51.549599 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:54:51.549506 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="a4a9e3e3-5f28-488a-ba71-75cae4779719" containerName="istio-proxy" Jul 12 19:54:51.552358 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:54:51.552343 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-515e44-openshift-default-6b9579f57c-th7p8" Jul 12 19:54:51.554647 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:54:51.554630 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-515e44-openshift-default-dockercfg-p9p9n\"" Jul 12 19:54:51.559423 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:54:51.559396 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dmkmr\" (UniqueName: \"kubernetes.io/projected/050ca555-2dae-4e2d-b95b-08c754b1add3-kube-api-access-dmkmr\") pod \"e2e-inf-a-515e44-openshift-default-6b9579f57c-th7p8\" (UID: \"050ca555-2dae-4e2d-b95b-08c754b1add3\") " pod="openshift-ingress/e2e-inf-a-515e44-openshift-default-6b9579f57c-th7p8" Jul 12 19:54:51.559546 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:54:51.559438 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/050ca555-2dae-4e2d-b95b-08c754b1add3-istio-envoy\") pod \"e2e-inf-a-515e44-openshift-default-6b9579f57c-th7p8\" (UID: \"050ca555-2dae-4e2d-b95b-08c754b1add3\") " pod="openshift-ingress/e2e-inf-a-515e44-openshift-default-6b9579f57c-th7p8" Jul 12 19:54:51.559546 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:54:51.559465 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/050ca555-2dae-4e2d-b95b-08c754b1add3-istio-data\") pod \"e2e-inf-a-515e44-openshift-default-6b9579f57c-th7p8\" (UID: \"050ca555-2dae-4e2d-b95b-08c754b1add3\") " pod="openshift-ingress/e2e-inf-a-515e44-openshift-default-6b9579f57c-th7p8" Jul 12 19:54:51.559637 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:54:51.559585 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/050ca555-2dae-4e2d-b95b-08c754b1add3-workload-socket\") pod \"e2e-inf-a-515e44-openshift-default-6b9579f57c-th7p8\" (UID: \"050ca555-2dae-4e2d-b95b-08c754b1add3\") " pod="openshift-ingress/e2e-inf-a-515e44-openshift-default-6b9579f57c-th7p8" Jul 12 19:54:51.559637 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:54:51.559621 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/050ca555-2dae-4e2d-b95b-08c754b1add3-istio-podinfo\") pod \"e2e-inf-a-515e44-openshift-default-6b9579f57c-th7p8\" (UID: \"050ca555-2dae-4e2d-b95b-08c754b1add3\") " pod="openshift-ingress/e2e-inf-a-515e44-openshift-default-6b9579f57c-th7p8" Jul 12 19:54:51.559712 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:54:51.559658 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/050ca555-2dae-4e2d-b95b-08c754b1add3-istiod-ca-cert\") pod \"e2e-inf-a-515e44-openshift-default-6b9579f57c-th7p8\" (UID: \"050ca555-2dae-4e2d-b95b-08c754b1add3\") " pod="openshift-ingress/e2e-inf-a-515e44-openshift-default-6b9579f57c-th7p8" Jul 12 19:54:51.559712 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:54:51.559691 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/050ca555-2dae-4e2d-b95b-08c754b1add3-istio-token\") pod \"e2e-inf-a-515e44-openshift-default-6b9579f57c-th7p8\" (UID: \"050ca555-2dae-4e2d-b95b-08c754b1add3\") " pod="openshift-ingress/e2e-inf-a-515e44-openshift-default-6b9579f57c-th7p8" Jul 12 19:54:51.559787 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:54:51.559759 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/050ca555-2dae-4e2d-b95b-08c754b1add3-credential-socket\") pod \"e2e-inf-a-515e44-openshift-default-6b9579f57c-th7p8\" (UID: \"050ca555-2dae-4e2d-b95b-08c754b1add3\") " pod="openshift-ingress/e2e-inf-a-515e44-openshift-default-6b9579f57c-th7p8" Jul 12 19:54:51.559822 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:54:51.559803 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/050ca555-2dae-4e2d-b95b-08c754b1add3-workload-certs\") pod \"e2e-inf-a-515e44-openshift-default-6b9579f57c-th7p8\" (UID: \"050ca555-2dae-4e2d-b95b-08c754b1add3\") " pod="openshift-ingress/e2e-inf-a-515e44-openshift-default-6b9579f57c-th7p8" Jul 12 19:54:51.563364 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:54:51.563344 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-515e44-openshift-default-6b9579f57c-th7p8"] Jul 12 19:54:51.570563 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:54:51.570540 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:54:51.661158 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:54:51.661118 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/050ca555-2dae-4e2d-b95b-08c754b1add3-workload-certs\") pod \"e2e-inf-a-515e44-openshift-default-6b9579f57c-th7p8\" (UID: \"050ca555-2dae-4e2d-b95b-08c754b1add3\") " pod="openshift-ingress/e2e-inf-a-515e44-openshift-default-6b9579f57c-th7p8" Jul 12 19:54:51.661322 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:54:51.661163 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmkmr\" (UniqueName: \"kubernetes.io/projected/050ca555-2dae-4e2d-b95b-08c754b1add3-kube-api-access-dmkmr\") pod \"e2e-inf-a-515e44-openshift-default-6b9579f57c-th7p8\" (UID: \"050ca555-2dae-4e2d-b95b-08c754b1add3\") " pod="openshift-ingress/e2e-inf-a-515e44-openshift-default-6b9579f57c-th7p8" Jul 12 19:54:51.661322 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:54:51.661183 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/050ca555-2dae-4e2d-b95b-08c754b1add3-istio-envoy\") pod \"e2e-inf-a-515e44-openshift-default-6b9579f57c-th7p8\" (UID: \"050ca555-2dae-4e2d-b95b-08c754b1add3\") " pod="openshift-ingress/e2e-inf-a-515e44-openshift-default-6b9579f57c-th7p8" Jul 12 19:54:51.661322 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:54:51.661292 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/050ca555-2dae-4e2d-b95b-08c754b1add3-istio-data\") pod \"e2e-inf-a-515e44-openshift-default-6b9579f57c-th7p8\" (UID: \"050ca555-2dae-4e2d-b95b-08c754b1add3\") " pod="openshift-ingress/e2e-inf-a-515e44-openshift-default-6b9579f57c-th7p8" Jul 12 19:54:51.661483 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:54:51.661365 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/050ca555-2dae-4e2d-b95b-08c754b1add3-workload-socket\") pod \"e2e-inf-a-515e44-openshift-default-6b9579f57c-th7p8\" (UID: \"050ca555-2dae-4e2d-b95b-08c754b1add3\") " pod="openshift-ingress/e2e-inf-a-515e44-openshift-default-6b9579f57c-th7p8" Jul 12 19:54:51.661483 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:54:51.661392 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/050ca555-2dae-4e2d-b95b-08c754b1add3-istio-podinfo\") pod \"e2e-inf-a-515e44-openshift-default-6b9579f57c-th7p8\" (UID: \"050ca555-2dae-4e2d-b95b-08c754b1add3\") " pod="openshift-ingress/e2e-inf-a-515e44-openshift-default-6b9579f57c-th7p8" Jul 12 19:54:51.661483 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:54:51.661428 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/050ca555-2dae-4e2d-b95b-08c754b1add3-istiod-ca-cert\") pod \"e2e-inf-a-515e44-openshift-default-6b9579f57c-th7p8\" (UID: \"050ca555-2dae-4e2d-b95b-08c754b1add3\") " pod="openshift-ingress/e2e-inf-a-515e44-openshift-default-6b9579f57c-th7p8" Jul 12 19:54:51.661483 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:54:51.661462 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/050ca555-2dae-4e2d-b95b-08c754b1add3-istio-token\") pod \"e2e-inf-a-515e44-openshift-default-6b9579f57c-th7p8\" (UID: \"050ca555-2dae-4e2d-b95b-08c754b1add3\") " pod="openshift-ingress/e2e-inf-a-515e44-openshift-default-6b9579f57c-th7p8" Jul 12 19:54:51.661707 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:54:51.661527 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/050ca555-2dae-4e2d-b95b-08c754b1add3-credential-socket\") pod \"e2e-inf-a-515e44-openshift-default-6b9579f57c-th7p8\" (UID: \"050ca555-2dae-4e2d-b95b-08c754b1add3\") " pod="openshift-ingress/e2e-inf-a-515e44-openshift-default-6b9579f57c-th7p8" Jul 12 19:54:51.661707 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:54:51.661635 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/050ca555-2dae-4e2d-b95b-08c754b1add3-workload-certs\") pod \"e2e-inf-a-515e44-openshift-default-6b9579f57c-th7p8\" (UID: \"050ca555-2dae-4e2d-b95b-08c754b1add3\") " pod="openshift-ingress/e2e-inf-a-515e44-openshift-default-6b9579f57c-th7p8" Jul 12 19:54:51.661838 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:54:51.661734 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/050ca555-2dae-4e2d-b95b-08c754b1add3-istio-data\") pod \"e2e-inf-a-515e44-openshift-default-6b9579f57c-th7p8\" (UID: \"050ca555-2dae-4e2d-b95b-08c754b1add3\") " pod="openshift-ingress/e2e-inf-a-515e44-openshift-default-6b9579f57c-th7p8" Jul 12 19:54:51.661912 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:54:51.661853 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/050ca555-2dae-4e2d-b95b-08c754b1add3-credential-socket\") pod \"e2e-inf-a-515e44-openshift-default-6b9579f57c-th7p8\" (UID: \"050ca555-2dae-4e2d-b95b-08c754b1add3\") " pod="openshift-ingress/e2e-inf-a-515e44-openshift-default-6b9579f57c-th7p8" Jul 12 19:54:51.662015 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:54:51.661994 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/050ca555-2dae-4e2d-b95b-08c754b1add3-workload-socket\") pod \"e2e-inf-a-515e44-openshift-default-6b9579f57c-th7p8\" (UID: \"050ca555-2dae-4e2d-b95b-08c754b1add3\") " pod="openshift-ingress/e2e-inf-a-515e44-openshift-default-6b9579f57c-th7p8" Jul 12 19:54:51.662267 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:54:51.662251 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/050ca555-2dae-4e2d-b95b-08c754b1add3-istiod-ca-cert\") pod \"e2e-inf-a-515e44-openshift-default-6b9579f57c-th7p8\" (UID: \"050ca555-2dae-4e2d-b95b-08c754b1add3\") " pod="openshift-ingress/e2e-inf-a-515e44-openshift-default-6b9579f57c-th7p8" Jul 12 19:54:51.663464 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:54:51.663441 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/050ca555-2dae-4e2d-b95b-08c754b1add3-istio-envoy\") pod \"e2e-inf-a-515e44-openshift-default-6b9579f57c-th7p8\" (UID: \"050ca555-2dae-4e2d-b95b-08c754b1add3\") " pod="openshift-ingress/e2e-inf-a-515e44-openshift-default-6b9579f57c-th7p8" Jul 12 19:54:51.663940 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:54:51.663917 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/050ca555-2dae-4e2d-b95b-08c754b1add3-istio-podinfo\") pod \"e2e-inf-a-515e44-openshift-default-6b9579f57c-th7p8\" (UID: \"050ca555-2dae-4e2d-b95b-08c754b1add3\") " pod="openshift-ingress/e2e-inf-a-515e44-openshift-default-6b9579f57c-th7p8" Jul 12 19:54:51.668534 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:54:51.668511 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dmkmr\" (UniqueName: \"kubernetes.io/projected/050ca555-2dae-4e2d-b95b-08c754b1add3-kube-api-access-dmkmr\") pod \"e2e-inf-a-515e44-openshift-default-6b9579f57c-th7p8\" (UID: \"050ca555-2dae-4e2d-b95b-08c754b1add3\") " pod="openshift-ingress/e2e-inf-a-515e44-openshift-default-6b9579f57c-th7p8" Jul 12 19:54:51.668633 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:54:51.668551 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/050ca555-2dae-4e2d-b95b-08c754b1add3-istio-token\") pod \"e2e-inf-a-515e44-openshift-default-6b9579f57c-th7p8\" (UID: \"050ca555-2dae-4e2d-b95b-08c754b1add3\") " pod="openshift-ingress/e2e-inf-a-515e44-openshift-default-6b9579f57c-th7p8" Jul 12 19:54:51.773353 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:54:51.773319 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:54:51.863014 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:54:51.862939 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-515e44-openshift-default-6b9579f57c-th7p8" Jul 12 19:54:51.995766 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:54:51.995741 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-515e44-openshift-default-6b9579f57c-th7p8"] Jul 12 19:54:51.998143 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:54:51.998115 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod050ca555_2dae_4e2d_b95b_08c754b1add3.slice/crio-8188be17aea16d34acd69f52efe9515fa395a3ed003f0f1ce02c7ac5ec67ce30 WatchSource:0}: Error finding container 8188be17aea16d34acd69f52efe9515fa395a3ed003f0f1ce02c7ac5ec67ce30: Status 404 returned error can't find the container with id 8188be17aea16d34acd69f52efe9515fa395a3ed003f0f1ce02c7ac5ec67ce30 Jul 12 19:54:52.000582 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:54:52.000548 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 12 19:54:52.000657 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:54:52.000626 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 12 19:54:52.000695 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:54:52.000659 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 12 19:54:52.199036 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:54:52.198997 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-515e44-openshift-default-6b9579f57c-th7p8" event={"ID":"050ca555-2dae-4e2d-b95b-08c754b1add3","Type":"ContainerStarted","Data":"4a5f030e609a34acefa0d9ccc031c13033b227ee6382174a5f65f2de8bc57196"} Jul 12 19:54:52.199211 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:54:52.199060 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-515e44-openshift-default-6b9579f57c-th7p8" event={"ID":"050ca555-2dae-4e2d-b95b-08c754b1add3","Type":"ContainerStarted","Data":"8188be17aea16d34acd69f52efe9515fa395a3ed003f0f1ce02c7ac5ec67ce30"} Jul 12 19:54:52.217473 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:54:52.217415 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-515e44-openshift-default-6b9579f57c-th7p8" podStartSLOduration=1.217395407 podStartE2EDuration="1.217395407s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 19:54:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 19:54:52.216044853 +0000 UTC m=+1956.948617379" watchObservedRunningTime="2026-07-12 19:54:52.217395407 +0000 UTC m=+1956.949967899" Jul 12 19:54:52.863562 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:54:52.863530 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-515e44-openshift-default-6b9579f57c-th7p8" Jul 12 19:54:53.868805 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:54:53.868773 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-515e44-openshift-default-6b9579f57c-th7p8" Jul 12 19:54:54.197099 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:54:54.197069 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:54:54.207716 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:54:54.207690 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-515e44-openshift-default-6b9579f57c-th7p8" Jul 12 19:54:54.208528 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:54:54.208509 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-515e44-openshift-default-6b9579f57c-th7p8" Jul 12 19:54:54.277420 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:54:54.277382 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:54:54.338079 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:54:54.338038 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:54:54.439273 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:54:54.439235 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:54:54.746082 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:54:54.746051 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:54:54.848636 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:54:54.848601 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:54:58.677414 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:54:58.677375 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:54:58.996504 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:54:58.996398 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:54:59.287769 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:54:59.287666 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:54:59.705227 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:54:59.705189 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:55:02.265226 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:55:02.265197 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:55:02.370200 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:55:02.370167 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:55:02.464493 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:55:02.464451 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:55:02.577692 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:55:02.577593 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:55:02.803078 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:55:02.803049 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:55:02.908439 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:55:02.908382 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:55:06.724253 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:55:06.724221 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:55:09.208990 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:55:09.208900 2565 patch_prober.go:28] interesting pod/e2e-inf-a-515e44-openshift-default-6b9579f57c-th7p8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body= Jul 12 19:55:09.208990 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:55:09.208956 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-515e44-openshift-default-6b9579f57c-th7p8" podUID="050ca555-2dae-4e2d-b95b-08c754b1add3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" Jul 12 19:55:09.275678 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:55:09.275639 2565 generic.go:356] "Generic (PLEG): container finished" podID="050ca555-2dae-4e2d-b95b-08c754b1add3" containerID="4a5f030e609a34acefa0d9ccc031c13033b227ee6382174a5f65f2de8bc57196" exitCode=0 Jul 12 19:55:09.275850 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:55:09.275712 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-515e44-openshift-default-6b9579f57c-th7p8" event={"ID":"050ca555-2dae-4e2d-b95b-08c754b1add3","Type":"ContainerDied","Data":"4a5f030e609a34acefa0d9ccc031c13033b227ee6382174a5f65f2de8bc57196"} Jul 12 19:55:09.276171 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:55:09.276154 2565 scope.go:117] "RemoveContainer" containerID="4a5f030e609a34acefa0d9ccc031c13033b227ee6382174a5f65f2de8bc57196" Jul 12 19:55:09.277115 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:55:09.277089 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 12 19:55:09.277176 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:55:09.277154 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 12 19:55:09.277211 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:55:09.277183 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 12 19:55:10.282962 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:55:10.282930 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-515e44-openshift-default-6b9579f57c-th7p8" event={"ID":"050ca555-2dae-4e2d-b95b-08c754b1add3","Type":"ContainerStarted","Data":"746d0c121626979ecb8249e4f44a8dfd2f1353537cd29acbef6ba8dfad1478c3"} Jul 12 19:55:10.863822 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:55:10.863784 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-515e44-openshift-default-6b9579f57c-th7p8" Jul 12 19:55:10.864192 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:55:10.864112 2565 patch_prober.go:28] interesting pod/e2e-inf-a-515e44-openshift-default-6b9579f57c-th7p8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body= Jul 12 19:55:10.864392 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:55:10.864364 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-515e44-openshift-default-6b9579f57c-th7p8" podUID="050ca555-2dae-4e2d-b95b-08c754b1add3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" Jul 12 19:55:11.864161 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:55:11.864115 2565 patch_prober.go:28] interesting pod/e2e-inf-a-515e44-openshift-default-6b9579f57c-th7p8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body= Jul 12 19:55:11.864682 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:55:11.864187 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-515e44-openshift-default-6b9579f57c-th7p8" podUID="050ca555-2dae-4e2d-b95b-08c754b1add3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" Jul 12 19:55:12.864120 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:55:12.864077 2565 patch_prober.go:28] interesting pod/e2e-inf-a-515e44-openshift-default-6b9579f57c-th7p8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body= Jul 12 19:55:12.864333 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:55:12.864146 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-515e44-openshift-default-6b9579f57c-th7p8" podUID="050ca555-2dae-4e2d-b95b-08c754b1add3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" Jul 12 19:55:13.863899 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:55:13.863838 2565 patch_prober.go:28] interesting pod/e2e-inf-a-515e44-openshift-default-6b9579f57c-th7p8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body= Jul 12 19:55:13.864107 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:55:13.863927 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-515e44-openshift-default-6b9579f57c-th7p8" podUID="050ca555-2dae-4e2d-b95b-08c754b1add3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" Jul 12 19:55:14.867278 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:55:14.867243 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-515e44-openshift-default-6b9579f57c-th7p8" Jul 12 19:55:14.867756 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:55:14.867635 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-515e44-openshift-default-6b9579f57c-th7p8" Jul 12 19:55:14.868368 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:55:14.868346 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-515e44-openshift-default-6b9579f57c-th7p8" Jul 12 19:55:15.573556 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:55:15.573518 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:55:15.674561 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:55:15.674528 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:55:24.327735 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:55:24.327697 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:55:24.429377 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:55:24.429338 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:55:26.029364 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:55:26.029327 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-515e44/test-model-515e44-kserve-5f9f4cc7c-4vd7j"] Jul 12 19:55:26.032826 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:55:26.032806 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-515e44/test-model-515e44-kserve-5f9f4cc7c-4vd7j" Jul 12 19:55:26.036067 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:55:26.036041 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-515e44\"/\"kube-root-ca.crt\"" Jul 12 19:55:26.036190 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:55:26.036041 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-515e44\"/\"test-model-515e44-kserve-self-signed-certs\"" Jul 12 19:55:26.036190 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:55:26.036096 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-515e44\"/\"openshift-service-ca.crt\"" Jul 12 19:55:26.036190 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:55:26.036041 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-515e44\"/\"default-dockercfg-2wvws\"" Jul 12 19:55:26.042130 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:55:26.042107 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-515e44/test-model-515e44-kserve-5f9f4cc7c-4vd7j"] Jul 12 19:55:26.093006 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:55:26.092975 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3789221-fbe7-4eae-84fc-79347c889948-tls-certs\") pod \"test-model-515e44-kserve-5f9f4cc7c-4vd7j\" (UID: \"d3789221-fbe7-4eae-84fc-79347c889948\") " pod="ai-tenant-e2e-inf-a-515e44/test-model-515e44-kserve-5f9f4cc7c-4vd7j" Jul 12 19:55:26.093006 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:55:26.093008 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d3789221-fbe7-4eae-84fc-79347c889948-dshm\") pod \"test-model-515e44-kserve-5f9f4cc7c-4vd7j\" (UID: \"d3789221-fbe7-4eae-84fc-79347c889948\") " pod="ai-tenant-e2e-inf-a-515e44/test-model-515e44-kserve-5f9f4cc7c-4vd7j" Jul 12 19:55:26.093227 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:55:26.093057 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d3789221-fbe7-4eae-84fc-79347c889948-model-cache\") pod \"test-model-515e44-kserve-5f9f4cc7c-4vd7j\" (UID: \"d3789221-fbe7-4eae-84fc-79347c889948\") " pod="ai-tenant-e2e-inf-a-515e44/test-model-515e44-kserve-5f9f4cc7c-4vd7j" Jul 12 19:55:26.093227 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:55:26.093100 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d3789221-fbe7-4eae-84fc-79347c889948-home\") pod \"test-model-515e44-kserve-5f9f4cc7c-4vd7j\" (UID: \"d3789221-fbe7-4eae-84fc-79347c889948\") " pod="ai-tenant-e2e-inf-a-515e44/test-model-515e44-kserve-5f9f4cc7c-4vd7j" Jul 12 19:55:26.093227 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:55:26.093132 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3789221-fbe7-4eae-84fc-79347c889948-tmp-dir\") pod \"test-model-515e44-kserve-5f9f4cc7c-4vd7j\" (UID: \"d3789221-fbe7-4eae-84fc-79347c889948\") " pod="ai-tenant-e2e-inf-a-515e44/test-model-515e44-kserve-5f9f4cc7c-4vd7j" Jul 12 19:55:26.093227 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:55:26.093151 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gsp4r\" (UniqueName: \"kubernetes.io/projected/d3789221-fbe7-4eae-84fc-79347c889948-kube-api-access-gsp4r\") pod \"test-model-515e44-kserve-5f9f4cc7c-4vd7j\" (UID: \"d3789221-fbe7-4eae-84fc-79347c889948\") " pod="ai-tenant-e2e-inf-a-515e44/test-model-515e44-kserve-5f9f4cc7c-4vd7j" Jul 12 19:55:26.133053 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:55:26.133016 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:55:26.194007 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:55:26.193972 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d3789221-fbe7-4eae-84fc-79347c889948-home\") pod \"test-model-515e44-kserve-5f9f4cc7c-4vd7j\" (UID: \"d3789221-fbe7-4eae-84fc-79347c889948\") " pod="ai-tenant-e2e-inf-a-515e44/test-model-515e44-kserve-5f9f4cc7c-4vd7j" Jul 12 19:55:26.194007 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:55:26.194012 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3789221-fbe7-4eae-84fc-79347c889948-tmp-dir\") pod \"test-model-515e44-kserve-5f9f4cc7c-4vd7j\" (UID: \"d3789221-fbe7-4eae-84fc-79347c889948\") " pod="ai-tenant-e2e-inf-a-515e44/test-model-515e44-kserve-5f9f4cc7c-4vd7j" Jul 12 19:55:26.194266 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:55:26.194033 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gsp4r\" (UniqueName: \"kubernetes.io/projected/d3789221-fbe7-4eae-84fc-79347c889948-kube-api-access-gsp4r\") pod \"test-model-515e44-kserve-5f9f4cc7c-4vd7j\" (UID: \"d3789221-fbe7-4eae-84fc-79347c889948\") " pod="ai-tenant-e2e-inf-a-515e44/test-model-515e44-kserve-5f9f4cc7c-4vd7j" Jul 12 19:55:26.194266 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:55:26.194097 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3789221-fbe7-4eae-84fc-79347c889948-tls-certs\") pod \"test-model-515e44-kserve-5f9f4cc7c-4vd7j\" (UID: \"d3789221-fbe7-4eae-84fc-79347c889948\") " pod="ai-tenant-e2e-inf-a-515e44/test-model-515e44-kserve-5f9f4cc7c-4vd7j" Jul 12 19:55:26.194266 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:55:26.194112 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d3789221-fbe7-4eae-84fc-79347c889948-dshm\") pod \"test-model-515e44-kserve-5f9f4cc7c-4vd7j\" (UID: \"d3789221-fbe7-4eae-84fc-79347c889948\") " pod="ai-tenant-e2e-inf-a-515e44/test-model-515e44-kserve-5f9f4cc7c-4vd7j" Jul 12 19:55:26.194410 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:55:26.194317 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d3789221-fbe7-4eae-84fc-79347c889948-model-cache\") pod \"test-model-515e44-kserve-5f9f4cc7c-4vd7j\" (UID: \"d3789221-fbe7-4eae-84fc-79347c889948\") " pod="ai-tenant-e2e-inf-a-515e44/test-model-515e44-kserve-5f9f4cc7c-4vd7j" Jul 12 19:55:26.194410 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:55:26.194394 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3789221-fbe7-4eae-84fc-79347c889948-tmp-dir\") pod \"test-model-515e44-kserve-5f9f4cc7c-4vd7j\" (UID: \"d3789221-fbe7-4eae-84fc-79347c889948\") " pod="ai-tenant-e2e-inf-a-515e44/test-model-515e44-kserve-5f9f4cc7c-4vd7j" Jul 12 19:55:26.194496 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:55:26.194412 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d3789221-fbe7-4eae-84fc-79347c889948-home\") pod \"test-model-515e44-kserve-5f9f4cc7c-4vd7j\" (UID: \"d3789221-fbe7-4eae-84fc-79347c889948\") " pod="ai-tenant-e2e-inf-a-515e44/test-model-515e44-kserve-5f9f4cc7c-4vd7j" Jul 12 19:55:26.194604 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:55:26.194587 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d3789221-fbe7-4eae-84fc-79347c889948-model-cache\") pod \"test-model-515e44-kserve-5f9f4cc7c-4vd7j\" (UID: \"d3789221-fbe7-4eae-84fc-79347c889948\") " pod="ai-tenant-e2e-inf-a-515e44/test-model-515e44-kserve-5f9f4cc7c-4vd7j" Jul 12 19:55:26.196447 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:55:26.196417 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d3789221-fbe7-4eae-84fc-79347c889948-dshm\") pod \"test-model-515e44-kserve-5f9f4cc7c-4vd7j\" (UID: \"d3789221-fbe7-4eae-84fc-79347c889948\") " pod="ai-tenant-e2e-inf-a-515e44/test-model-515e44-kserve-5f9f4cc7c-4vd7j" Jul 12 19:55:26.196557 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:55:26.196513 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3789221-fbe7-4eae-84fc-79347c889948-tls-certs\") pod \"test-model-515e44-kserve-5f9f4cc7c-4vd7j\" (UID: \"d3789221-fbe7-4eae-84fc-79347c889948\") " pod="ai-tenant-e2e-inf-a-515e44/test-model-515e44-kserve-5f9f4cc7c-4vd7j" Jul 12 19:55:26.201559 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:55:26.201538 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gsp4r\" (UniqueName: \"kubernetes.io/projected/d3789221-fbe7-4eae-84fc-79347c889948-kube-api-access-gsp4r\") pod \"test-model-515e44-kserve-5f9f4cc7c-4vd7j\" (UID: \"d3789221-fbe7-4eae-84fc-79347c889948\") " pod="ai-tenant-e2e-inf-a-515e44/test-model-515e44-kserve-5f9f4cc7c-4vd7j" Jul 12 19:55:26.342873 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:55:26.342768 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-515e44/test-model-515e44-kserve-5f9f4cc7c-4vd7j" Jul 12 19:55:26.469969 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:55:26.469937 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-515e44/test-model-515e44-kserve-5f9f4cc7c-4vd7j"] Jul 12 19:55:26.473028 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:55:26.472997 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd3789221_fbe7_4eae_84fc_79347c889948.slice/crio-e7ac949df918dabe5e8711057746020d0d8c73e5c886617bca794d016ecdb52d WatchSource:0}: Error finding container e7ac949df918dabe5e8711057746020d0d8c73e5c886617bca794d016ecdb52d: Status 404 returned error can't find the container with id e7ac949df918dabe5e8711057746020d0d8c73e5c886617bca794d016ecdb52d Jul 12 19:55:27.357527 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:55:27.357493 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-515e44/test-model-515e44-kserve-5f9f4cc7c-4vd7j" event={"ID":"d3789221-fbe7-4eae-84fc-79347c889948","Type":"ContainerStarted","Data":"f2e75d88fd121df92ed2f9fd2e9ae58d6cf3bfeb4f53e7b092e7f5301971bcb9"} Jul 12 19:55:27.357962 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:55:27.357534 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-515e44/test-model-515e44-kserve-5f9f4cc7c-4vd7j" event={"ID":"d3789221-fbe7-4eae-84fc-79347c889948","Type":"ContainerStarted","Data":"e7ac949df918dabe5e8711057746020d0d8c73e5c886617bca794d016ecdb52d"} Jul 12 19:55:27.373061 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:55:27.373013 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-515e44/test-model-515e44-kserve-5f9f4cc7c-4vd7j" podStartSLOduration=1.372999083 podStartE2EDuration="1.372999083s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 19:55:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 19:55:27.37161387 +0000 UTC m=+1992.104186364" watchObservedRunningTime="2026-07-12 19:55:27.372999083 +0000 UTC m=+1992.105571572" Jul 12 19:55:33.589959 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:55:33.589925 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:55:33.637925 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:55:33.637879 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:55:36.343696 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:55:36.343666 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-515e44/test-model-515e44-kserve-5f9f4cc7c-4vd7j" Jul 12 19:55:36.344105 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:55:36.343714 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-515e44/test-model-515e44-kserve-5f9f4cc7c-4vd7j" Jul 12 19:55:36.355971 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:55:36.355940 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-515e44/test-model-515e44-kserve-5f9f4cc7c-4vd7j" Jul 12 19:55:36.409095 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:55:36.409069 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-515e44/test-model-515e44-kserve-5f9f4cc7c-4vd7j" Jul 12 19:55:41.739100 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:55:41.739063 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:55:41.789902 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:55:41.789846 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:55:43.732556 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:55:43.732522 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-a78ace/test-model-a78ace-kserve-6b65fd5999-2k2xk"] Jul 12 19:55:43.735920 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:55:43.735903 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-a78ace/test-model-a78ace-kserve-6b65fd5999-2k2xk" Jul 12 19:55:43.739530 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:55:43.739496 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-a78ace\"/\"test-model-a78ace-kserve-self-signed-certs\"" Jul 12 19:55:43.739662 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:55:43.739581 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-a78ace\"/\"openshift-service-ca.crt\"" Jul 12 19:55:43.739662 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:55:43.739496 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-a78ace\"/\"kube-root-ca.crt\"" Jul 12 19:55:43.739765 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:55:43.739665 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-a78ace\"/\"default-dockercfg-hccgx\"" Jul 12 19:55:43.745390 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:55:43.745364 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-a78ace/test-model-a78ace-kserve-6b65fd5999-2k2xk"] Jul 12 19:55:43.765012 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:55:43.764981 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:55:43.846331 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:55:43.846289 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bf73efd2-8ee4-4834-9b14-c92649b2bd02-home\") pod \"test-model-a78ace-kserve-6b65fd5999-2k2xk\" (UID: \"bf73efd2-8ee4-4834-9b14-c92649b2bd02\") " pod="ai-tenant-e2e-inf-b-a78ace/test-model-a78ace-kserve-6b65fd5999-2k2xk" Jul 12 19:55:43.846331 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:55:43.846328 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf73efd2-8ee4-4834-9b14-c92649b2bd02-dshm\") pod \"test-model-a78ace-kserve-6b65fd5999-2k2xk\" (UID: \"bf73efd2-8ee4-4834-9b14-c92649b2bd02\") " pod="ai-tenant-e2e-inf-b-a78ace/test-model-a78ace-kserve-6b65fd5999-2k2xk" Jul 12 19:55:43.846559 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:55:43.846354 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf73efd2-8ee4-4834-9b14-c92649b2bd02-tmp-dir\") pod \"test-model-a78ace-kserve-6b65fd5999-2k2xk\" (UID: \"bf73efd2-8ee4-4834-9b14-c92649b2bd02\") " pod="ai-tenant-e2e-inf-b-a78ace/test-model-a78ace-kserve-6b65fd5999-2k2xk" Jul 12 19:55:43.846559 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:55:43.846380 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g7lvf\" (UniqueName: \"kubernetes.io/projected/bf73efd2-8ee4-4834-9b14-c92649b2bd02-kube-api-access-g7lvf\") pod \"test-model-a78ace-kserve-6b65fd5999-2k2xk\" (UID: \"bf73efd2-8ee4-4834-9b14-c92649b2bd02\") " pod="ai-tenant-e2e-inf-b-a78ace/test-model-a78ace-kserve-6b65fd5999-2k2xk" Jul 12 19:55:43.846559 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:55:43.846445 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf73efd2-8ee4-4834-9b14-c92649b2bd02-model-cache\") pod \"test-model-a78ace-kserve-6b65fd5999-2k2xk\" (UID: \"bf73efd2-8ee4-4834-9b14-c92649b2bd02\") " pod="ai-tenant-e2e-inf-b-a78ace/test-model-a78ace-kserve-6b65fd5999-2k2xk" Jul 12 19:55:43.846559 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:55:43.846495 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf73efd2-8ee4-4834-9b14-c92649b2bd02-tls-certs\") pod \"test-model-a78ace-kserve-6b65fd5999-2k2xk\" (UID: \"bf73efd2-8ee4-4834-9b14-c92649b2bd02\") " pod="ai-tenant-e2e-inf-b-a78ace/test-model-a78ace-kserve-6b65fd5999-2k2xk" Jul 12 19:55:43.947130 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:55:43.947096 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf73efd2-8ee4-4834-9b14-c92649b2bd02-tls-certs\") pod \"test-model-a78ace-kserve-6b65fd5999-2k2xk\" (UID: \"bf73efd2-8ee4-4834-9b14-c92649b2bd02\") " pod="ai-tenant-e2e-inf-b-a78ace/test-model-a78ace-kserve-6b65fd5999-2k2xk" Jul 12 19:55:43.947329 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:55:43.947209 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bf73efd2-8ee4-4834-9b14-c92649b2bd02-home\") pod \"test-model-a78ace-kserve-6b65fd5999-2k2xk\" (UID: \"bf73efd2-8ee4-4834-9b14-c92649b2bd02\") " pod="ai-tenant-e2e-inf-b-a78ace/test-model-a78ace-kserve-6b65fd5999-2k2xk" Jul 12 19:55:43.947329 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:55:43.947225 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf73efd2-8ee4-4834-9b14-c92649b2bd02-dshm\") pod \"test-model-a78ace-kserve-6b65fd5999-2k2xk\" (UID: \"bf73efd2-8ee4-4834-9b14-c92649b2bd02\") " pod="ai-tenant-e2e-inf-b-a78ace/test-model-a78ace-kserve-6b65fd5999-2k2xk" Jul 12 19:55:43.947329 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:55:43.947244 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf73efd2-8ee4-4834-9b14-c92649b2bd02-tmp-dir\") pod \"test-model-a78ace-kserve-6b65fd5999-2k2xk\" (UID: \"bf73efd2-8ee4-4834-9b14-c92649b2bd02\") " pod="ai-tenant-e2e-inf-b-a78ace/test-model-a78ace-kserve-6b65fd5999-2k2xk" Jul 12 19:55:43.947329 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:55:43.947262 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7lvf\" (UniqueName: \"kubernetes.io/projected/bf73efd2-8ee4-4834-9b14-c92649b2bd02-kube-api-access-g7lvf\") pod \"test-model-a78ace-kserve-6b65fd5999-2k2xk\" (UID: \"bf73efd2-8ee4-4834-9b14-c92649b2bd02\") " pod="ai-tenant-e2e-inf-b-a78ace/test-model-a78ace-kserve-6b65fd5999-2k2xk" Jul 12 19:55:43.947547 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:55:43.947333 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf73efd2-8ee4-4834-9b14-c92649b2bd02-model-cache\") pod \"test-model-a78ace-kserve-6b65fd5999-2k2xk\" (UID: \"bf73efd2-8ee4-4834-9b14-c92649b2bd02\") " pod="ai-tenant-e2e-inf-b-a78ace/test-model-a78ace-kserve-6b65fd5999-2k2xk" Jul 12 19:55:43.947767 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:55:43.947742 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf73efd2-8ee4-4834-9b14-c92649b2bd02-model-cache\") pod \"test-model-a78ace-kserve-6b65fd5999-2k2xk\" (UID: \"bf73efd2-8ee4-4834-9b14-c92649b2bd02\") " pod="ai-tenant-e2e-inf-b-a78ace/test-model-a78ace-kserve-6b65fd5999-2k2xk" Jul 12 19:55:43.947767 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:55:43.947758 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf73efd2-8ee4-4834-9b14-c92649b2bd02-tmp-dir\") pod \"test-model-a78ace-kserve-6b65fd5999-2k2xk\" (UID: \"bf73efd2-8ee4-4834-9b14-c92649b2bd02\") " pod="ai-tenant-e2e-inf-b-a78ace/test-model-a78ace-kserve-6b65fd5999-2k2xk" Jul 12 19:55:43.947911 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:55:43.947788 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bf73efd2-8ee4-4834-9b14-c92649b2bd02-home\") pod \"test-model-a78ace-kserve-6b65fd5999-2k2xk\" (UID: \"bf73efd2-8ee4-4834-9b14-c92649b2bd02\") " pod="ai-tenant-e2e-inf-b-a78ace/test-model-a78ace-kserve-6b65fd5999-2k2xk" Jul 12 19:55:43.949423 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:55:43.949401 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf73efd2-8ee4-4834-9b14-c92649b2bd02-dshm\") pod \"test-model-a78ace-kserve-6b65fd5999-2k2xk\" (UID: \"bf73efd2-8ee4-4834-9b14-c92649b2bd02\") " pod="ai-tenant-e2e-inf-b-a78ace/test-model-a78ace-kserve-6b65fd5999-2k2xk" Jul 12 19:55:43.949656 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:55:43.949638 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf73efd2-8ee4-4834-9b14-c92649b2bd02-tls-certs\") pod \"test-model-a78ace-kserve-6b65fd5999-2k2xk\" (UID: \"bf73efd2-8ee4-4834-9b14-c92649b2bd02\") " pod="ai-tenant-e2e-inf-b-a78ace/test-model-a78ace-kserve-6b65fd5999-2k2xk" Jul 12 19:55:43.954566 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:55:43.954546 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7lvf\" (UniqueName: \"kubernetes.io/projected/bf73efd2-8ee4-4834-9b14-c92649b2bd02-kube-api-access-g7lvf\") pod \"test-model-a78ace-kserve-6b65fd5999-2k2xk\" (UID: \"bf73efd2-8ee4-4834-9b14-c92649b2bd02\") " pod="ai-tenant-e2e-inf-b-a78ace/test-model-a78ace-kserve-6b65fd5999-2k2xk" Jul 12 19:55:44.046621 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:55:44.046525 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-a78ace/test-model-a78ace-kserve-6b65fd5999-2k2xk" Jul 12 19:55:44.182872 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:55:44.182847 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-a78ace/test-model-a78ace-kserve-6b65fd5999-2k2xk"] Jul 12 19:55:44.185021 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:55:44.184994 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbf73efd2_8ee4_4834_9b14_c92649b2bd02.slice/crio-e7137e2f41c5097b7f22c6e89c5c230e2e6214943c8e6f7951172e0d1bd6f45f WatchSource:0}: Error finding container e7137e2f41c5097b7f22c6e89c5c230e2e6214943c8e6f7951172e0d1bd6f45f: Status 404 returned error can't find the container with id e7137e2f41c5097b7f22c6e89c5c230e2e6214943c8e6f7951172e0d1bd6f45f Jul 12 19:55:44.433705 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:55:44.433672 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-a78ace/test-model-a78ace-kserve-6b65fd5999-2k2xk" event={"ID":"bf73efd2-8ee4-4834-9b14-c92649b2bd02","Type":"ContainerStarted","Data":"6f961549a1727936e0bdd7317056b0614fd460e10a09128467ec5bead89c96a3"} Jul 12 19:55:44.433705 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:55:44.433712 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-a78ace/test-model-a78ace-kserve-6b65fd5999-2k2xk" event={"ID":"bf73efd2-8ee4-4834-9b14-c92649b2bd02","Type":"ContainerStarted","Data":"e7137e2f41c5097b7f22c6e89c5c230e2e6214943c8e6f7951172e0d1bd6f45f"} Jul 12 19:55:44.450776 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:55:44.450726 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-a78ace/test-model-a78ace-kserve-6b65fd5999-2k2xk" podStartSLOduration=1.450707156 podStartE2EDuration="1.450707156s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 19:55:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 19:55:44.448557917 +0000 UTC m=+2009.181130407" watchObservedRunningTime="2026-07-12 19:55:44.450707156 +0000 UTC m=+2009.183279645" Jul 12 19:55:54.046735 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:55:54.046695 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-a78ace/test-model-a78ace-kserve-6b65fd5999-2k2xk" Jul 12 19:55:54.047245 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:55:54.046786 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-a78ace/test-model-a78ace-kserve-6b65fd5999-2k2xk" Jul 12 19:55:54.060295 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:55:54.060265 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-a78ace/test-model-a78ace-kserve-6b65fd5999-2k2xk" Jul 12 19:55:54.492779 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:55:54.492742 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-a78ace/test-model-a78ace-kserve-6b65fd5999-2k2xk" Jul 12 19:56:05.019277 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:56:05.019240 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-515e44/test-model-515e44-kserve-5f9f4cc7c-4vd7j"] Jul 12 19:56:05.043492 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:56:05.043447 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:56:05.889330 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:56:05.889296 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-a78ace/test-model-a78ace-kserve-6b65fd5999-2k2xk"] Jul 12 19:56:05.906395 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:56:05.906361 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:56:06.077739 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:56:06.077704 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:56:06.465361 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:56:06.465325 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:56:06.584774 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:56:06.584744 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:56:06.686411 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:56:06.686377 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:56:06.814534 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:56:06.814447 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:56:11.537252 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:56:11.537218 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:56:20.041015 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:56:20.040974 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-515e44/test-model-515e44-kserve-5f9f4cc7c-4vd7j" podUID="d3789221-fbe7-4eae-84fc-79347c889948" containerName="main" containerID="cri-o://f2e75d88fd121df92ed2f9fd2e9ae58d6cf3bfeb4f53e7b092e7f5301971bcb9" gracePeriod=45 Jul 12 19:56:20.294690 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:56:20.294629 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-515e44/test-model-515e44-kserve-5f9f4cc7c-4vd7j" Jul 12 19:56:20.384868 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:56:20.384837 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3789221-fbe7-4eae-84fc-79347c889948-tls-certs\") pod \"d3789221-fbe7-4eae-84fc-79347c889948\" (UID: \"d3789221-fbe7-4eae-84fc-79347c889948\") " Jul 12 19:56:20.385043 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:56:20.384979 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d3789221-fbe7-4eae-84fc-79347c889948-home\") pod \"d3789221-fbe7-4eae-84fc-79347c889948\" (UID: \"d3789221-fbe7-4eae-84fc-79347c889948\") " Jul 12 19:56:20.385043 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:56:20.385019 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d3789221-fbe7-4eae-84fc-79347c889948-dshm\") pod \"d3789221-fbe7-4eae-84fc-79347c889948\" (UID: \"d3789221-fbe7-4eae-84fc-79347c889948\") " Jul 12 19:56:20.385181 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:56:20.385042 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gsp4r\" (UniqueName: \"kubernetes.io/projected/d3789221-fbe7-4eae-84fc-79347c889948-kube-api-access-gsp4r\") pod \"d3789221-fbe7-4eae-84fc-79347c889948\" (UID: \"d3789221-fbe7-4eae-84fc-79347c889948\") " Jul 12 19:56:20.385181 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:56:20.385082 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d3789221-fbe7-4eae-84fc-79347c889948-model-cache\") pod \"d3789221-fbe7-4eae-84fc-79347c889948\" (UID: \"d3789221-fbe7-4eae-84fc-79347c889948\") " Jul 12 19:56:20.385181 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:56:20.385124 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3789221-fbe7-4eae-84fc-79347c889948-tmp-dir\") pod \"d3789221-fbe7-4eae-84fc-79347c889948\" (UID: \"d3789221-fbe7-4eae-84fc-79347c889948\") " Jul 12 19:56:20.385344 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:56:20.385294 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3789221-fbe7-4eae-84fc-79347c889948-home" (OuterVolumeSpecName: "home") pod "d3789221-fbe7-4eae-84fc-79347c889948" (UID: "d3789221-fbe7-4eae-84fc-79347c889948"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 19:56:20.385416 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:56:20.385395 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3789221-fbe7-4eae-84fc-79347c889948-model-cache" (OuterVolumeSpecName: "model-cache") pod "d3789221-fbe7-4eae-84fc-79347c889948" (UID: "d3789221-fbe7-4eae-84fc-79347c889948"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 19:56:20.385505 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:56:20.385431 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3789221-fbe7-4eae-84fc-79347c889948-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d3789221-fbe7-4eae-84fc-79347c889948" (UID: "d3789221-fbe7-4eae-84fc-79347c889948"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 19:56:20.385505 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:56:20.385477 2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d3789221-fbe7-4eae-84fc-79347c889948-home\") on node \"ip-10-0-139-170.ec2.internal\" DevicePath \"\"" Jul 12 19:56:20.385505 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:56:20.385498 2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d3789221-fbe7-4eae-84fc-79347c889948-model-cache\") on node \"ip-10-0-139-170.ec2.internal\" DevicePath \"\"" Jul 12 19:56:20.387199 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:56:20.387167 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3789221-fbe7-4eae-84fc-79347c889948-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d3789221-fbe7-4eae-84fc-79347c889948" (UID: "d3789221-fbe7-4eae-84fc-79347c889948"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 12 19:56:20.387274 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:56:20.387236 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3789221-fbe7-4eae-84fc-79347c889948-dshm" (OuterVolumeSpecName: "dshm") pod "d3789221-fbe7-4eae-84fc-79347c889948" (UID: "d3789221-fbe7-4eae-84fc-79347c889948"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 19:56:20.387274 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:56:20.387251 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3789221-fbe7-4eae-84fc-79347c889948-kube-api-access-gsp4r" (OuterVolumeSpecName: "kube-api-access-gsp4r") pod "d3789221-fbe7-4eae-84fc-79347c889948" (UID: "d3789221-fbe7-4eae-84fc-79347c889948"). InnerVolumeSpecName "kube-api-access-gsp4r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 12 19:56:20.486788 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:56:20.486739 2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d3789221-fbe7-4eae-84fc-79347c889948-dshm\") on node \"ip-10-0-139-170.ec2.internal\" DevicePath \"\"" Jul 12 19:56:20.486788 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:56:20.486776 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gsp4r\" (UniqueName: \"kubernetes.io/projected/d3789221-fbe7-4eae-84fc-79347c889948-kube-api-access-gsp4r\") on node \"ip-10-0-139-170.ec2.internal\" DevicePath \"\"" Jul 12 19:56:20.486788 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:56:20.486787 2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3789221-fbe7-4eae-84fc-79347c889948-tmp-dir\") on node \"ip-10-0-139-170.ec2.internal\" DevicePath \"\"" Jul 12 19:56:20.486788 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:56:20.486796 2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3789221-fbe7-4eae-84fc-79347c889948-tls-certs\") on node \"ip-10-0-139-170.ec2.internal\" DevicePath \"\"" Jul 12 19:56:20.601740 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:56:20.601651 2565 generic.go:356] "Generic (PLEG): container finished" podID="d3789221-fbe7-4eae-84fc-79347c889948" containerID="f2e75d88fd121df92ed2f9fd2e9ae58d6cf3bfeb4f53e7b092e7f5301971bcb9" exitCode=0 Jul 12 19:56:20.601740 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:56:20.601718 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-515e44/test-model-515e44-kserve-5f9f4cc7c-4vd7j" event={"ID":"d3789221-fbe7-4eae-84fc-79347c889948","Type":"ContainerDied","Data":"f2e75d88fd121df92ed2f9fd2e9ae58d6cf3bfeb4f53e7b092e7f5301971bcb9"} Jul 12 19:56:20.602009 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:56:20.601753 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-515e44/test-model-515e44-kserve-5f9f4cc7c-4vd7j" event={"ID":"d3789221-fbe7-4eae-84fc-79347c889948","Type":"ContainerDied","Data":"e7ac949df918dabe5e8711057746020d0d8c73e5c886617bca794d016ecdb52d"} Jul 12 19:56:20.602009 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:56:20.601776 2565 scope.go:117] "RemoveContainer" containerID="f2e75d88fd121df92ed2f9fd2e9ae58d6cf3bfeb4f53e7b092e7f5301971bcb9" Jul 12 19:56:20.602009 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:56:20.601723 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-515e44/test-model-515e44-kserve-5f9f4cc7c-4vd7j" Jul 12 19:56:20.610113 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:56:20.610096 2565 scope.go:117] "RemoveContainer" containerID="f2e75d88fd121df92ed2f9fd2e9ae58d6cf3bfeb4f53e7b092e7f5301971bcb9" Jul 12 19:56:20.610368 ip-10-0-139-170 kubenswrapper[2565]: E0712 19:56:20.610349 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2e75d88fd121df92ed2f9fd2e9ae58d6cf3bfeb4f53e7b092e7f5301971bcb9\": container with ID starting with f2e75d88fd121df92ed2f9fd2e9ae58d6cf3bfeb4f53e7b092e7f5301971bcb9 not found: ID does not exist" containerID="f2e75d88fd121df92ed2f9fd2e9ae58d6cf3bfeb4f53e7b092e7f5301971bcb9" Jul 12 19:56:20.610412 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:56:20.610375 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2e75d88fd121df92ed2f9fd2e9ae58d6cf3bfeb4f53e7b092e7f5301971bcb9"} err="failed to get container status \"f2e75d88fd121df92ed2f9fd2e9ae58d6cf3bfeb4f53e7b092e7f5301971bcb9\": rpc error: code = NotFound desc = could not find container \"f2e75d88fd121df92ed2f9fd2e9ae58d6cf3bfeb4f53e7b092e7f5301971bcb9\": container with ID starting with f2e75d88fd121df92ed2f9fd2e9ae58d6cf3bfeb4f53e7b092e7f5301971bcb9 not found: ID does not exist" Jul 12 19:56:20.623976 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:56:20.623952 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-515e44/test-model-515e44-kserve-5f9f4cc7c-4vd7j"] Jul 12 19:56:20.626905 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:56:20.626869 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-515e44/test-model-515e44-kserve-5f9f4cc7c-4vd7j"] Jul 12 19:56:20.909937 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:56:20.909870 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-a78ace/test-model-a78ace-kserve-6b65fd5999-2k2xk" podUID="bf73efd2-8ee4-4834-9b14-c92649b2bd02" containerName="main" containerID="cri-o://6f961549a1727936e0bdd7317056b0614fd460e10a09128467ec5bead89c96a3" gracePeriod=45 Jul 12 19:56:21.149006 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:56:21.148980 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-a78ace/test-model-a78ace-kserve-6b65fd5999-2k2xk" Jul 12 19:56:21.193277 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:56:21.193191 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf73efd2-8ee4-4834-9b14-c92649b2bd02-dshm\") pod \"bf73efd2-8ee4-4834-9b14-c92649b2bd02\" (UID: \"bf73efd2-8ee4-4834-9b14-c92649b2bd02\") " Jul 12 19:56:21.193277 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:56:21.193242 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf73efd2-8ee4-4834-9b14-c92649b2bd02-tls-certs\") pod \"bf73efd2-8ee4-4834-9b14-c92649b2bd02\" (UID: \"bf73efd2-8ee4-4834-9b14-c92649b2bd02\") " Jul 12 19:56:21.193277 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:56:21.193274 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g7lvf\" (UniqueName: \"kubernetes.io/projected/bf73efd2-8ee4-4834-9b14-c92649b2bd02-kube-api-access-g7lvf\") pod \"bf73efd2-8ee4-4834-9b14-c92649b2bd02\" (UID: \"bf73efd2-8ee4-4834-9b14-c92649b2bd02\") " Jul 12 19:56:21.193548 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:56:21.193294 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf73efd2-8ee4-4834-9b14-c92649b2bd02-tmp-dir\") pod \"bf73efd2-8ee4-4834-9b14-c92649b2bd02\" (UID: \"bf73efd2-8ee4-4834-9b14-c92649b2bd02\") " Jul 12 19:56:21.193548 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:56:21.193322 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bf73efd2-8ee4-4834-9b14-c92649b2bd02-home\") pod \"bf73efd2-8ee4-4834-9b14-c92649b2bd02\" (UID: \"bf73efd2-8ee4-4834-9b14-c92649b2bd02\") " Jul 12 19:56:21.193548 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:56:21.193360 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf73efd2-8ee4-4834-9b14-c92649b2bd02-model-cache\") pod \"bf73efd2-8ee4-4834-9b14-c92649b2bd02\" (UID: \"bf73efd2-8ee4-4834-9b14-c92649b2bd02\") " Jul 12 19:56:21.193693 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:56:21.193615 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf73efd2-8ee4-4834-9b14-c92649b2bd02-model-cache" (OuterVolumeSpecName: "model-cache") pod "bf73efd2-8ee4-4834-9b14-c92649b2bd02" (UID: "bf73efd2-8ee4-4834-9b14-c92649b2bd02"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 19:56:21.193693 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:56:21.193626 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf73efd2-8ee4-4834-9b14-c92649b2bd02-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "bf73efd2-8ee4-4834-9b14-c92649b2bd02" (UID: "bf73efd2-8ee4-4834-9b14-c92649b2bd02"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 19:56:21.193693 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:56:21.193650 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf73efd2-8ee4-4834-9b14-c92649b2bd02-home" (OuterVolumeSpecName: "home") pod "bf73efd2-8ee4-4834-9b14-c92649b2bd02" (UID: "bf73efd2-8ee4-4834-9b14-c92649b2bd02"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 19:56:21.195537 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:56:21.195508 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf73efd2-8ee4-4834-9b14-c92649b2bd02-dshm" (OuterVolumeSpecName: "dshm") pod "bf73efd2-8ee4-4834-9b14-c92649b2bd02" (UID: "bf73efd2-8ee4-4834-9b14-c92649b2bd02"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 19:56:21.195647 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:56:21.195564 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf73efd2-8ee4-4834-9b14-c92649b2bd02-kube-api-access-g7lvf" (OuterVolumeSpecName: "kube-api-access-g7lvf") pod "bf73efd2-8ee4-4834-9b14-c92649b2bd02" (UID: "bf73efd2-8ee4-4834-9b14-c92649b2bd02"). InnerVolumeSpecName "kube-api-access-g7lvf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 12 19:56:21.195647 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:56:21.195568 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf73efd2-8ee4-4834-9b14-c92649b2bd02-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "bf73efd2-8ee4-4834-9b14-c92649b2bd02" (UID: "bf73efd2-8ee4-4834-9b14-c92649b2bd02"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 12 19:56:21.294936 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:56:21.294900 2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf73efd2-8ee4-4834-9b14-c92649b2bd02-dshm\") on node \"ip-10-0-139-170.ec2.internal\" DevicePath \"\"" Jul 12 19:56:21.294936 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:56:21.294930 2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf73efd2-8ee4-4834-9b14-c92649b2bd02-tls-certs\") on node \"ip-10-0-139-170.ec2.internal\" DevicePath \"\"" Jul 12 19:56:21.294936 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:56:21.294941 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g7lvf\" (UniqueName: \"kubernetes.io/projected/bf73efd2-8ee4-4834-9b14-c92649b2bd02-kube-api-access-g7lvf\") on node \"ip-10-0-139-170.ec2.internal\" DevicePath \"\"" Jul 12 19:56:21.295173 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:56:21.294972 2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf73efd2-8ee4-4834-9b14-c92649b2bd02-tmp-dir\") on node \"ip-10-0-139-170.ec2.internal\" DevicePath \"\"" Jul 12 19:56:21.295173 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:56:21.294981 2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bf73efd2-8ee4-4834-9b14-c92649b2bd02-home\") on node \"ip-10-0-139-170.ec2.internal\" DevicePath \"\"" Jul 12 19:56:21.295173 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:56:21.294989 2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf73efd2-8ee4-4834-9b14-c92649b2bd02-model-cache\") on node \"ip-10-0-139-170.ec2.internal\" DevicePath \"\"" Jul 12 19:56:21.606842 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:56:21.606750 2565 generic.go:356] "Generic (PLEG): container finished" podID="bf73efd2-8ee4-4834-9b14-c92649b2bd02" containerID="6f961549a1727936e0bdd7317056b0614fd460e10a09128467ec5bead89c96a3" exitCode=0 Jul 12 19:56:21.606842 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:56:21.606793 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-a78ace/test-model-a78ace-kserve-6b65fd5999-2k2xk" Jul 12 19:56:21.607102 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:56:21.606844 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-a78ace/test-model-a78ace-kserve-6b65fd5999-2k2xk" event={"ID":"bf73efd2-8ee4-4834-9b14-c92649b2bd02","Type":"ContainerDied","Data":"6f961549a1727936e0bdd7317056b0614fd460e10a09128467ec5bead89c96a3"} Jul 12 19:56:21.607102 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:56:21.606913 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-a78ace/test-model-a78ace-kserve-6b65fd5999-2k2xk" event={"ID":"bf73efd2-8ee4-4834-9b14-c92649b2bd02","Type":"ContainerDied","Data":"e7137e2f41c5097b7f22c6e89c5c230e2e6214943c8e6f7951172e0d1bd6f45f"} Jul 12 19:56:21.607102 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:56:21.606937 2565 scope.go:117] "RemoveContainer" containerID="6f961549a1727936e0bdd7317056b0614fd460e10a09128467ec5bead89c96a3" Jul 12 19:56:21.615741 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:56:21.615726 2565 scope.go:117] "RemoveContainer" containerID="6f961549a1727936e0bdd7317056b0614fd460e10a09128467ec5bead89c96a3" Jul 12 19:56:21.616028 ip-10-0-139-170 kubenswrapper[2565]: E0712 19:56:21.616005 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f961549a1727936e0bdd7317056b0614fd460e10a09128467ec5bead89c96a3\": container with ID starting with 6f961549a1727936e0bdd7317056b0614fd460e10a09128467ec5bead89c96a3 not found: ID does not exist" containerID="6f961549a1727936e0bdd7317056b0614fd460e10a09128467ec5bead89c96a3" Jul 12 19:56:21.616117 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:56:21.616033 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f961549a1727936e0bdd7317056b0614fd460e10a09128467ec5bead89c96a3"} err="failed to get container status \"6f961549a1727936e0bdd7317056b0614fd460e10a09128467ec5bead89c96a3\": rpc error: code = NotFound desc = could not find container \"6f961549a1727936e0bdd7317056b0614fd460e10a09128467ec5bead89c96a3\": container with ID starting with 6f961549a1727936e0bdd7317056b0614fd460e10a09128467ec5bead89c96a3 not found: ID does not exist" Jul 12 19:56:21.627261 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:56:21.627239 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-a78ace/test-model-a78ace-kserve-6b65fd5999-2k2xk"] Jul 12 19:56:21.630699 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:56:21.630678 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-a78ace/test-model-a78ace-kserve-6b65fd5999-2k2xk"] Jul 12 19:56:21.856797 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:56:21.856769 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bf73efd2-8ee4-4834-9b14-c92649b2bd02" path="/var/lib/kubelet/pods/bf73efd2-8ee4-4834-9b14-c92649b2bd02/volumes" Jul 12 19:56:21.857202 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:56:21.857162 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d3789221-fbe7-4eae-84fc-79347c889948" path="/var/lib/kubelet/pods/d3789221-fbe7-4eae-84fc-79347c889948/volumes" Jul 12 19:56:48.192148 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:56:48.192055 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:56:48.442145 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:56:48.442112 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-515e44-openshift-default-6b9579f57c-th7p8"] Jul 12 19:56:48.442376 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:56:48.442328 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-515e44-openshift-default-6b9579f57c-th7p8" podUID="050ca555-2dae-4e2d-b95b-08c754b1add3" containerName="istio-proxy" containerID="cri-o://746d0c121626979ecb8249e4f44a8dfd2f1353537cd29acbef6ba8dfad1478c3" gracePeriod=30 Jul 12 19:56:48.457051 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:56:48.457024 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:56:48.676787 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:56:48.676755 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:56:48.839223 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:56:48.839139 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:56:49.227026 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:56:49.226996 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:56:49.424175 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:56:49.424142 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:56:49.526575 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:56:49.526489 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:56:49.701140 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:56:49.701106 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:56:53.697040 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:56:53.697016 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-515e44-openshift-default-6b9579f57c-th7p8" Jul 12 19:56:53.751725 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:56:53.751626 2565 generic.go:356] "Generic (PLEG): container finished" podID="050ca555-2dae-4e2d-b95b-08c754b1add3" containerID="746d0c121626979ecb8249e4f44a8dfd2f1353537cd29acbef6ba8dfad1478c3" exitCode=0 Jul 12 19:56:53.751725 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:56:53.751675 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-515e44-openshift-default-6b9579f57c-th7p8" event={"ID":"050ca555-2dae-4e2d-b95b-08c754b1add3","Type":"ContainerDied","Data":"746d0c121626979ecb8249e4f44a8dfd2f1353537cd29acbef6ba8dfad1478c3"} Jul 12 19:56:53.751960 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:56:53.751691 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-515e44-openshift-default-6b9579f57c-th7p8" Jul 12 19:56:53.751960 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:56:53.751769 2565 scope.go:117] "RemoveContainer" containerID="746d0c121626979ecb8249e4f44a8dfd2f1353537cd29acbef6ba8dfad1478c3" Jul 12 19:56:53.751960 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:56:53.751756 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-515e44-openshift-default-6b9579f57c-th7p8" event={"ID":"050ca555-2dae-4e2d-b95b-08c754b1add3","Type":"ContainerDied","Data":"8188be17aea16d34acd69f52efe9515fa395a3ed003f0f1ce02c7ac5ec67ce30"} Jul 12 19:56:53.759707 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:56:53.759692 2565 scope.go:117] "RemoveContainer" containerID="4a5f030e609a34acefa0d9ccc031c13033b227ee6382174a5f65f2de8bc57196" Jul 12 19:56:53.767191 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:56:53.767174 2565 scope.go:117] "RemoveContainer" containerID="746d0c121626979ecb8249e4f44a8dfd2f1353537cd29acbef6ba8dfad1478c3" Jul 12 19:56:53.767435 ip-10-0-139-170 kubenswrapper[2565]: E0712 19:56:53.767415 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"746d0c121626979ecb8249e4f44a8dfd2f1353537cd29acbef6ba8dfad1478c3\": container with ID starting with 746d0c121626979ecb8249e4f44a8dfd2f1353537cd29acbef6ba8dfad1478c3 not found: ID does not exist" containerID="746d0c121626979ecb8249e4f44a8dfd2f1353537cd29acbef6ba8dfad1478c3" Jul 12 19:56:53.767513 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:56:53.767448 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"746d0c121626979ecb8249e4f44a8dfd2f1353537cd29acbef6ba8dfad1478c3"} err="failed to get container status \"746d0c121626979ecb8249e4f44a8dfd2f1353537cd29acbef6ba8dfad1478c3\": rpc error: code = NotFound desc = could not find container \"746d0c121626979ecb8249e4f44a8dfd2f1353537cd29acbef6ba8dfad1478c3\": container with ID starting with 746d0c121626979ecb8249e4f44a8dfd2f1353537cd29acbef6ba8dfad1478c3 not found: ID does not exist" Jul 12 19:56:53.767513 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:56:53.767476 2565 scope.go:117] "RemoveContainer" containerID="4a5f030e609a34acefa0d9ccc031c13033b227ee6382174a5f65f2de8bc57196" Jul 12 19:56:53.767714 ip-10-0-139-170 kubenswrapper[2565]: E0712 19:56:53.767698 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a5f030e609a34acefa0d9ccc031c13033b227ee6382174a5f65f2de8bc57196\": container with ID starting with 4a5f030e609a34acefa0d9ccc031c13033b227ee6382174a5f65f2de8bc57196 not found: ID does not exist" containerID="4a5f030e609a34acefa0d9ccc031c13033b227ee6382174a5f65f2de8bc57196" Jul 12 19:56:53.767760 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:56:53.767719 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a5f030e609a34acefa0d9ccc031c13033b227ee6382174a5f65f2de8bc57196"} err="failed to get container status \"4a5f030e609a34acefa0d9ccc031c13033b227ee6382174a5f65f2de8bc57196\": rpc error: code = NotFound desc = could not find container \"4a5f030e609a34acefa0d9ccc031c13033b227ee6382174a5f65f2de8bc57196\": container with ID starting with 4a5f030e609a34acefa0d9ccc031c13033b227ee6382174a5f65f2de8bc57196 not found: ID does not exist" Jul 12 19:56:53.784210 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:56:53.784183 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/050ca555-2dae-4e2d-b95b-08c754b1add3-workload-certs\") pod \"050ca555-2dae-4e2d-b95b-08c754b1add3\" (UID: \"050ca555-2dae-4e2d-b95b-08c754b1add3\") " Jul 12 19:56:53.784305 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:56:53.784243 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/050ca555-2dae-4e2d-b95b-08c754b1add3-workload-socket\") pod \"050ca555-2dae-4e2d-b95b-08c754b1add3\" (UID: \"050ca555-2dae-4e2d-b95b-08c754b1add3\") " Jul 12 19:56:53.784305 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:56:53.784263 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/050ca555-2dae-4e2d-b95b-08c754b1add3-istio-token\") pod \"050ca555-2dae-4e2d-b95b-08c754b1add3\" (UID: \"050ca555-2dae-4e2d-b95b-08c754b1add3\") " Jul 12 19:56:53.784305 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:56:53.784285 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/050ca555-2dae-4e2d-b95b-08c754b1add3-istio-data\") pod \"050ca555-2dae-4e2d-b95b-08c754b1add3\" (UID: \"050ca555-2dae-4e2d-b95b-08c754b1add3\") " Jul 12 19:56:53.784305 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:56:53.784302 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/050ca555-2dae-4e2d-b95b-08c754b1add3-istio-podinfo\") pod \"050ca555-2dae-4e2d-b95b-08c754b1add3\" (UID: \"050ca555-2dae-4e2d-b95b-08c754b1add3\") " Jul 12 19:56:53.784500 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:56:53.784328 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/050ca555-2dae-4e2d-b95b-08c754b1add3-istio-envoy\") pod \"050ca555-2dae-4e2d-b95b-08c754b1add3\" (UID: \"050ca555-2dae-4e2d-b95b-08c754b1add3\") " Jul 12 19:56:53.784500 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:56:53.784343 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/050ca555-2dae-4e2d-b95b-08c754b1add3-istiod-ca-cert\") pod \"050ca555-2dae-4e2d-b95b-08c754b1add3\" (UID: \"050ca555-2dae-4e2d-b95b-08c754b1add3\") " Jul 12 19:56:53.784619 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:56:53.784530 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/050ca555-2dae-4e2d-b95b-08c754b1add3-credential-socket\") pod \"050ca555-2dae-4e2d-b95b-08c754b1add3\" (UID: \"050ca555-2dae-4e2d-b95b-08c754b1add3\") " Jul 12 19:56:53.784619 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:56:53.784521 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/050ca555-2dae-4e2d-b95b-08c754b1add3-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "050ca555-2dae-4e2d-b95b-08c754b1add3" (UID: "050ca555-2dae-4e2d-b95b-08c754b1add3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 19:56:53.784619 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:56:53.784575 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dmkmr\" (UniqueName: \"kubernetes.io/projected/050ca555-2dae-4e2d-b95b-08c754b1add3-kube-api-access-dmkmr\") pod \"050ca555-2dae-4e2d-b95b-08c754b1add3\" (UID: \"050ca555-2dae-4e2d-b95b-08c754b1add3\") " Jul 12 19:56:53.784619 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:56:53.784548 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/050ca555-2dae-4e2d-b95b-08c754b1add3-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "050ca555-2dae-4e2d-b95b-08c754b1add3" (UID: "050ca555-2dae-4e2d-b95b-08c754b1add3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 19:56:53.784813 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:56:53.784745 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/050ca555-2dae-4e2d-b95b-08c754b1add3-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "050ca555-2dae-4e2d-b95b-08c754b1add3" (UID: "050ca555-2dae-4e2d-b95b-08c754b1add3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 19:56:53.784813 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:56:53.784761 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/050ca555-2dae-4e2d-b95b-08c754b1add3-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "050ca555-2dae-4e2d-b95b-08c754b1add3" (UID: "050ca555-2dae-4e2d-b95b-08c754b1add3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 12 19:56:53.784943 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:56:53.784918 2565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/050ca555-2dae-4e2d-b95b-08c754b1add3-istiod-ca-cert\") on node \"ip-10-0-139-170.ec2.internal\" DevicePath \"\"" Jul 12 19:56:53.784943 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:56:53.784935 2565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/050ca555-2dae-4e2d-b95b-08c754b1add3-credential-socket\") on node \"ip-10-0-139-170.ec2.internal\" DevicePath \"\"" Jul 12 19:56:53.785037 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:56:53.784945 2565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/050ca555-2dae-4e2d-b95b-08c754b1add3-workload-certs\") on node \"ip-10-0-139-170.ec2.internal\" DevicePath \"\"" Jul 12 19:56:53.785037 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:56:53.784961 2565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/050ca555-2dae-4e2d-b95b-08c754b1add3-workload-socket\") on node \"ip-10-0-139-170.ec2.internal\" DevicePath \"\"" Jul 12 19:56:53.785295 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:56:53.785270 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/050ca555-2dae-4e2d-b95b-08c754b1add3-istio-data" (OuterVolumeSpecName: "istio-data") pod "050ca555-2dae-4e2d-b95b-08c754b1add3" (UID: "050ca555-2dae-4e2d-b95b-08c754b1add3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 19:56:53.786572 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:56:53.786540 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/050ca555-2dae-4e2d-b95b-08c754b1add3-istio-token" (OuterVolumeSpecName: "istio-token") pod "050ca555-2dae-4e2d-b95b-08c754b1add3" (UID: "050ca555-2dae-4e2d-b95b-08c754b1add3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 12 19:56:53.786653 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:56:53.786569 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/050ca555-2dae-4e2d-b95b-08c754b1add3-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "050ca555-2dae-4e2d-b95b-08c754b1add3" (UID: "050ca555-2dae-4e2d-b95b-08c754b1add3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 19:56:53.786707 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:56:53.786659 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/050ca555-2dae-4e2d-b95b-08c754b1add3-kube-api-access-dmkmr" (OuterVolumeSpecName: "kube-api-access-dmkmr") pod "050ca555-2dae-4e2d-b95b-08c754b1add3" (UID: "050ca555-2dae-4e2d-b95b-08c754b1add3"). InnerVolumeSpecName "kube-api-access-dmkmr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 12 19:56:53.786707 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:56:53.786687 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/050ca555-2dae-4e2d-b95b-08c754b1add3-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "050ca555-2dae-4e2d-b95b-08c754b1add3" (UID: "050ca555-2dae-4e2d-b95b-08c754b1add3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 12 19:56:53.885864 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:56:53.885832 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dmkmr\" (UniqueName: \"kubernetes.io/projected/050ca555-2dae-4e2d-b95b-08c754b1add3-kube-api-access-dmkmr\") on node \"ip-10-0-139-170.ec2.internal\" DevicePath \"\"" Jul 12 19:56:53.885864 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:56:53.885860 2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/050ca555-2dae-4e2d-b95b-08c754b1add3-istio-token\") on node \"ip-10-0-139-170.ec2.internal\" DevicePath \"\"" Jul 12 19:56:53.885864 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:56:53.885869 2565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/050ca555-2dae-4e2d-b95b-08c754b1add3-istio-data\") on node \"ip-10-0-139-170.ec2.internal\" DevicePath \"\"" Jul 12 19:56:53.886090 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:56:53.885877 2565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/050ca555-2dae-4e2d-b95b-08c754b1add3-istio-podinfo\") on node \"ip-10-0-139-170.ec2.internal\" DevicePath \"\"" Jul 12 19:56:53.886090 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:56:53.885912 2565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/050ca555-2dae-4e2d-b95b-08c754b1add3-istio-envoy\") on node \"ip-10-0-139-170.ec2.internal\" DevicePath \"\"" Jul 12 19:56:54.067391 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:56:54.067319 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-515e44-openshift-default-6b9579f57c-th7p8"] Jul 12 19:56:54.070922 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:56:54.070899 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-515e44-openshift-default-6b9579f57c-th7p8"] Jul 12 19:56:55.045456 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:56:55.045419 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:56:55.104239 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:56:55.104204 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:56:55.294529 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:56:55.294496 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:56:55.506730 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:56:55.506694 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:56:55.833005 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:56:55.832910 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:56:55.857501 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:56:55.857471 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="050ca555-2dae-4e2d-b95b-08c754b1add3" path="/var/lib/kubelet/pods/050ca555-2dae-4e2d-b95b-08c754b1add3/volumes" Jul 12 19:56:56.107450 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:56:56.107367 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-b032b1-openshift-default-9dfddd7f9-4c48z"] Jul 12 19:56:56.107792 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:56:56.107674 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="050ca555-2dae-4e2d-b95b-08c754b1add3" containerName="istio-proxy" Jul 12 19:56:56.107792 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:56:56.107684 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="050ca555-2dae-4e2d-b95b-08c754b1add3" containerName="istio-proxy" Jul 12 19:56:56.107792 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:56:56.107691 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d3789221-fbe7-4eae-84fc-79347c889948" containerName="main" Jul 12 19:56:56.107792 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:56:56.107696 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="d3789221-fbe7-4eae-84fc-79347c889948" containerName="main" Jul 12 19:56:56.107792 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:56:56.107706 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bf73efd2-8ee4-4834-9b14-c92649b2bd02" containerName="main" Jul 12 19:56:56.107792 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:56:56.107712 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="bf73efd2-8ee4-4834-9b14-c92649b2bd02" containerName="main" Jul 12 19:56:56.107792 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:56:56.107787 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="d3789221-fbe7-4eae-84fc-79347c889948" containerName="main" Jul 12 19:56:56.108016 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:56:56.107801 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="050ca555-2dae-4e2d-b95b-08c754b1add3" containerName="istio-proxy" Jul 12 19:56:56.108016 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:56:56.107808 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="bf73efd2-8ee4-4834-9b14-c92649b2bd02" containerName="main" Jul 12 19:56:56.108016 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:56:56.107813 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="050ca555-2dae-4e2d-b95b-08c754b1add3" containerName="istio-proxy" Jul 12 19:56:56.111943 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:56:56.111925 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-b032b1-openshift-default-9dfddd7f9-4c48z" Jul 12 19:56:56.114266 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:56:56.114249 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-b032b1-openshift-default-dockercfg-4n29v\"" Jul 12 19:56:56.122307 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:56:56.122281 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-b032b1-openshift-default-9dfddd7f9-4c48z"] Jul 12 19:56:56.127126 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:56:56.127105 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:56:56.206619 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:56:56.206578 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2d87dbb-8509-45ac-bf49-2d7eb67555ae-istio-token\") pod \"e2e-api-a-b032b1-openshift-default-9dfddd7f9-4c48z\" (UID: \"b2d87dbb-8509-45ac-bf49-2d7eb67555ae\") " pod="openshift-ingress/e2e-api-a-b032b1-openshift-default-9dfddd7f9-4c48z" Jul 12 19:56:56.206802 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:56:56.206644 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2d87dbb-8509-45ac-bf49-2d7eb67555ae-workload-socket\") pod \"e2e-api-a-b032b1-openshift-default-9dfddd7f9-4c48z\" (UID: \"b2d87dbb-8509-45ac-bf49-2d7eb67555ae\") " pod="openshift-ingress/e2e-api-a-b032b1-openshift-default-9dfddd7f9-4c48z" Jul 12 19:56:56.206802 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:56:56.206663 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2d87dbb-8509-45ac-bf49-2d7eb67555ae-istio-envoy\") pod \"e2e-api-a-b032b1-openshift-default-9dfddd7f9-4c48z\" (UID: \"b2d87dbb-8509-45ac-bf49-2d7eb67555ae\") " pod="openshift-ingress/e2e-api-a-b032b1-openshift-default-9dfddd7f9-4c48z" Jul 12 19:56:56.206802 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:56:56.206703 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2d87dbb-8509-45ac-bf49-2d7eb67555ae-istio-podinfo\") pod \"e2e-api-a-b032b1-openshift-default-9dfddd7f9-4c48z\" (UID: \"b2d87dbb-8509-45ac-bf49-2d7eb67555ae\") " pod="openshift-ingress/e2e-api-a-b032b1-openshift-default-9dfddd7f9-4c48z" Jul 12 19:56:56.206802 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:56:56.206743 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2d87dbb-8509-45ac-bf49-2d7eb67555ae-istiod-ca-cert\") pod \"e2e-api-a-b032b1-openshift-default-9dfddd7f9-4c48z\" (UID: \"b2d87dbb-8509-45ac-bf49-2d7eb67555ae\") " pod="openshift-ingress/e2e-api-a-b032b1-openshift-default-9dfddd7f9-4c48z" Jul 12 19:56:56.206994 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:56:56.206822 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2d87dbb-8509-45ac-bf49-2d7eb67555ae-workload-certs\") pod \"e2e-api-a-b032b1-openshift-default-9dfddd7f9-4c48z\" (UID: \"b2d87dbb-8509-45ac-bf49-2d7eb67555ae\") " pod="openshift-ingress/e2e-api-a-b032b1-openshift-default-9dfddd7f9-4c48z" Jul 12 19:56:56.206994 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:56:56.206904 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbj4t\" (UniqueName: \"kubernetes.io/projected/b2d87dbb-8509-45ac-bf49-2d7eb67555ae-kube-api-access-tbj4t\") pod \"e2e-api-a-b032b1-openshift-default-9dfddd7f9-4c48z\" (UID: \"b2d87dbb-8509-45ac-bf49-2d7eb67555ae\") " pod="openshift-ingress/e2e-api-a-b032b1-openshift-default-9dfddd7f9-4c48z" Jul 12 19:56:56.206994 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:56:56.206956 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2d87dbb-8509-45ac-bf49-2d7eb67555ae-istio-data\") pod \"e2e-api-a-b032b1-openshift-default-9dfddd7f9-4c48z\" (UID: \"b2d87dbb-8509-45ac-bf49-2d7eb67555ae\") " pod="openshift-ingress/e2e-api-a-b032b1-openshift-default-9dfddd7f9-4c48z" Jul 12 19:56:56.206994 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:56:56.206975 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2d87dbb-8509-45ac-bf49-2d7eb67555ae-credential-socket\") pod \"e2e-api-a-b032b1-openshift-default-9dfddd7f9-4c48z\" (UID: \"b2d87dbb-8509-45ac-bf49-2d7eb67555ae\") " pod="openshift-ingress/e2e-api-a-b032b1-openshift-default-9dfddd7f9-4c48z" Jul 12 19:56:56.212940 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:56:56.212914 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:56:56.307633 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:56:56.307603 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2d87dbb-8509-45ac-bf49-2d7eb67555ae-istiod-ca-cert\") pod \"e2e-api-a-b032b1-openshift-default-9dfddd7f9-4c48z\" (UID: \"b2d87dbb-8509-45ac-bf49-2d7eb67555ae\") " pod="openshift-ingress/e2e-api-a-b032b1-openshift-default-9dfddd7f9-4c48z" Jul 12 19:56:56.307831 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:56:56.307652 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2d87dbb-8509-45ac-bf49-2d7eb67555ae-workload-certs\") pod \"e2e-api-a-b032b1-openshift-default-9dfddd7f9-4c48z\" (UID: \"b2d87dbb-8509-45ac-bf49-2d7eb67555ae\") " pod="openshift-ingress/e2e-api-a-b032b1-openshift-default-9dfddd7f9-4c48z" Jul 12 19:56:56.307831 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:56:56.307770 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbj4t\" (UniqueName: \"kubernetes.io/projected/b2d87dbb-8509-45ac-bf49-2d7eb67555ae-kube-api-access-tbj4t\") pod \"e2e-api-a-b032b1-openshift-default-9dfddd7f9-4c48z\" (UID: \"b2d87dbb-8509-45ac-bf49-2d7eb67555ae\") " pod="openshift-ingress/e2e-api-a-b032b1-openshift-default-9dfddd7f9-4c48z" Jul 12 19:56:56.307831 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:56:56.307813 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2d87dbb-8509-45ac-bf49-2d7eb67555ae-istio-data\") pod \"e2e-api-a-b032b1-openshift-default-9dfddd7f9-4c48z\" (UID: \"b2d87dbb-8509-45ac-bf49-2d7eb67555ae\") " pod="openshift-ingress/e2e-api-a-b032b1-openshift-default-9dfddd7f9-4c48z" Jul 12 19:56:56.308046 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:56:56.307833 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2d87dbb-8509-45ac-bf49-2d7eb67555ae-credential-socket\") pod \"e2e-api-a-b032b1-openshift-default-9dfddd7f9-4c48z\" (UID: \"b2d87dbb-8509-45ac-bf49-2d7eb67555ae\") " pod="openshift-ingress/e2e-api-a-b032b1-openshift-default-9dfddd7f9-4c48z" Jul 12 19:56:56.308046 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:56:56.307864 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2d87dbb-8509-45ac-bf49-2d7eb67555ae-istio-token\") pod \"e2e-api-a-b032b1-openshift-default-9dfddd7f9-4c48z\" (UID: \"b2d87dbb-8509-45ac-bf49-2d7eb67555ae\") " pod="openshift-ingress/e2e-api-a-b032b1-openshift-default-9dfddd7f9-4c48z" Jul 12 19:56:56.308046 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:56:56.307950 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2d87dbb-8509-45ac-bf49-2d7eb67555ae-workload-socket\") pod \"e2e-api-a-b032b1-openshift-default-9dfddd7f9-4c48z\" (UID: \"b2d87dbb-8509-45ac-bf49-2d7eb67555ae\") " pod="openshift-ingress/e2e-api-a-b032b1-openshift-default-9dfddd7f9-4c48z" Jul 12 19:56:56.308046 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:56:56.307976 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2d87dbb-8509-45ac-bf49-2d7eb67555ae-istio-envoy\") pod \"e2e-api-a-b032b1-openshift-default-9dfddd7f9-4c48z\" (UID: \"b2d87dbb-8509-45ac-bf49-2d7eb67555ae\") " pod="openshift-ingress/e2e-api-a-b032b1-openshift-default-9dfddd7f9-4c48z" Jul 12 19:56:56.308046 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:56:56.308005 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2d87dbb-8509-45ac-bf49-2d7eb67555ae-istio-podinfo\") pod \"e2e-api-a-b032b1-openshift-default-9dfddd7f9-4c48z\" (UID: \"b2d87dbb-8509-45ac-bf49-2d7eb67555ae\") " pod="openshift-ingress/e2e-api-a-b032b1-openshift-default-9dfddd7f9-4c48z" Jul 12 19:56:56.308046 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:56:56.308040 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2d87dbb-8509-45ac-bf49-2d7eb67555ae-workload-certs\") pod \"e2e-api-a-b032b1-openshift-default-9dfddd7f9-4c48z\" (UID: \"b2d87dbb-8509-45ac-bf49-2d7eb67555ae\") " pod="openshift-ingress/e2e-api-a-b032b1-openshift-default-9dfddd7f9-4c48z" Jul 12 19:56:56.308354 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:56:56.308122 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2d87dbb-8509-45ac-bf49-2d7eb67555ae-istio-data\") pod \"e2e-api-a-b032b1-openshift-default-9dfddd7f9-4c48z\" (UID: \"b2d87dbb-8509-45ac-bf49-2d7eb67555ae\") " pod="openshift-ingress/e2e-api-a-b032b1-openshift-default-9dfddd7f9-4c48z" Jul 12 19:56:56.308354 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:56:56.308246 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2d87dbb-8509-45ac-bf49-2d7eb67555ae-credential-socket\") pod \"e2e-api-a-b032b1-openshift-default-9dfddd7f9-4c48z\" (UID: \"b2d87dbb-8509-45ac-bf49-2d7eb67555ae\") " pod="openshift-ingress/e2e-api-a-b032b1-openshift-default-9dfddd7f9-4c48z" Jul 12 19:56:56.308354 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:56:56.308328 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2d87dbb-8509-45ac-bf49-2d7eb67555ae-workload-socket\") pod \"e2e-api-a-b032b1-openshift-default-9dfddd7f9-4c48z\" (UID: \"b2d87dbb-8509-45ac-bf49-2d7eb67555ae\") " pod="openshift-ingress/e2e-api-a-b032b1-openshift-default-9dfddd7f9-4c48z" Jul 12 19:56:56.308505 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:56:56.308433 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2d87dbb-8509-45ac-bf49-2d7eb67555ae-istiod-ca-cert\") pod \"e2e-api-a-b032b1-openshift-default-9dfddd7f9-4c48z\" (UID: \"b2d87dbb-8509-45ac-bf49-2d7eb67555ae\") " pod="openshift-ingress/e2e-api-a-b032b1-openshift-default-9dfddd7f9-4c48z" Jul 12 19:56:56.310329 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:56:56.310312 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2d87dbb-8509-45ac-bf49-2d7eb67555ae-istio-envoy\") pod \"e2e-api-a-b032b1-openshift-default-9dfddd7f9-4c48z\" (UID: \"b2d87dbb-8509-45ac-bf49-2d7eb67555ae\") " pod="openshift-ingress/e2e-api-a-b032b1-openshift-default-9dfddd7f9-4c48z" Jul 12 19:56:56.310397 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:56:56.310339 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2d87dbb-8509-45ac-bf49-2d7eb67555ae-istio-podinfo\") pod \"e2e-api-a-b032b1-openshift-default-9dfddd7f9-4c48z\" (UID: \"b2d87dbb-8509-45ac-bf49-2d7eb67555ae\") " pod="openshift-ingress/e2e-api-a-b032b1-openshift-default-9dfddd7f9-4c48z" Jul 12 19:56:56.315168 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:56:56.315149 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2d87dbb-8509-45ac-bf49-2d7eb67555ae-istio-token\") pod \"e2e-api-a-b032b1-openshift-default-9dfddd7f9-4c48z\" (UID: \"b2d87dbb-8509-45ac-bf49-2d7eb67555ae\") " pod="openshift-ingress/e2e-api-a-b032b1-openshift-default-9dfddd7f9-4c48z" Jul 12 19:56:56.315400 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:56:56.315385 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbj4t\" (UniqueName: \"kubernetes.io/projected/b2d87dbb-8509-45ac-bf49-2d7eb67555ae-kube-api-access-tbj4t\") pod \"e2e-api-a-b032b1-openshift-default-9dfddd7f9-4c48z\" (UID: \"b2d87dbb-8509-45ac-bf49-2d7eb67555ae\") " pod="openshift-ingress/e2e-api-a-b032b1-openshift-default-9dfddd7f9-4c48z" Jul 12 19:56:56.422799 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:56:56.422764 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-b032b1-openshift-default-9dfddd7f9-4c48z" Jul 12 19:56:56.549429 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:56:56.549400 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-b032b1-openshift-default-9dfddd7f9-4c48z"] Jul 12 19:56:56.552329 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:56:56.552303 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2d87dbb_8509_45ac_bf49_2d7eb67555ae.slice/crio-f087924495fbbbce862042442f452edaa16c73bd23e5d1e3344af5d1f829bdce WatchSource:0}: Error finding container f087924495fbbbce862042442f452edaa16c73bd23e5d1e3344af5d1f829bdce: Status 404 returned error can't find the container with id f087924495fbbbce862042442f452edaa16c73bd23e5d1e3344af5d1f829bdce Jul 12 19:56:56.554549 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:56:56.554516 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 12 19:56:56.554638 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:56:56.554577 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 12 19:56:56.554638 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:56:56.554606 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 12 19:56:56.767636 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:56:56.767538 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-b032b1-openshift-default-9dfddd7f9-4c48z" event={"ID":"b2d87dbb-8509-45ac-bf49-2d7eb67555ae","Type":"ContainerStarted","Data":"c679d604866dbb48607b118dda7b659b4e2fcfb70f0c54dea0dfe29a92948bed"} Jul 12 19:56:56.767636 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:56:56.767601 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-b032b1-openshift-default-9dfddd7f9-4c48z" event={"ID":"b2d87dbb-8509-45ac-bf49-2d7eb67555ae","Type":"ContainerStarted","Data":"f087924495fbbbce862042442f452edaa16c73bd23e5d1e3344af5d1f829bdce"} Jul 12 19:56:56.786495 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:56:56.786441 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-b032b1-openshift-default-9dfddd7f9-4c48z" podStartSLOduration=0.786422083 podStartE2EDuration="786.422083ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 19:56:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 19:56:56.784254606 +0000 UTC m=+2081.516827121" watchObservedRunningTime="2026-07-12 19:56:56.786422083 +0000 UTC m=+2081.518994572" Jul 12 19:56:57.423455 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:56:57.423420 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-b032b1-openshift-default-9dfddd7f9-4c48z" Jul 12 19:56:57.428478 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:56:57.428453 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-b032b1-openshift-default-9dfddd7f9-4c48z" Jul 12 19:56:57.771653 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:56:57.771567 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-b032b1-openshift-default-9dfddd7f9-4c48z" Jul 12 19:56:57.772610 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:56:57.772586 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-b032b1-openshift-default-9dfddd7f9-4c48z" Jul 12 19:56:58.733163 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:56:58.733129 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:56:58.815503 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:56:58.815471 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:56:58.882594 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:56:58.882561 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:56:59.176447 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:56:59.176405 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:56:59.284633 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:56:59.284589 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:56:59.406105 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:56:59.406070 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:57:01.745525 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:57:01.745489 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:57:01.847275 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:57:01.847237 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:57:03.209478 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:57:03.209432 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:57:03.638450 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:57:03.638356 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:57:03.972492 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:57:03.972451 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:57:04.089613 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:57:04.089585 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:57:06.990992 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:57:06.990954 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:57:07.094046 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:57:07.094012 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:57:07.173259 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:57:07.173219 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:57:07.383798 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:57:07.383721 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:57:07.772112 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:57:07.772067 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:57:07.872932 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:57:07.872894 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:57:10.829316 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:57:10.829284 2565 generic.go:356] "Generic (PLEG): container finished" podID="b2d87dbb-8509-45ac-bf49-2d7eb67555ae" containerID="c679d604866dbb48607b118dda7b659b4e2fcfb70f0c54dea0dfe29a92948bed" exitCode=0 Jul 12 19:57:10.829732 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:57:10.829363 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-b032b1-openshift-default-9dfddd7f9-4c48z" event={"ID":"b2d87dbb-8509-45ac-bf49-2d7eb67555ae","Type":"ContainerDied","Data":"c679d604866dbb48607b118dda7b659b4e2fcfb70f0c54dea0dfe29a92948bed"} Jul 12 19:57:10.829776 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:57:10.829747 2565 scope.go:117] "RemoveContainer" containerID="c679d604866dbb48607b118dda7b659b4e2fcfb70f0c54dea0dfe29a92948bed" Jul 12 19:57:10.830794 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:57:10.830764 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 12 19:57:10.830875 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:57:10.830856 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 12 19:57:10.830936 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:57:10.830921 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 12 19:57:11.462733 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:57:11.462687 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:57:11.835017 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:57:11.834931 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-b032b1-openshift-default-9dfddd7f9-4c48z" event={"ID":"b2d87dbb-8509-45ac-bf49-2d7eb67555ae","Type":"ContainerStarted","Data":"995397c55681d05fb78ccf031ea83d895e378ff20912bbf3be6811ed0e106426"} Jul 12 19:57:12.423096 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:57:12.423060 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-b032b1-openshift-default-9dfddd7f9-4c48z" Jul 12 19:57:12.424140 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:57:12.424117 2565 patch_prober.go:28] interesting pod/e2e-api-a-b032b1-openshift-default-9dfddd7f9-4c48z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body= Jul 12 19:57:12.424202 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:57:12.424175 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-b032b1-openshift-default-9dfddd7f9-4c48z" podUID="b2d87dbb-8509-45ac-bf49-2d7eb67555ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" Jul 12 19:57:12.772642 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:57:12.772498 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-b032b1-openshift-default-9dfddd7f9-4c48z" Jul 12 19:57:13.423645 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:57:13.423611 2565 patch_prober.go:28] interesting pod/e2e-api-a-b032b1-openshift-default-9dfddd7f9-4c48z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body= Jul 12 19:57:13.424063 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:57:13.423689 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-b032b1-openshift-default-9dfddd7f9-4c48z" podUID="b2d87dbb-8509-45ac-bf49-2d7eb67555ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" Jul 12 19:57:14.423937 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:57:14.423906 2565 patch_prober.go:28] interesting pod/e2e-api-a-b032b1-openshift-default-9dfddd7f9-4c48z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body= Jul 12 19:57:14.424314 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:57:14.423964 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-b032b1-openshift-default-9dfddd7f9-4c48z" podUID="b2d87dbb-8509-45ac-bf49-2d7eb67555ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" Jul 12 19:57:15.423806 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:57:15.423770 2565 patch_prober.go:28] interesting pod/e2e-api-a-b032b1-openshift-default-9dfddd7f9-4c48z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body= Jul 12 19:57:15.424011 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:57:15.423827 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-b032b1-openshift-default-9dfddd7f9-4c48z" podUID="b2d87dbb-8509-45ac-bf49-2d7eb67555ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" Jul 12 19:57:15.850363 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:57:15.850283 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k5h9m_8e016c2e-0c3a-49da-83d4-f526738c1cbd/ovn-acl-logging/0.log" Jul 12 19:57:15.853553 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:57:15.853527 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k5h9m_8e016c2e-0c3a-49da-83d4-f526738c1cbd/ovn-acl-logging/0.log" Jul 12 19:57:16.427325 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:57:16.427296 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-b032b1-openshift-default-9dfddd7f9-4c48z" Jul 12 19:57:16.428205 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:57:16.428188 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-b032b1-openshift-default-9dfddd7f9-4c48z" Jul 12 19:57:19.393955 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:57:19.393903 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:57:19.772164 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:57:19.772079 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:57:19.930249 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:57:19.930217 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:57:20.025945 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:57:20.025854 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:57:20.142734 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:57:20.142689 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:57:25.449060 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:57:25.449008 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:57:25.501648 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:57:25.501612 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:57:25.672082 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:57:25.672043 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-b032b1-openshift-default-9dfddd7f9-4c48z"] Jul 12 19:57:25.672260 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:57:25.672232 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-b032b1-openshift-default-9dfddd7f9-4c48z" podUID="b2d87dbb-8509-45ac-bf49-2d7eb67555ae" containerName="istio-proxy" containerID="cri-o://995397c55681d05fb78ccf031ea83d895e378ff20912bbf3be6811ed0e106426" gracePeriod=30 Jul 12 19:57:25.692029 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:57:25.692000 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:57:25.902194 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:57:25.902158 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:57:26.062897 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:57:26.062855 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:57:26.455270 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:57:26.455240 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:57:26.595065 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:57:26.595015 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:57:26.695761 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:57:26.695717 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:57:26.836477 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:57:26.836388 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:57:30.920076 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:57:30.920046 2565 generic.go:356] "Generic (PLEG): container finished" podID="b2d87dbb-8509-45ac-bf49-2d7eb67555ae" containerID="995397c55681d05fb78ccf031ea83d895e378ff20912bbf3be6811ed0e106426" exitCode=0 Jul 12 19:57:30.920443 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:57:30.920124 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-b032b1-openshift-default-9dfddd7f9-4c48z" event={"ID":"b2d87dbb-8509-45ac-bf49-2d7eb67555ae","Type":"ContainerDied","Data":"995397c55681d05fb78ccf031ea83d895e378ff20912bbf3be6811ed0e106426"} Jul 12 19:57:30.920443 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:57:30.920193 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-b032b1-openshift-default-9dfddd7f9-4c48z" event={"ID":"b2d87dbb-8509-45ac-bf49-2d7eb67555ae","Type":"ContainerDied","Data":"f087924495fbbbce862042442f452edaa16c73bd23e5d1e3344af5d1f829bdce"} Jul 12 19:57:30.920443 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:57:30.920217 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f087924495fbbbce862042442f452edaa16c73bd23e5d1e3344af5d1f829bdce" Jul 12 19:57:30.920443 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:57:30.920239 2565 scope.go:117] "RemoveContainer" containerID="c679d604866dbb48607b118dda7b659b4e2fcfb70f0c54dea0dfe29a92948bed" Jul 12 19:57:30.929091 ip-10-0-139-170 kubenswrapper[2565]: E0712 19:57:30.929069 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c679d604866dbb48607b118dda7b659b4e2fcfb70f0c54dea0dfe29a92948bed\": container with ID starting with c679d604866dbb48607b118dda7b659b4e2fcfb70f0c54dea0dfe29a92948bed not found: ID does not exist" containerID="c679d604866dbb48607b118dda7b659b4e2fcfb70f0c54dea0dfe29a92948bed" Jul 12 19:57:30.932710 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:57:30.932667 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-b032b1-openshift-default-9dfddd7f9-4c48z" Jul 12 19:57:31.031790 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:57:31.031754 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2d87dbb-8509-45ac-bf49-2d7eb67555ae-istio-token\") pod \"b2d87dbb-8509-45ac-bf49-2d7eb67555ae\" (UID: \"b2d87dbb-8509-45ac-bf49-2d7eb67555ae\") " Jul 12 19:57:31.032009 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:57:31.031812 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2d87dbb-8509-45ac-bf49-2d7eb67555ae-workload-certs\") pod \"b2d87dbb-8509-45ac-bf49-2d7eb67555ae\" (UID: \"b2d87dbb-8509-45ac-bf49-2d7eb67555ae\") " Jul 12 19:57:31.032009 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:57:31.031851 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2d87dbb-8509-45ac-bf49-2d7eb67555ae-istiod-ca-cert\") pod \"b2d87dbb-8509-45ac-bf49-2d7eb67555ae\" (UID: \"b2d87dbb-8509-45ac-bf49-2d7eb67555ae\") " Jul 12 19:57:31.032009 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:57:31.031879 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2d87dbb-8509-45ac-bf49-2d7eb67555ae-workload-socket\") pod \"b2d87dbb-8509-45ac-bf49-2d7eb67555ae\" (UID: \"b2d87dbb-8509-45ac-bf49-2d7eb67555ae\") " Jul 12 19:57:31.032009 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:57:31.031948 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tbj4t\" (UniqueName: \"kubernetes.io/projected/b2d87dbb-8509-45ac-bf49-2d7eb67555ae-kube-api-access-tbj4t\") pod \"b2d87dbb-8509-45ac-bf49-2d7eb67555ae\" (UID: \"b2d87dbb-8509-45ac-bf49-2d7eb67555ae\") " Jul 12 19:57:31.032239 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:57:31.032010 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2d87dbb-8509-45ac-bf49-2d7eb67555ae-istio-podinfo\") pod \"b2d87dbb-8509-45ac-bf49-2d7eb67555ae\" (UID: \"b2d87dbb-8509-45ac-bf49-2d7eb67555ae\") " Jul 12 19:57:31.032239 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:57:31.032047 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2d87dbb-8509-45ac-bf49-2d7eb67555ae-istio-data\") pod \"b2d87dbb-8509-45ac-bf49-2d7eb67555ae\" (UID: \"b2d87dbb-8509-45ac-bf49-2d7eb67555ae\") " Jul 12 19:57:31.032239 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:57:31.032088 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2d87dbb-8509-45ac-bf49-2d7eb67555ae-istio-envoy\") pod \"b2d87dbb-8509-45ac-bf49-2d7eb67555ae\" (UID: \"b2d87dbb-8509-45ac-bf49-2d7eb67555ae\") " Jul 12 19:57:31.032239 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:57:31.032110 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2d87dbb-8509-45ac-bf49-2d7eb67555ae-credential-socket\") pod \"b2d87dbb-8509-45ac-bf49-2d7eb67555ae\" (UID: \"b2d87dbb-8509-45ac-bf49-2d7eb67555ae\") " Jul 12 19:57:31.032239 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:57:31.032182 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2d87dbb-8509-45ac-bf49-2d7eb67555ae-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "b2d87dbb-8509-45ac-bf49-2d7eb67555ae" (UID: "b2d87dbb-8509-45ac-bf49-2d7eb67555ae"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 19:57:31.032478 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:57:31.032234 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2d87dbb-8509-45ac-bf49-2d7eb67555ae-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "b2d87dbb-8509-45ac-bf49-2d7eb67555ae" (UID: "b2d87dbb-8509-45ac-bf49-2d7eb67555ae"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 19:57:31.032478 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:57:31.032411 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2d87dbb-8509-45ac-bf49-2d7eb67555ae-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "b2d87dbb-8509-45ac-bf49-2d7eb67555ae" (UID: "b2d87dbb-8509-45ac-bf49-2d7eb67555ae"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 19:57:31.032478 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:57:31.032334 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2d87dbb-8509-45ac-bf49-2d7eb67555ae-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "b2d87dbb-8509-45ac-bf49-2d7eb67555ae" (UID: "b2d87dbb-8509-45ac-bf49-2d7eb67555ae"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 12 19:57:31.032652 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:57:31.032502 2565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2d87dbb-8509-45ac-bf49-2d7eb67555ae-credential-socket\") on node \"ip-10-0-139-170.ec2.internal\" DevicePath \"\"" Jul 12 19:57:31.032652 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:57:31.032558 2565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2d87dbb-8509-45ac-bf49-2d7eb67555ae-workload-certs\") on node \"ip-10-0-139-170.ec2.internal\" DevicePath \"\"" Jul 12 19:57:31.032652 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:57:31.032574 2565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2d87dbb-8509-45ac-bf49-2d7eb67555ae-istiod-ca-cert\") on node \"ip-10-0-139-170.ec2.internal\" DevicePath \"\"" Jul 12 19:57:31.032652 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:57:31.032585 2565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2d87dbb-8509-45ac-bf49-2d7eb67555ae-workload-socket\") on node \"ip-10-0-139-170.ec2.internal\" DevicePath \"\"" Jul 12 19:57:31.033227 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:57:31.033199 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2d87dbb-8509-45ac-bf49-2d7eb67555ae-istio-data" (OuterVolumeSpecName: "istio-data") pod "b2d87dbb-8509-45ac-bf49-2d7eb67555ae" (UID: "b2d87dbb-8509-45ac-bf49-2d7eb67555ae"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 19:57:31.034064 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:57:31.034044 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2d87dbb-8509-45ac-bf49-2d7eb67555ae-istio-token" (OuterVolumeSpecName: "istio-token") pod "b2d87dbb-8509-45ac-bf49-2d7eb67555ae" (UID: "b2d87dbb-8509-45ac-bf49-2d7eb67555ae"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 12 19:57:31.034182 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:57:31.034164 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2d87dbb-8509-45ac-bf49-2d7eb67555ae-kube-api-access-tbj4t" (OuterVolumeSpecName: "kube-api-access-tbj4t") pod "b2d87dbb-8509-45ac-bf49-2d7eb67555ae" (UID: "b2d87dbb-8509-45ac-bf49-2d7eb67555ae"). InnerVolumeSpecName "kube-api-access-tbj4t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 12 19:57:31.034619 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:57:31.034597 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2d87dbb-8509-45ac-bf49-2d7eb67555ae-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "b2d87dbb-8509-45ac-bf49-2d7eb67555ae" (UID: "b2d87dbb-8509-45ac-bf49-2d7eb67555ae"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 19:57:31.034619 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:57:31.034605 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b2d87dbb-8509-45ac-bf49-2d7eb67555ae-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "b2d87dbb-8509-45ac-bf49-2d7eb67555ae" (UID: "b2d87dbb-8509-45ac-bf49-2d7eb67555ae"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 12 19:57:31.134108 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:57:31.134062 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tbj4t\" (UniqueName: \"kubernetes.io/projected/b2d87dbb-8509-45ac-bf49-2d7eb67555ae-kube-api-access-tbj4t\") on node \"ip-10-0-139-170.ec2.internal\" DevicePath \"\"" Jul 12 19:57:31.134108 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:57:31.134101 2565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2d87dbb-8509-45ac-bf49-2d7eb67555ae-istio-podinfo\") on node \"ip-10-0-139-170.ec2.internal\" DevicePath \"\"" Jul 12 19:57:31.134108 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:57:31.134110 2565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2d87dbb-8509-45ac-bf49-2d7eb67555ae-istio-data\") on node \"ip-10-0-139-170.ec2.internal\" DevicePath \"\"" Jul 12 19:57:31.134108 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:57:31.134118 2565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2d87dbb-8509-45ac-bf49-2d7eb67555ae-istio-envoy\") on node \"ip-10-0-139-170.ec2.internal\" DevicePath \"\"" Jul 12 19:57:31.134382 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:57:31.134126 2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2d87dbb-8509-45ac-bf49-2d7eb67555ae-istio-token\") on node \"ip-10-0-139-170.ec2.internal\" DevicePath \"\"" Jul 12 19:57:31.925625 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:57:31.925598 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-b032b1-openshift-default-9dfddd7f9-4c48z" Jul 12 19:57:31.943425 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:57:31.943396 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-b032b1-openshift-default-9dfddd7f9-4c48z"] Jul 12 19:57:31.949640 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:57:31.949617 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-b032b1-openshift-default-9dfddd7f9-4c48z"] Jul 12 19:57:32.150168 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:57:32.150134 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:57:32.207328 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:57:32.207297 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:57:32.451591 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:57:32.451510 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:57:32.611789 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:57:32.611751 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:57:32.924645 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:57:32.924600 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:57:33.178697 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:57:33.178607 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-e68438-openshift-default-d95d995-95pg5"] Jul 12 19:57:33.179168 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:57:33.178975 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b2d87dbb-8509-45ac-bf49-2d7eb67555ae" containerName="istio-proxy" Jul 12 19:57:33.179168 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:57:33.178989 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="b2d87dbb-8509-45ac-bf49-2d7eb67555ae" containerName="istio-proxy" Jul 12 19:57:33.179168 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:57:33.179007 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="050ca555-2dae-4e2d-b95b-08c754b1add3" containerName="istio-proxy" Jul 12 19:57:33.179168 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:57:33.179012 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="050ca555-2dae-4e2d-b95b-08c754b1add3" containerName="istio-proxy" Jul 12 19:57:33.179168 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:57:33.179019 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b2d87dbb-8509-45ac-bf49-2d7eb67555ae" containerName="istio-proxy" Jul 12 19:57:33.179168 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:57:33.179024 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="b2d87dbb-8509-45ac-bf49-2d7eb67555ae" containerName="istio-proxy" Jul 12 19:57:33.179168 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:57:33.179083 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="b2d87dbb-8509-45ac-bf49-2d7eb67555ae" containerName="istio-proxy" Jul 12 19:57:33.182556 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:57:33.182536 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-e68438-openshift-default-d95d995-95pg5" Jul 12 19:57:33.185042 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:57:33.185018 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-e68438-openshift-default-dockercfg-twczw\"" Jul 12 19:57:33.194112 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:57:33.194086 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-e68438-openshift-default-d95d995-95pg5"] Jul 12 19:57:33.207812 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:57:33.207763 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:57:33.353397 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:57:33.353353 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e0fbb77-4b94-4dc0-9cb1-60ec64311952-istio-podinfo\") pod \"e2e-shared-a-e68438-openshift-default-d95d995-95pg5\" (UID: \"1e0fbb77-4b94-4dc0-9cb1-60ec64311952\") " pod="openshift-ingress/e2e-shared-a-e68438-openshift-default-d95d995-95pg5" Jul 12 19:57:33.353584 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:57:33.353403 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e0fbb77-4b94-4dc0-9cb1-60ec64311952-istio-token\") pod \"e2e-shared-a-e68438-openshift-default-d95d995-95pg5\" (UID: \"1e0fbb77-4b94-4dc0-9cb1-60ec64311952\") " pod="openshift-ingress/e2e-shared-a-e68438-openshift-default-d95d995-95pg5" Jul 12 19:57:33.353584 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:57:33.353458 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e0fbb77-4b94-4dc0-9cb1-60ec64311952-workload-socket\") pod \"e2e-shared-a-e68438-openshift-default-d95d995-95pg5\" (UID: \"1e0fbb77-4b94-4dc0-9cb1-60ec64311952\") " pod="openshift-ingress/e2e-shared-a-e68438-openshift-default-d95d995-95pg5" Jul 12 19:57:33.353584 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:57:33.353496 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e0fbb77-4b94-4dc0-9cb1-60ec64311952-istiod-ca-cert\") pod \"e2e-shared-a-e68438-openshift-default-d95d995-95pg5\" (UID: \"1e0fbb77-4b94-4dc0-9cb1-60ec64311952\") " pod="openshift-ingress/e2e-shared-a-e68438-openshift-default-d95d995-95pg5" Jul 12 19:57:33.353584 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:57:33.353547 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e0fbb77-4b94-4dc0-9cb1-60ec64311952-istio-envoy\") pod \"e2e-shared-a-e68438-openshift-default-d95d995-95pg5\" (UID: \"1e0fbb77-4b94-4dc0-9cb1-60ec64311952\") " pod="openshift-ingress/e2e-shared-a-e68438-openshift-default-d95d995-95pg5" Jul 12 19:57:33.353584 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:57:33.353574 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e0fbb77-4b94-4dc0-9cb1-60ec64311952-istio-data\") pod \"e2e-shared-a-e68438-openshift-default-d95d995-95pg5\" (UID: \"1e0fbb77-4b94-4dc0-9cb1-60ec64311952\") " pod="openshift-ingress/e2e-shared-a-e68438-openshift-default-d95d995-95pg5" Jul 12 19:57:33.353783 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:57:33.353601 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6kh7f\" (UniqueName: \"kubernetes.io/projected/1e0fbb77-4b94-4dc0-9cb1-60ec64311952-kube-api-access-6kh7f\") pod \"e2e-shared-a-e68438-openshift-default-d95d995-95pg5\" (UID: \"1e0fbb77-4b94-4dc0-9cb1-60ec64311952\") " pod="openshift-ingress/e2e-shared-a-e68438-openshift-default-d95d995-95pg5" Jul 12 19:57:33.353783 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:57:33.353639 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e0fbb77-4b94-4dc0-9cb1-60ec64311952-credential-socket\") pod \"e2e-shared-a-e68438-openshift-default-d95d995-95pg5\" (UID: \"1e0fbb77-4b94-4dc0-9cb1-60ec64311952\") " pod="openshift-ingress/e2e-shared-a-e68438-openshift-default-d95d995-95pg5" Jul 12 19:57:33.353783 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:57:33.353735 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e0fbb77-4b94-4dc0-9cb1-60ec64311952-workload-certs\") pod \"e2e-shared-a-e68438-openshift-default-d95d995-95pg5\" (UID: \"1e0fbb77-4b94-4dc0-9cb1-60ec64311952\") " pod="openshift-ingress/e2e-shared-a-e68438-openshift-default-d95d995-95pg5" Jul 12 19:57:33.393617 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:57:33.393583 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:57:33.455031 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:57:33.454934 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e0fbb77-4b94-4dc0-9cb1-60ec64311952-istio-podinfo\") pod \"e2e-shared-a-e68438-openshift-default-d95d995-95pg5\" (UID: \"1e0fbb77-4b94-4dc0-9cb1-60ec64311952\") " pod="openshift-ingress/e2e-shared-a-e68438-openshift-default-d95d995-95pg5" Jul 12 19:57:33.455031 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:57:33.454979 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e0fbb77-4b94-4dc0-9cb1-60ec64311952-istio-token\") pod \"e2e-shared-a-e68438-openshift-default-d95d995-95pg5\" (UID: \"1e0fbb77-4b94-4dc0-9cb1-60ec64311952\") " pod="openshift-ingress/e2e-shared-a-e68438-openshift-default-d95d995-95pg5" Jul 12 19:57:33.455031 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:57:33.455014 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e0fbb77-4b94-4dc0-9cb1-60ec64311952-workload-socket\") pod \"e2e-shared-a-e68438-openshift-default-d95d995-95pg5\" (UID: \"1e0fbb77-4b94-4dc0-9cb1-60ec64311952\") " pod="openshift-ingress/e2e-shared-a-e68438-openshift-default-d95d995-95pg5" Jul 12 19:57:33.455343 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:57:33.455047 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e0fbb77-4b94-4dc0-9cb1-60ec64311952-istiod-ca-cert\") pod \"e2e-shared-a-e68438-openshift-default-d95d995-95pg5\" (UID: \"1e0fbb77-4b94-4dc0-9cb1-60ec64311952\") " pod="openshift-ingress/e2e-shared-a-e68438-openshift-default-d95d995-95pg5" Jul 12 19:57:33.455343 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:57:33.455082 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e0fbb77-4b94-4dc0-9cb1-60ec64311952-istio-envoy\") pod \"e2e-shared-a-e68438-openshift-default-d95d995-95pg5\" (UID: \"1e0fbb77-4b94-4dc0-9cb1-60ec64311952\") " pod="openshift-ingress/e2e-shared-a-e68438-openshift-default-d95d995-95pg5" Jul 12 19:57:33.455343 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:57:33.455106 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e0fbb77-4b94-4dc0-9cb1-60ec64311952-istio-data\") pod \"e2e-shared-a-e68438-openshift-default-d95d995-95pg5\" (UID: \"1e0fbb77-4b94-4dc0-9cb1-60ec64311952\") " pod="openshift-ingress/e2e-shared-a-e68438-openshift-default-d95d995-95pg5" Jul 12 19:57:33.455343 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:57:33.455130 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6kh7f\" (UniqueName: \"kubernetes.io/projected/1e0fbb77-4b94-4dc0-9cb1-60ec64311952-kube-api-access-6kh7f\") pod \"e2e-shared-a-e68438-openshift-default-d95d995-95pg5\" (UID: \"1e0fbb77-4b94-4dc0-9cb1-60ec64311952\") " pod="openshift-ingress/e2e-shared-a-e68438-openshift-default-d95d995-95pg5" Jul 12 19:57:33.455343 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:57:33.455165 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e0fbb77-4b94-4dc0-9cb1-60ec64311952-credential-socket\") pod \"e2e-shared-a-e68438-openshift-default-d95d995-95pg5\" (UID: \"1e0fbb77-4b94-4dc0-9cb1-60ec64311952\") " pod="openshift-ingress/e2e-shared-a-e68438-openshift-default-d95d995-95pg5" Jul 12 19:57:33.455343 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:57:33.455230 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e0fbb77-4b94-4dc0-9cb1-60ec64311952-workload-certs\") pod \"e2e-shared-a-e68438-openshift-default-d95d995-95pg5\" (UID: \"1e0fbb77-4b94-4dc0-9cb1-60ec64311952\") " pod="openshift-ingress/e2e-shared-a-e68438-openshift-default-d95d995-95pg5" Jul 12 19:57:33.455649 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:57:33.455474 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e0fbb77-4b94-4dc0-9cb1-60ec64311952-workload-socket\") pod \"e2e-shared-a-e68438-openshift-default-d95d995-95pg5\" (UID: \"1e0fbb77-4b94-4dc0-9cb1-60ec64311952\") " pod="openshift-ingress/e2e-shared-a-e68438-openshift-default-d95d995-95pg5" Jul 12 19:57:33.455649 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:57:33.455569 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e0fbb77-4b94-4dc0-9cb1-60ec64311952-istio-data\") pod \"e2e-shared-a-e68438-openshift-default-d95d995-95pg5\" (UID: \"1e0fbb77-4b94-4dc0-9cb1-60ec64311952\") " pod="openshift-ingress/e2e-shared-a-e68438-openshift-default-d95d995-95pg5" Jul 12 19:57:33.455649 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:57:33.455632 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e0fbb77-4b94-4dc0-9cb1-60ec64311952-workload-certs\") pod \"e2e-shared-a-e68438-openshift-default-d95d995-95pg5\" (UID: \"1e0fbb77-4b94-4dc0-9cb1-60ec64311952\") " pod="openshift-ingress/e2e-shared-a-e68438-openshift-default-d95d995-95pg5" Jul 12 19:57:33.455765 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:57:33.455667 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e0fbb77-4b94-4dc0-9cb1-60ec64311952-credential-socket\") pod \"e2e-shared-a-e68438-openshift-default-d95d995-95pg5\" (UID: \"1e0fbb77-4b94-4dc0-9cb1-60ec64311952\") " pod="openshift-ingress/e2e-shared-a-e68438-openshift-default-d95d995-95pg5" Jul 12 19:57:33.456116 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:57:33.456094 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e0fbb77-4b94-4dc0-9cb1-60ec64311952-istiod-ca-cert\") pod \"e2e-shared-a-e68438-openshift-default-d95d995-95pg5\" (UID: \"1e0fbb77-4b94-4dc0-9cb1-60ec64311952\") " pod="openshift-ingress/e2e-shared-a-e68438-openshift-default-d95d995-95pg5" Jul 12 19:57:33.457366 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:57:33.457348 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e0fbb77-4b94-4dc0-9cb1-60ec64311952-istio-envoy\") pod \"e2e-shared-a-e68438-openshift-default-d95d995-95pg5\" (UID: \"1e0fbb77-4b94-4dc0-9cb1-60ec64311952\") " pod="openshift-ingress/e2e-shared-a-e68438-openshift-default-d95d995-95pg5" Jul 12 19:57:33.457654 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:57:33.457635 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e0fbb77-4b94-4dc0-9cb1-60ec64311952-istio-podinfo\") pod \"e2e-shared-a-e68438-openshift-default-d95d995-95pg5\" (UID: \"1e0fbb77-4b94-4dc0-9cb1-60ec64311952\") " pod="openshift-ingress/e2e-shared-a-e68438-openshift-default-d95d995-95pg5" Jul 12 19:57:33.462576 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:57:33.462552 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6kh7f\" (UniqueName: \"kubernetes.io/projected/1e0fbb77-4b94-4dc0-9cb1-60ec64311952-kube-api-access-6kh7f\") pod \"e2e-shared-a-e68438-openshift-default-d95d995-95pg5\" (UID: \"1e0fbb77-4b94-4dc0-9cb1-60ec64311952\") " pod="openshift-ingress/e2e-shared-a-e68438-openshift-default-d95d995-95pg5" Jul 12 19:57:33.462682 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:57:33.462607 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e0fbb77-4b94-4dc0-9cb1-60ec64311952-istio-token\") pod \"e2e-shared-a-e68438-openshift-default-d95d995-95pg5\" (UID: \"1e0fbb77-4b94-4dc0-9cb1-60ec64311952\") " pod="openshift-ingress/e2e-shared-a-e68438-openshift-default-d95d995-95pg5" Jul 12 19:57:33.493094 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:57:33.493067 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-e68438-openshift-default-d95d995-95pg5" Jul 12 19:57:33.620630 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:57:33.620599 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-e68438-openshift-default-d95d995-95pg5"] Jul 12 19:57:33.623367 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:57:33.623340 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e0fbb77_4b94_4dc0_9cb1_60ec64311952.slice/crio-5923e9754b57be4f2cf88c6ae5b7ad2363fde8f79f5dc90bb7a1e361fae43f6f WatchSource:0}: Error finding container 5923e9754b57be4f2cf88c6ae5b7ad2363fde8f79f5dc90bb7a1e361fae43f6f: Status 404 returned error can't find the container with id 5923e9754b57be4f2cf88c6ae5b7ad2363fde8f79f5dc90bb7a1e361fae43f6f Jul 12 19:57:33.625506 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:57:33.625474 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 12 19:57:33.625570 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:57:33.625541 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 12 19:57:33.625603 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:57:33.625569 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 12 19:57:33.857961 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:57:33.857840 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b2d87dbb-8509-45ac-bf49-2d7eb67555ae" path="/var/lib/kubelet/pods/b2d87dbb-8509-45ac-bf49-2d7eb67555ae/volumes" Jul 12 19:57:33.934784 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:57:33.934744 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-e68438-openshift-default-d95d995-95pg5" event={"ID":"1e0fbb77-4b94-4dc0-9cb1-60ec64311952","Type":"ContainerStarted","Data":"8faf52477a5aa92e322f8cec9fac6834497d87684719560c25c30d91af2c43b0"} Jul 12 19:57:33.934999 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:57:33.934807 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-e68438-openshift-default-d95d995-95pg5" event={"ID":"1e0fbb77-4b94-4dc0-9cb1-60ec64311952","Type":"ContainerStarted","Data":"5923e9754b57be4f2cf88c6ae5b7ad2363fde8f79f5dc90bb7a1e361fae43f6f"} Jul 12 19:57:33.952973 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:57:33.952927 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-e68438-openshift-default-d95d995-95pg5" podStartSLOduration=0.952913088 podStartE2EDuration="952.913088ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 19:57:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 19:57:33.951318934 +0000 UTC m=+2118.683891427" watchObservedRunningTime="2026-07-12 19:57:33.952913088 +0000 UTC m=+2118.685485578" Jul 12 19:57:34.493672 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:57:34.493635 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-e68438-openshift-default-d95d995-95pg5" Jul 12 19:57:34.498450 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:57:34.498428 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-e68438-openshift-default-d95d995-95pg5" Jul 12 19:57:34.939363 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:57:34.939330 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-e68438-openshift-default-d95d995-95pg5" Jul 12 19:57:34.940195 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:57:34.940176 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-e68438-openshift-default-d95d995-95pg5" Jul 12 19:57:35.906803 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:57:35.906764 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:57:35.978929 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:57:35.978898 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:57:36.037730 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:57:36.037702 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:57:36.546168 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:57:36.546140 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:57:36.652345 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:57:36.652313 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:57:36.752241 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:57:36.752194 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:57:40.390107 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:57:40.390073 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:57:40.691041 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:57:40.690951 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:57:40.985671 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:57:40.985592 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:57:41.186698 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:57:41.186654 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:57:44.184462 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:57:44.184422 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:57:44.246499 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:57:44.246425 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:57:44.314100 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:57:44.314056 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:57:44.740802 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:57:44.740762 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:57:44.839694 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:57:44.839650 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:57:44.945673 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:57:44.945637 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:57:48.653215 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:57:48.653178 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:57:49.011113 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:57:49.011011 2565 generic.go:356] "Generic (PLEG): container finished" podID="1e0fbb77-4b94-4dc0-9cb1-60ec64311952" containerID="8faf52477a5aa92e322f8cec9fac6834497d87684719560c25c30d91af2c43b0" exitCode=0 Jul 12 19:57:49.011113 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:57:49.011087 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-e68438-openshift-default-d95d995-95pg5" event={"ID":"1e0fbb77-4b94-4dc0-9cb1-60ec64311952","Type":"ContainerDied","Data":"8faf52477a5aa92e322f8cec9fac6834497d87684719560c25c30d91af2c43b0"} Jul 12 19:57:49.011481 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:57:49.011468 2565 scope.go:117] "RemoveContainer" containerID="8faf52477a5aa92e322f8cec9fac6834497d87684719560c25c30d91af2c43b0" Jul 12 19:57:49.012468 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:57:49.012438 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 12 19:57:49.012584 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:57:49.012521 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 12 19:57:49.012584 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:57:49.012566 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 12 19:57:49.939356 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:57:49.939326 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-e68438-openshift-default-d95d995-95pg5" Jul 12 19:57:50.016898 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:57:50.016851 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-e68438-openshift-default-d95d995-95pg5" event={"ID":"1e0fbb77-4b94-4dc0-9cb1-60ec64311952","Type":"ContainerStarted","Data":"d51c41bda76230c0458c14ced26824f00a2b93066d1a8469f022f60a20ed25a1"} Jul 12 19:57:50.493966 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:57:50.493930 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-e68438-openshift-default-d95d995-95pg5" Jul 12 19:57:50.495063 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:57:50.495035 2565 patch_prober.go:28] interesting pod/e2e-shared-a-e68438-openshift-default-d95d995-95pg5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Jul 12 19:57:50.495196 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:57:50.495120 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-e68438-openshift-default-d95d995-95pg5" podUID="1e0fbb77-4b94-4dc0-9cb1-60ec64311952" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Jul 12 19:57:51.494055 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:57:51.494015 2565 patch_prober.go:28] interesting pod/e2e-shared-a-e68438-openshift-default-d95d995-95pg5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Jul 12 19:57:51.494513 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:57:51.494075 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-e68438-openshift-default-d95d995-95pg5" podUID="1e0fbb77-4b94-4dc0-9cb1-60ec64311952" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Jul 12 19:57:52.494048 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:57:52.494010 2565 patch_prober.go:28] interesting pod/e2e-shared-a-e68438-openshift-default-d95d995-95pg5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Jul 12 19:57:52.494222 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:57:52.494064 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-e68438-openshift-default-d95d995-95pg5" podUID="1e0fbb77-4b94-4dc0-9cb1-60ec64311952" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Jul 12 19:57:53.493771 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:57:53.493735 2565 patch_prober.go:28] interesting pod/e2e-shared-a-e68438-openshift-default-d95d995-95pg5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Jul 12 19:57:53.493988 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:57:53.493793 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-e68438-openshift-default-d95d995-95pg5" podUID="1e0fbb77-4b94-4dc0-9cb1-60ec64311952" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Jul 12 19:57:54.498133 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:57:54.498102 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-e68438-openshift-default-d95d995-95pg5" Jul 12 19:57:54.498597 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:57:54.498486 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-e68438-openshift-default-d95d995-95pg5" Jul 12 19:57:54.499154 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:57:54.499138 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-e68438-openshift-default-d95d995-95pg5" Jul 12 19:57:58.029614 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:57:58.029580 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:57:58.129758 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:57:58.129722 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:58:01.515207 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:58:01.515175 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-e68438/auth-test-model-e68438-kserve-8478c59d4c-6sgr8"] Jul 12 19:58:01.515583 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:58:01.515542 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="b2d87dbb-8509-45ac-bf49-2d7eb67555ae" containerName="istio-proxy" Jul 12 19:58:01.518545 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:58:01.518525 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-e68438/auth-test-model-e68438-kserve-8478c59d4c-6sgr8" Jul 12 19:58:01.521852 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:58:01.521807 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-e68438\"/\"default-dockercfg-4779w\"" Jul 12 19:58:01.522029 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:58:01.521960 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-e68438\"/\"auth-test-model-e68438-kserve-self-signed-certs\"" Jul 12 19:58:01.522029 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:58:01.521815 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-e68438\"/\"openshift-service-ca.crt\"" Jul 12 19:58:01.522342 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:58:01.522326 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-e68438\"/\"kube-root-ca.crt\"" Jul 12 19:58:01.527547 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:58:01.527523 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-e68438/auth-test-model-e68438-kserve-8478c59d4c-6sgr8"] Jul 12 19:58:01.566909 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:58:01.566847 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:58:01.607868 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:58:01.607822 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62771a54-22e2-4344-882a-27adb79d46c1-tmp-dir\") pod \"auth-test-model-e68438-kserve-8478c59d4c-6sgr8\" (UID: \"62771a54-22e2-4344-882a-27adb79d46c1\") " pod="ai-tenant-e2e-shared-a-e68438/auth-test-model-e68438-kserve-8478c59d4c-6sgr8" Jul 12 19:58:01.608071 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:58:01.607874 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/62771a54-22e2-4344-882a-27adb79d46c1-model-cache\") pod \"auth-test-model-e68438-kserve-8478c59d4c-6sgr8\" (UID: \"62771a54-22e2-4344-882a-27adb79d46c1\") " pod="ai-tenant-e2e-shared-a-e68438/auth-test-model-e68438-kserve-8478c59d4c-6sgr8" Jul 12 19:58:01.608071 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:58:01.607992 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/62771a54-22e2-4344-882a-27adb79d46c1-dshm\") pod \"auth-test-model-e68438-kserve-8478c59d4c-6sgr8\" (UID: \"62771a54-22e2-4344-882a-27adb79d46c1\") " pod="ai-tenant-e2e-shared-a-e68438/auth-test-model-e68438-kserve-8478c59d4c-6sgr8" Jul 12 19:58:01.608071 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:58:01.608050 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/62771a54-22e2-4344-882a-27adb79d46c1-home\") pod \"auth-test-model-e68438-kserve-8478c59d4c-6sgr8\" (UID: \"62771a54-22e2-4344-882a-27adb79d46c1\") " pod="ai-tenant-e2e-shared-a-e68438/auth-test-model-e68438-kserve-8478c59d4c-6sgr8" Jul 12 19:58:01.608180 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:58:01.608076 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-thjqh\" (UniqueName: \"kubernetes.io/projected/62771a54-22e2-4344-882a-27adb79d46c1-kube-api-access-thjqh\") pod \"auth-test-model-e68438-kserve-8478c59d4c-6sgr8\" (UID: \"62771a54-22e2-4344-882a-27adb79d46c1\") " pod="ai-tenant-e2e-shared-a-e68438/auth-test-model-e68438-kserve-8478c59d4c-6sgr8" Jul 12 19:58:01.608180 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:58:01.608106 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62771a54-22e2-4344-882a-27adb79d46c1-tls-certs\") pod \"auth-test-model-e68438-kserve-8478c59d4c-6sgr8\" (UID: \"62771a54-22e2-4344-882a-27adb79d46c1\") " pod="ai-tenant-e2e-shared-a-e68438/auth-test-model-e68438-kserve-8478c59d4c-6sgr8" Jul 12 19:58:01.709229 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:58:01.709190 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62771a54-22e2-4344-882a-27adb79d46c1-tmp-dir\") pod \"auth-test-model-e68438-kserve-8478c59d4c-6sgr8\" (UID: \"62771a54-22e2-4344-882a-27adb79d46c1\") " pod="ai-tenant-e2e-shared-a-e68438/auth-test-model-e68438-kserve-8478c59d4c-6sgr8" Jul 12 19:58:01.709229 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:58:01.709230 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/62771a54-22e2-4344-882a-27adb79d46c1-model-cache\") pod \"auth-test-model-e68438-kserve-8478c59d4c-6sgr8\" (UID: \"62771a54-22e2-4344-882a-27adb79d46c1\") " pod="ai-tenant-e2e-shared-a-e68438/auth-test-model-e68438-kserve-8478c59d4c-6sgr8" Jul 12 19:58:01.709469 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:58:01.709259 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/62771a54-22e2-4344-882a-27adb79d46c1-dshm\") pod \"auth-test-model-e68438-kserve-8478c59d4c-6sgr8\" (UID: \"62771a54-22e2-4344-882a-27adb79d46c1\") " pod="ai-tenant-e2e-shared-a-e68438/auth-test-model-e68438-kserve-8478c59d4c-6sgr8" Jul 12 19:58:01.709469 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:58:01.709308 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/62771a54-22e2-4344-882a-27adb79d46c1-home\") pod \"auth-test-model-e68438-kserve-8478c59d4c-6sgr8\" (UID: \"62771a54-22e2-4344-882a-27adb79d46c1\") " pod="ai-tenant-e2e-shared-a-e68438/auth-test-model-e68438-kserve-8478c59d4c-6sgr8" Jul 12 19:58:01.709469 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:58:01.709331 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thjqh\" (UniqueName: \"kubernetes.io/projected/62771a54-22e2-4344-882a-27adb79d46c1-kube-api-access-thjqh\") pod \"auth-test-model-e68438-kserve-8478c59d4c-6sgr8\" (UID: \"62771a54-22e2-4344-882a-27adb79d46c1\") " pod="ai-tenant-e2e-shared-a-e68438/auth-test-model-e68438-kserve-8478c59d4c-6sgr8" Jul 12 19:58:01.709469 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:58:01.709404 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62771a54-22e2-4344-882a-27adb79d46c1-tls-certs\") pod \"auth-test-model-e68438-kserve-8478c59d4c-6sgr8\" (UID: \"62771a54-22e2-4344-882a-27adb79d46c1\") " pod="ai-tenant-e2e-shared-a-e68438/auth-test-model-e68438-kserve-8478c59d4c-6sgr8" Jul 12 19:58:01.709669 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:58:01.709648 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62771a54-22e2-4344-882a-27adb79d46c1-tmp-dir\") pod \"auth-test-model-e68438-kserve-8478c59d4c-6sgr8\" (UID: \"62771a54-22e2-4344-882a-27adb79d46c1\") " pod="ai-tenant-e2e-shared-a-e68438/auth-test-model-e68438-kserve-8478c59d4c-6sgr8" Jul 12 19:58:01.709723 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:58:01.709694 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/62771a54-22e2-4344-882a-27adb79d46c1-model-cache\") pod \"auth-test-model-e68438-kserve-8478c59d4c-6sgr8\" (UID: \"62771a54-22e2-4344-882a-27adb79d46c1\") " pod="ai-tenant-e2e-shared-a-e68438/auth-test-model-e68438-kserve-8478c59d4c-6sgr8" Jul 12 19:58:01.709812 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:58:01.709790 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/62771a54-22e2-4344-882a-27adb79d46c1-home\") pod \"auth-test-model-e68438-kserve-8478c59d4c-6sgr8\" (UID: \"62771a54-22e2-4344-882a-27adb79d46c1\") " pod="ai-tenant-e2e-shared-a-e68438/auth-test-model-e68438-kserve-8478c59d4c-6sgr8" Jul 12 19:58:01.712019 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:58:01.711990 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/62771a54-22e2-4344-882a-27adb79d46c1-dshm\") pod \"auth-test-model-e68438-kserve-8478c59d4c-6sgr8\" (UID: \"62771a54-22e2-4344-882a-27adb79d46c1\") " pod="ai-tenant-e2e-shared-a-e68438/auth-test-model-e68438-kserve-8478c59d4c-6sgr8" Jul 12 19:58:01.712628 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:58:01.712588 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62771a54-22e2-4344-882a-27adb79d46c1-tls-certs\") pod \"auth-test-model-e68438-kserve-8478c59d4c-6sgr8\" (UID: \"62771a54-22e2-4344-882a-27adb79d46c1\") " pod="ai-tenant-e2e-shared-a-e68438/auth-test-model-e68438-kserve-8478c59d4c-6sgr8" Jul 12 19:58:01.718095 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:58:01.718070 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-thjqh\" (UniqueName: \"kubernetes.io/projected/62771a54-22e2-4344-882a-27adb79d46c1-kube-api-access-thjqh\") pod \"auth-test-model-e68438-kserve-8478c59d4c-6sgr8\" (UID: \"62771a54-22e2-4344-882a-27adb79d46c1\") " pod="ai-tenant-e2e-shared-a-e68438/auth-test-model-e68438-kserve-8478c59d4c-6sgr8" Jul 12 19:58:01.830042 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:58:01.829953 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-e68438/auth-test-model-e68438-kserve-8478c59d4c-6sgr8" Jul 12 19:58:01.962196 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:58:01.962159 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-e68438/auth-test-model-e68438-kserve-8478c59d4c-6sgr8"] Jul 12 19:58:01.966112 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:58:01.966078 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod62771a54_22e2_4344_882a_27adb79d46c1.slice/crio-b63d5c46477aebeec3b61ffda088490ee563e300f08cbfbed68c754c06e75049 WatchSource:0}: Error finding container b63d5c46477aebeec3b61ffda088490ee563e300f08cbfbed68c754c06e75049: Status 404 returned error can't find the container with id b63d5c46477aebeec3b61ffda088490ee563e300f08cbfbed68c754c06e75049 Jul 12 19:58:02.069633 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:58:02.069592 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-e68438/auth-test-model-e68438-kserve-8478c59d4c-6sgr8" event={"ID":"62771a54-22e2-4344-882a-27adb79d46c1","Type":"ContainerStarted","Data":"47e180a2b3d6849b74c4cf48133db4cc853ec6b8820eeb61c1673abbcb4b4d76"} Jul 12 19:58:02.069633 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:58:02.069638 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-e68438/auth-test-model-e68438-kserve-8478c59d4c-6sgr8" event={"ID":"62771a54-22e2-4344-882a-27adb79d46c1","Type":"ContainerStarted","Data":"b63d5c46477aebeec3b61ffda088490ee563e300f08cbfbed68c754c06e75049"} Jul 12 19:58:02.086670 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:58:02.086566 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-e68438/auth-test-model-e68438-kserve-8478c59d4c-6sgr8" podStartSLOduration=1.086550274 podStartE2EDuration="1.086550274s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 19:58:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 19:58:02.08466186 +0000 UTC m=+2146.817234349" watchObservedRunningTime="2026-07-12 19:58:02.086550274 +0000 UTC m=+2146.819122763" Jul 12 19:58:05.592189 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:58:05.592155 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:58:05.693440 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:58:05.693393 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:58:08.729630 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:58:08.729546 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:58:11.830996 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:58:11.830951 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-e68438/auth-test-model-e68438-kserve-8478c59d4c-6sgr8" Jul 12 19:58:11.831399 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:58:11.831030 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-e68438/auth-test-model-e68438-kserve-8478c59d4c-6sgr8" Jul 12 19:58:11.843390 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:58:11.843361 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-e68438/auth-test-model-e68438-kserve-8478c59d4c-6sgr8" Jul 12 19:58:12.131043 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:58:12.130964 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-e68438/auth-test-model-e68438-kserve-8478c59d4c-6sgr8" Jul 12 19:58:20.132573 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:58:20.132545 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-5865e5/auth-test-model-5865e5-kserve-76c8db95ff-6tcbr"] Jul 12 19:58:20.136479 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:58:20.136462 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-5865e5/auth-test-model-5865e5-kserve-76c8db95ff-6tcbr" Jul 12 19:58:20.138826 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:58:20.138798 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-5865e5\"/\"kube-root-ca.crt\"" Jul 12 19:58:20.139752 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:58:20.139731 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-5865e5\"/\"openshift-service-ca.crt\"" Jul 12 19:58:20.139847 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:58:20.139777 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-5865e5\"/\"default-dockercfg-t2pbn\"" Jul 12 19:58:20.140023 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:58:20.140002 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-5865e5\"/\"auth-test-model-5865e5-kserve-self-signed-certs\"" Jul 12 19:58:20.145211 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:58:20.145187 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-5865e5/auth-test-model-5865e5-kserve-76c8db95ff-6tcbr"] Jul 12 19:58:20.165932 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:58:20.165901 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 19:58:20.275147 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:58:20.275116 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/51d9cc0e-7fe3-4764-8937-b8bbeae60e06-dshm\") pod \"auth-test-model-5865e5-kserve-76c8db95ff-6tcbr\" (UID: \"51d9cc0e-7fe3-4764-8937-b8bbeae60e06\") " pod="ai-tenant-e2e-shared-b-5865e5/auth-test-model-5865e5-kserve-76c8db95ff-6tcbr" Jul 12 19:58:20.275333 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:58:20.275197 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/51d9cc0e-7fe3-4764-8937-b8bbeae60e06-model-cache\") pod \"auth-test-model-5865e5-kserve-76c8db95ff-6tcbr\" (UID: \"51d9cc0e-7fe3-4764-8937-b8bbeae60e06\") " pod="ai-tenant-e2e-shared-b-5865e5/auth-test-model-5865e5-kserve-76c8db95ff-6tcbr" Jul 12 19:58:20.275333 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:58:20.275241 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/51d9cc0e-7fe3-4764-8937-b8bbeae60e06-tmp-dir\") pod \"auth-test-model-5865e5-kserve-76c8db95ff-6tcbr\" (UID: \"51d9cc0e-7fe3-4764-8937-b8bbeae60e06\") " pod="ai-tenant-e2e-shared-b-5865e5/auth-test-model-5865e5-kserve-76c8db95ff-6tcbr" Jul 12 19:58:20.275333 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:58:20.275296 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5c5b\" (UniqueName: \"kubernetes.io/projected/51d9cc0e-7fe3-4764-8937-b8bbeae60e06-kube-api-access-r5c5b\") pod \"auth-test-model-5865e5-kserve-76c8db95ff-6tcbr\" (UID: \"51d9cc0e-7fe3-4764-8937-b8bbeae60e06\") " pod="ai-tenant-e2e-shared-b-5865e5/auth-test-model-5865e5-kserve-76c8db95ff-6tcbr" Jul 12 19:58:20.275474 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:58:20.275354 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/51d9cc0e-7fe3-4764-8937-b8bbeae60e06-home\") pod \"auth-test-model-5865e5-kserve-76c8db95ff-6tcbr\" (UID: \"51d9cc0e-7fe3-4764-8937-b8bbeae60e06\") " pod="ai-tenant-e2e-shared-b-5865e5/auth-test-model-5865e5-kserve-76c8db95ff-6tcbr" Jul 12 19:58:20.275474 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:58:20.275378 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/51d9cc0e-7fe3-4764-8937-b8bbeae60e06-tls-certs\") pod \"auth-test-model-5865e5-kserve-76c8db95ff-6tcbr\" (UID: \"51d9cc0e-7fe3-4764-8937-b8bbeae60e06\") " pod="ai-tenant-e2e-shared-b-5865e5/auth-test-model-5865e5-kserve-76c8db95ff-6tcbr" Jul 12 19:58:20.376346 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:58:20.376307 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/51d9cc0e-7fe3-4764-8937-b8bbeae60e06-home\") pod \"auth-test-model-5865e5-kserve-76c8db95ff-6tcbr\" (UID: \"51d9cc0e-7fe3-4764-8937-b8bbeae60e06\") " pod="ai-tenant-e2e-shared-b-5865e5/auth-test-model-5865e5-kserve-76c8db95ff-6tcbr" Jul 12 19:58:20.376346 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:58:20.376351 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/51d9cc0e-7fe3-4764-8937-b8bbeae60e06-tls-certs\") pod \"auth-test-model-5865e5-kserve-76c8db95ff-6tcbr\" (UID: \"51d9cc0e-7fe3-4764-8937-b8bbeae60e06\") " pod="ai-tenant-e2e-shared-b-5865e5/auth-test-model-5865e5-kserve-76c8db95ff-6tcbr" Jul 12 19:58:20.376578 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:58:20.376488 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/51d9cc0e-7fe3-4764-8937-b8bbeae60e06-dshm\") pod \"auth-test-model-5865e5-kserve-76c8db95ff-6tcbr\" (UID: \"51d9cc0e-7fe3-4764-8937-b8bbeae60e06\") " pod="ai-tenant-e2e-shared-b-5865e5/auth-test-model-5865e5-kserve-76c8db95ff-6tcbr" Jul 12 19:58:20.376646 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:58:20.376629 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/51d9cc0e-7fe3-4764-8937-b8bbeae60e06-model-cache\") pod \"auth-test-model-5865e5-kserve-76c8db95ff-6tcbr\" (UID: \"51d9cc0e-7fe3-4764-8937-b8bbeae60e06\") " pod="ai-tenant-e2e-shared-b-5865e5/auth-test-model-5865e5-kserve-76c8db95ff-6tcbr" Jul 12 19:58:20.376686 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:58:20.376661 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/51d9cc0e-7fe3-4764-8937-b8bbeae60e06-tmp-dir\") pod \"auth-test-model-5865e5-kserve-76c8db95ff-6tcbr\" (UID: \"51d9cc0e-7fe3-4764-8937-b8bbeae60e06\") " pod="ai-tenant-e2e-shared-b-5865e5/auth-test-model-5865e5-kserve-76c8db95ff-6tcbr" Jul 12 19:58:20.376744 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:58:20.376691 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5c5b\" (UniqueName: \"kubernetes.io/projected/51d9cc0e-7fe3-4764-8937-b8bbeae60e06-kube-api-access-r5c5b\") pod \"auth-test-model-5865e5-kserve-76c8db95ff-6tcbr\" (UID: \"51d9cc0e-7fe3-4764-8937-b8bbeae60e06\") " pod="ai-tenant-e2e-shared-b-5865e5/auth-test-model-5865e5-kserve-76c8db95ff-6tcbr" Jul 12 19:58:20.376804 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:58:20.376766 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/51d9cc0e-7fe3-4764-8937-b8bbeae60e06-home\") pod \"auth-test-model-5865e5-kserve-76c8db95ff-6tcbr\" (UID: \"51d9cc0e-7fe3-4764-8937-b8bbeae60e06\") " pod="ai-tenant-e2e-shared-b-5865e5/auth-test-model-5865e5-kserve-76c8db95ff-6tcbr" Jul 12 19:58:20.376980 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:58:20.376957 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/51d9cc0e-7fe3-4764-8937-b8bbeae60e06-tmp-dir\") pod \"auth-test-model-5865e5-kserve-76c8db95ff-6tcbr\" (UID: \"51d9cc0e-7fe3-4764-8937-b8bbeae60e06\") " pod="ai-tenant-e2e-shared-b-5865e5/auth-test-model-5865e5-kserve-76c8db95ff-6tcbr" Jul 12 19:58:20.377070 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:58:20.377049 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/51d9cc0e-7fe3-4764-8937-b8bbeae60e06-model-cache\") pod \"auth-test-model-5865e5-kserve-76c8db95ff-6tcbr\" (UID: \"51d9cc0e-7fe3-4764-8937-b8bbeae60e06\") " pod="ai-tenant-e2e-shared-b-5865e5/auth-test-model-5865e5-kserve-76c8db95ff-6tcbr" Jul 12 19:58:20.379164 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:58:20.379137 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/51d9cc0e-7fe3-4764-8937-b8bbeae60e06-dshm\") pod \"auth-test-model-5865e5-kserve-76c8db95ff-6tcbr\" (UID: \"51d9cc0e-7fe3-4764-8937-b8bbeae60e06\") " pod="ai-tenant-e2e-shared-b-5865e5/auth-test-model-5865e5-kserve-76c8db95ff-6tcbr" Jul 12 19:58:20.379395 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:58:20.379381 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/51d9cc0e-7fe3-4764-8937-b8bbeae60e06-tls-certs\") pod \"auth-test-model-5865e5-kserve-76c8db95ff-6tcbr\" (UID: \"51d9cc0e-7fe3-4764-8937-b8bbeae60e06\") " pod="ai-tenant-e2e-shared-b-5865e5/auth-test-model-5865e5-kserve-76c8db95ff-6tcbr" Jul 12 19:58:20.384050 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:58:20.383997 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5c5b\" (UniqueName: \"kubernetes.io/projected/51d9cc0e-7fe3-4764-8937-b8bbeae60e06-kube-api-access-r5c5b\") pod \"auth-test-model-5865e5-kserve-76c8db95ff-6tcbr\" (UID: \"51d9cc0e-7fe3-4764-8937-b8bbeae60e06\") " pod="ai-tenant-e2e-shared-b-5865e5/auth-test-model-5865e5-kserve-76c8db95ff-6tcbr" Jul 12 19:58:20.447098 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:58:20.447050 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-5865e5/auth-test-model-5865e5-kserve-76c8db95ff-6tcbr" Jul 12 19:58:20.580518 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:58:20.580492 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-5865e5/auth-test-model-5865e5-kserve-76c8db95ff-6tcbr"] Jul 12 19:58:20.582190 ip-10-0-139-170 kubenswrapper[2565]: W0712 19:58:20.582163 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod51d9cc0e_7fe3_4764_8937_b8bbeae60e06.slice/crio-dc50e9b0b907f53d6d156697bf1eaac5da753ce94fcd48967c9f427b712b8d55 WatchSource:0}: Error finding container dc50e9b0b907f53d6d156697bf1eaac5da753ce94fcd48967c9f427b712b8d55: Status 404 returned error can't find the container with id dc50e9b0b907f53d6d156697bf1eaac5da753ce94fcd48967c9f427b712b8d55 Jul 12 19:58:21.160323 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:58:21.160284 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-5865e5/auth-test-model-5865e5-kserve-76c8db95ff-6tcbr" event={"ID":"51d9cc0e-7fe3-4764-8937-b8bbeae60e06","Type":"ContainerStarted","Data":"90832ce716c456d9e0092f4a84ec9210eb387135e29507c0c7038e39c35637ef"} Jul 12 19:58:21.160705 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:58:21.160331 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-5865e5/auth-test-model-5865e5-kserve-76c8db95ff-6tcbr" event={"ID":"51d9cc0e-7fe3-4764-8937-b8bbeae60e06","Type":"ContainerStarted","Data":"dc50e9b0b907f53d6d156697bf1eaac5da753ce94fcd48967c9f427b712b8d55"} Jul 12 19:58:21.176855 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:58:21.176805 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-5865e5/auth-test-model-5865e5-kserve-76c8db95ff-6tcbr" podStartSLOduration=1.176790452 podStartE2EDuration="1.176790452s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 19:58:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 19:58:21.174849915 +0000 UTC m=+2165.907422414" watchObservedRunningTime="2026-07-12 19:58:21.176790452 +0000 UTC m=+2165.909362967" Jul 12 19:58:30.447435 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:58:30.447400 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-5865e5/auth-test-model-5865e5-kserve-76c8db95ff-6tcbr" Jul 12 19:58:30.447871 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:58:30.447452 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-5865e5/auth-test-model-5865e5-kserve-76c8db95ff-6tcbr" Jul 12 19:58:30.459146 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:58:30.459124 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-5865e5/auth-test-model-5865e5-kserve-76c8db95ff-6tcbr" Jul 12 19:58:31.216821 ip-10-0-139-170 kubenswrapper[2565]: I0712 19:58:31.216790 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-5865e5/auth-test-model-5865e5-kserve-76c8db95ff-6tcbr" Jul 12 20:00:15.435545 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:00:15.435465 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-e68438/auth-test-model-e68438-kserve-8478c59d4c-6sgr8"] Jul 12 20:00:15.487783 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:00:15.487735 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 20:00:16.483988 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:00:16.483954 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-5865e5/auth-test-model-5865e5-kserve-76c8db95ff-6tcbr"] Jul 12 20:00:16.649266 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:00:16.649185 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 20:00:23.430921 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:00:23.430872 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-e68438/sub-test-model-e68438-kserve-746bdc4495-j59gm"] Jul 12 20:00:23.434512 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:00:23.434489 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-e68438/sub-test-model-e68438-kserve-746bdc4495-j59gm" Jul 12 20:00:23.436663 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:00:23.436645 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-e68438\"/\"sub-test-model-e68438-kserve-self-signed-certs\"" Jul 12 20:00:23.445542 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:00:23.445519 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-e68438/sub-test-model-e68438-kserve-746bdc4495-j59gm"] Jul 12 20:00:23.510310 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:00:23.510282 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1e15a170-2d30-4d03-b422-3d8114845933-dshm\") pod \"sub-test-model-e68438-kserve-746bdc4495-j59gm\" (UID: \"1e15a170-2d30-4d03-b422-3d8114845933\") " pod="ai-tenant-e2e-shared-a-e68438/sub-test-model-e68438-kserve-746bdc4495-j59gm" Jul 12 20:00:23.510468 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:00:23.510315 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1e15a170-2d30-4d03-b422-3d8114845933-model-cache\") pod \"sub-test-model-e68438-kserve-746bdc4495-j59gm\" (UID: \"1e15a170-2d30-4d03-b422-3d8114845933\") " pod="ai-tenant-e2e-shared-a-e68438/sub-test-model-e68438-kserve-746bdc4495-j59gm" Jul 12 20:00:23.510468 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:00:23.510400 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e15a170-2d30-4d03-b422-3d8114845933-tmp-dir\") pod \"sub-test-model-e68438-kserve-746bdc4495-j59gm\" (UID: \"1e15a170-2d30-4d03-b422-3d8114845933\") " pod="ai-tenant-e2e-shared-a-e68438/sub-test-model-e68438-kserve-746bdc4495-j59gm" Jul 12 20:00:23.510468 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:00:23.510444 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1e15a170-2d30-4d03-b422-3d8114845933-tls-certs\") pod \"sub-test-model-e68438-kserve-746bdc4495-j59gm\" (UID: \"1e15a170-2d30-4d03-b422-3d8114845933\") " pod="ai-tenant-e2e-shared-a-e68438/sub-test-model-e68438-kserve-746bdc4495-j59gm" Jul 12 20:00:23.510584 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:00:23.510506 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1e15a170-2d30-4d03-b422-3d8114845933-home\") pod \"sub-test-model-e68438-kserve-746bdc4495-j59gm\" (UID: \"1e15a170-2d30-4d03-b422-3d8114845933\") " pod="ai-tenant-e2e-shared-a-e68438/sub-test-model-e68438-kserve-746bdc4495-j59gm" Jul 12 20:00:23.510584 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:00:23.510533 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6knxs\" (UniqueName: \"kubernetes.io/projected/1e15a170-2d30-4d03-b422-3d8114845933-kube-api-access-6knxs\") pod \"sub-test-model-e68438-kserve-746bdc4495-j59gm\" (UID: \"1e15a170-2d30-4d03-b422-3d8114845933\") " pod="ai-tenant-e2e-shared-a-e68438/sub-test-model-e68438-kserve-746bdc4495-j59gm" Jul 12 20:00:23.530052 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:00:23.530025 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 20:00:23.611478 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:00:23.611432 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1e15a170-2d30-4d03-b422-3d8114845933-home\") pod \"sub-test-model-e68438-kserve-746bdc4495-j59gm\" (UID: \"1e15a170-2d30-4d03-b422-3d8114845933\") " pod="ai-tenant-e2e-shared-a-e68438/sub-test-model-e68438-kserve-746bdc4495-j59gm" Jul 12 20:00:23.611478 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:00:23.611477 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6knxs\" (UniqueName: \"kubernetes.io/projected/1e15a170-2d30-4d03-b422-3d8114845933-kube-api-access-6knxs\") pod \"sub-test-model-e68438-kserve-746bdc4495-j59gm\" (UID: \"1e15a170-2d30-4d03-b422-3d8114845933\") " pod="ai-tenant-e2e-shared-a-e68438/sub-test-model-e68438-kserve-746bdc4495-j59gm" Jul 12 20:00:23.611709 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:00:23.611535 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1e15a170-2d30-4d03-b422-3d8114845933-dshm\") pod \"sub-test-model-e68438-kserve-746bdc4495-j59gm\" (UID: \"1e15a170-2d30-4d03-b422-3d8114845933\") " pod="ai-tenant-e2e-shared-a-e68438/sub-test-model-e68438-kserve-746bdc4495-j59gm" Jul 12 20:00:23.611709 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:00:23.611567 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1e15a170-2d30-4d03-b422-3d8114845933-model-cache\") pod \"sub-test-model-e68438-kserve-746bdc4495-j59gm\" (UID: \"1e15a170-2d30-4d03-b422-3d8114845933\") " pod="ai-tenant-e2e-shared-a-e68438/sub-test-model-e68438-kserve-746bdc4495-j59gm" Jul 12 20:00:23.611809 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:00:23.611761 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e15a170-2d30-4d03-b422-3d8114845933-tmp-dir\") pod \"sub-test-model-e68438-kserve-746bdc4495-j59gm\" (UID: \"1e15a170-2d30-4d03-b422-3d8114845933\") " pod="ai-tenant-e2e-shared-a-e68438/sub-test-model-e68438-kserve-746bdc4495-j59gm" Jul 12 20:00:23.611864 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:00:23.611812 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1e15a170-2d30-4d03-b422-3d8114845933-tls-certs\") pod \"sub-test-model-e68438-kserve-746bdc4495-j59gm\" (UID: \"1e15a170-2d30-4d03-b422-3d8114845933\") " pod="ai-tenant-e2e-shared-a-e68438/sub-test-model-e68438-kserve-746bdc4495-j59gm" Jul 12 20:00:23.612013 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:00:23.611875 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1e15a170-2d30-4d03-b422-3d8114845933-home\") pod \"sub-test-model-e68438-kserve-746bdc4495-j59gm\" (UID: \"1e15a170-2d30-4d03-b422-3d8114845933\") " pod="ai-tenant-e2e-shared-a-e68438/sub-test-model-e68438-kserve-746bdc4495-j59gm" Jul 12 20:00:23.612013 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:00:23.611926 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1e15a170-2d30-4d03-b422-3d8114845933-model-cache\") pod \"sub-test-model-e68438-kserve-746bdc4495-j59gm\" (UID: \"1e15a170-2d30-4d03-b422-3d8114845933\") " pod="ai-tenant-e2e-shared-a-e68438/sub-test-model-e68438-kserve-746bdc4495-j59gm" Jul 12 20:00:23.612095 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:00:23.612076 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e15a170-2d30-4d03-b422-3d8114845933-tmp-dir\") pod \"sub-test-model-e68438-kserve-746bdc4495-j59gm\" (UID: \"1e15a170-2d30-4d03-b422-3d8114845933\") " pod="ai-tenant-e2e-shared-a-e68438/sub-test-model-e68438-kserve-746bdc4495-j59gm" Jul 12 20:00:23.613799 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:00:23.613775 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1e15a170-2d30-4d03-b422-3d8114845933-dshm\") pod \"sub-test-model-e68438-kserve-746bdc4495-j59gm\" (UID: \"1e15a170-2d30-4d03-b422-3d8114845933\") " pod="ai-tenant-e2e-shared-a-e68438/sub-test-model-e68438-kserve-746bdc4495-j59gm" Jul 12 20:00:23.614076 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:00:23.614059 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1e15a170-2d30-4d03-b422-3d8114845933-tls-certs\") pod \"sub-test-model-e68438-kserve-746bdc4495-j59gm\" (UID: \"1e15a170-2d30-4d03-b422-3d8114845933\") " pod="ai-tenant-e2e-shared-a-e68438/sub-test-model-e68438-kserve-746bdc4495-j59gm" Jul 12 20:00:23.618338 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:00:23.618319 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6knxs\" (UniqueName: \"kubernetes.io/projected/1e15a170-2d30-4d03-b422-3d8114845933-kube-api-access-6knxs\") pod \"sub-test-model-e68438-kserve-746bdc4495-j59gm\" (UID: \"1e15a170-2d30-4d03-b422-3d8114845933\") " pod="ai-tenant-e2e-shared-a-e68438/sub-test-model-e68438-kserve-746bdc4495-j59gm" Jul 12 20:00:23.743074 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:00:23.742993 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-e68438/sub-test-model-e68438-kserve-746bdc4495-j59gm" Jul 12 20:00:23.866253 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:00:23.866116 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-e68438/sub-test-model-e68438-kserve-746bdc4495-j59gm"] Jul 12 20:00:23.868950 ip-10-0-139-170 kubenswrapper[2565]: W0712 20:00:23.868918 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1e15a170_2d30_4d03_b422_3d8114845933.slice/crio-84177b6a11994862c7a6de1da1a238f87ef99efb3d0c0a30a86402aeda2bc84b WatchSource:0}: Error finding container 84177b6a11994862c7a6de1da1a238f87ef99efb3d0c0a30a86402aeda2bc84b: Status 404 returned error can't find the container with id 84177b6a11994862c7a6de1da1a238f87ef99efb3d0c0a30a86402aeda2bc84b Jul 12 20:00:24.696801 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:00:24.696764 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-e68438/sub-test-model-e68438-kserve-746bdc4495-j59gm" event={"ID":"1e15a170-2d30-4d03-b422-3d8114845933","Type":"ContainerStarted","Data":"dca20b7633e3a653a7f6096fa311f201ba7eecb68f66804d7099167d4f053738"} Jul 12 20:00:24.696801 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:00:24.696806 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-e68438/sub-test-model-e68438-kserve-746bdc4495-j59gm" event={"ID":"1e15a170-2d30-4d03-b422-3d8114845933","Type":"ContainerStarted","Data":"84177b6a11994862c7a6de1da1a238f87ef99efb3d0c0a30a86402aeda2bc84b"} Jul 12 20:00:24.712371 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:00:24.712324 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-e68438/sub-test-model-e68438-kserve-746bdc4495-j59gm" podStartSLOduration=1.7123089089999999 podStartE2EDuration="1.712308909s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 20:00:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 20:00:24.711233959 +0000 UTC m=+2289.443806455" watchObservedRunningTime="2026-07-12 20:00:24.712308909 +0000 UTC m=+2289.444881398" Jul 12 20:00:30.461119 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:00:30.461065 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-e68438/auth-test-model-e68438-kserve-8478c59d4c-6sgr8" podUID="62771a54-22e2-4344-882a-27adb79d46c1" containerName="main" containerID="cri-o://47e180a2b3d6849b74c4cf48133db4cc853ec6b8820eeb61c1673abbcb4b4d76" gracePeriod=45 Jul 12 20:00:30.714515 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:00:30.714456 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-e68438/auth-test-model-e68438-kserve-8478c59d4c-6sgr8" Jul 12 20:00:30.723235 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:00:30.723204 2565 generic.go:356] "Generic (PLEG): container finished" podID="62771a54-22e2-4344-882a-27adb79d46c1" containerID="47e180a2b3d6849b74c4cf48133db4cc853ec6b8820eeb61c1673abbcb4b4d76" exitCode=0 Jul 12 20:00:30.723374 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:00:30.723243 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-e68438/auth-test-model-e68438-kserve-8478c59d4c-6sgr8" Jul 12 20:00:30.723374 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:00:30.723288 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-e68438/auth-test-model-e68438-kserve-8478c59d4c-6sgr8" event={"ID":"62771a54-22e2-4344-882a-27adb79d46c1","Type":"ContainerDied","Data":"47e180a2b3d6849b74c4cf48133db4cc853ec6b8820eeb61c1673abbcb4b4d76"} Jul 12 20:00:30.723374 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:00:30.723335 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-e68438/auth-test-model-e68438-kserve-8478c59d4c-6sgr8" event={"ID":"62771a54-22e2-4344-882a-27adb79d46c1","Type":"ContainerDied","Data":"b63d5c46477aebeec3b61ffda088490ee563e300f08cbfbed68c754c06e75049"} Jul 12 20:00:30.723374 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:00:30.723349 2565 scope.go:117] "RemoveContainer" containerID="47e180a2b3d6849b74c4cf48133db4cc853ec6b8820eeb61c1673abbcb4b4d76" Jul 12 20:00:30.732588 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:00:30.732570 2565 scope.go:117] "RemoveContainer" containerID="47e180a2b3d6849b74c4cf48133db4cc853ec6b8820eeb61c1673abbcb4b4d76" Jul 12 20:00:30.732877 ip-10-0-139-170 kubenswrapper[2565]: E0712 20:00:30.732856 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47e180a2b3d6849b74c4cf48133db4cc853ec6b8820eeb61c1673abbcb4b4d76\": container with ID starting with 47e180a2b3d6849b74c4cf48133db4cc853ec6b8820eeb61c1673abbcb4b4d76 not found: ID does not exist" containerID="47e180a2b3d6849b74c4cf48133db4cc853ec6b8820eeb61c1673abbcb4b4d76" Jul 12 20:00:30.732877 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:00:30.732911 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47e180a2b3d6849b74c4cf48133db4cc853ec6b8820eeb61c1673abbcb4b4d76"} err="failed to get container status \"47e180a2b3d6849b74c4cf48133db4cc853ec6b8820eeb61c1673abbcb4b4d76\": rpc error: code = NotFound desc = could not find container \"47e180a2b3d6849b74c4cf48133db4cc853ec6b8820eeb61c1673abbcb4b4d76\": container with ID starting with 47e180a2b3d6849b74c4cf48133db4cc853ec6b8820eeb61c1673abbcb4b4d76 not found: ID does not exist" Jul 12 20:00:30.773284 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:00:30.773256 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62771a54-22e2-4344-882a-27adb79d46c1-tmp-dir\") pod \"62771a54-22e2-4344-882a-27adb79d46c1\" (UID: \"62771a54-22e2-4344-882a-27adb79d46c1\") " Jul 12 20:00:30.773440 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:00:30.773304 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/62771a54-22e2-4344-882a-27adb79d46c1-model-cache\") pod \"62771a54-22e2-4344-882a-27adb79d46c1\" (UID: \"62771a54-22e2-4344-882a-27adb79d46c1\") " Jul 12 20:00:30.773440 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:00:30.773338 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62771a54-22e2-4344-882a-27adb79d46c1-tls-certs\") pod \"62771a54-22e2-4344-882a-27adb79d46c1\" (UID: \"62771a54-22e2-4344-882a-27adb79d46c1\") " Jul 12 20:00:30.773440 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:00:30.773359 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-thjqh\" (UniqueName: \"kubernetes.io/projected/62771a54-22e2-4344-882a-27adb79d46c1-kube-api-access-thjqh\") pod \"62771a54-22e2-4344-882a-27adb79d46c1\" (UID: \"62771a54-22e2-4344-882a-27adb79d46c1\") " Jul 12 20:00:30.773440 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:00:30.773394 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/62771a54-22e2-4344-882a-27adb79d46c1-dshm\") pod \"62771a54-22e2-4344-882a-27adb79d46c1\" (UID: \"62771a54-22e2-4344-882a-27adb79d46c1\") " Jul 12 20:00:30.773663 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:00:30.773465 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/62771a54-22e2-4344-882a-27adb79d46c1-home\") pod \"62771a54-22e2-4344-882a-27adb79d46c1\" (UID: \"62771a54-22e2-4344-882a-27adb79d46c1\") " Jul 12 20:00:30.773663 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:00:30.773514 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62771a54-22e2-4344-882a-27adb79d46c1-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "62771a54-22e2-4344-882a-27adb79d46c1" (UID: "62771a54-22e2-4344-882a-27adb79d46c1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 20:00:30.773663 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:00:30.773600 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62771a54-22e2-4344-882a-27adb79d46c1-model-cache" (OuterVolumeSpecName: "model-cache") pod "62771a54-22e2-4344-882a-27adb79d46c1" (UID: "62771a54-22e2-4344-882a-27adb79d46c1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 20:00:30.773797 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:00:30.773730 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62771a54-22e2-4344-882a-27adb79d46c1-home" (OuterVolumeSpecName: "home") pod "62771a54-22e2-4344-882a-27adb79d46c1" (UID: "62771a54-22e2-4344-882a-27adb79d46c1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 20:00:30.773854 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:00:30.773837 2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/62771a54-22e2-4344-882a-27adb79d46c1-home\") on node \"ip-10-0-139-170.ec2.internal\" DevicePath \"\"" Jul 12 20:00:30.773916 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:00:30.773860 2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62771a54-22e2-4344-882a-27adb79d46c1-tmp-dir\") on node \"ip-10-0-139-170.ec2.internal\" DevicePath \"\"" Jul 12 20:00:30.773916 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:00:30.773875 2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/62771a54-22e2-4344-882a-27adb79d46c1-model-cache\") on node \"ip-10-0-139-170.ec2.internal\" DevicePath \"\"" Jul 12 20:00:30.775649 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:00:30.775626 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62771a54-22e2-4344-882a-27adb79d46c1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "62771a54-22e2-4344-882a-27adb79d46c1" (UID: "62771a54-22e2-4344-882a-27adb79d46c1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 12 20:00:30.775929 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:00:30.775908 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62771a54-22e2-4344-882a-27adb79d46c1-dshm" (OuterVolumeSpecName: "dshm") pod "62771a54-22e2-4344-882a-27adb79d46c1" (UID: "62771a54-22e2-4344-882a-27adb79d46c1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 20:00:30.775987 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:00:30.775943 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62771a54-22e2-4344-882a-27adb79d46c1-kube-api-access-thjqh" (OuterVolumeSpecName: "kube-api-access-thjqh") pod "62771a54-22e2-4344-882a-27adb79d46c1" (UID: "62771a54-22e2-4344-882a-27adb79d46c1"). InnerVolumeSpecName "kube-api-access-thjqh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 12 20:00:30.874643 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:00:30.874595 2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62771a54-22e2-4344-882a-27adb79d46c1-tls-certs\") on node \"ip-10-0-139-170.ec2.internal\" DevicePath \"\"" Jul 12 20:00:30.874643 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:00:30.874641 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-thjqh\" (UniqueName: \"kubernetes.io/projected/62771a54-22e2-4344-882a-27adb79d46c1-kube-api-access-thjqh\") on node \"ip-10-0-139-170.ec2.internal\" DevicePath \"\"" Jul 12 20:00:30.874643 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:00:30.874650 2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/62771a54-22e2-4344-882a-27adb79d46c1-dshm\") on node \"ip-10-0-139-170.ec2.internal\" DevicePath \"\"" Jul 12 20:00:31.043466 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:00:31.043431 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-e68438/auth-test-model-e68438-kserve-8478c59d4c-6sgr8"] Jul 12 20:00:31.046353 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:00:31.046327 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-e68438/auth-test-model-e68438-kserve-8478c59d4c-6sgr8"] Jul 12 20:00:31.503868 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:00:31.503830 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-5865e5/auth-test-model-5865e5-kserve-76c8db95ff-6tcbr" podUID="51d9cc0e-7fe3-4764-8937-b8bbeae60e06" containerName="main" containerID="cri-o://90832ce716c456d9e0092f4a84ec9210eb387135e29507c0c7038e39c35637ef" gracePeriod=45 Jul 12 20:00:31.729940 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:00:31.729911 2565 generic.go:356] "Generic (PLEG): container finished" podID="51d9cc0e-7fe3-4764-8937-b8bbeae60e06" containerID="90832ce716c456d9e0092f4a84ec9210eb387135e29507c0c7038e39c35637ef" exitCode=0 Jul 12 20:00:31.730086 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:00:31.729982 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-5865e5/auth-test-model-5865e5-kserve-76c8db95ff-6tcbr" event={"ID":"51d9cc0e-7fe3-4764-8937-b8bbeae60e06","Type":"ContainerDied","Data":"90832ce716c456d9e0092f4a84ec9210eb387135e29507c0c7038e39c35637ef"} Jul 12 20:00:31.748074 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:00:31.748054 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-5865e5/auth-test-model-5865e5-kserve-76c8db95ff-6tcbr" Jul 12 20:00:31.783237 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:00:31.783172 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/51d9cc0e-7fe3-4764-8937-b8bbeae60e06-tls-certs\") pod \"51d9cc0e-7fe3-4764-8937-b8bbeae60e06\" (UID: \"51d9cc0e-7fe3-4764-8937-b8bbeae60e06\") " Jul 12 20:00:31.783237 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:00:31.783225 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/51d9cc0e-7fe3-4764-8937-b8bbeae60e06-dshm\") pod \"51d9cc0e-7fe3-4764-8937-b8bbeae60e06\" (UID: \"51d9cc0e-7fe3-4764-8937-b8bbeae60e06\") " Jul 12 20:00:31.783418 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:00:31.783241 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/51d9cc0e-7fe3-4764-8937-b8bbeae60e06-model-cache\") pod \"51d9cc0e-7fe3-4764-8937-b8bbeae60e06\" (UID: \"51d9cc0e-7fe3-4764-8937-b8bbeae60e06\") " Jul 12 20:00:31.783418 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:00:31.783275 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/51d9cc0e-7fe3-4764-8937-b8bbeae60e06-tmp-dir\") pod \"51d9cc0e-7fe3-4764-8937-b8bbeae60e06\" (UID: \"51d9cc0e-7fe3-4764-8937-b8bbeae60e06\") " Jul 12 20:00:31.783418 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:00:31.783331 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/51d9cc0e-7fe3-4764-8937-b8bbeae60e06-home\") pod \"51d9cc0e-7fe3-4764-8937-b8bbeae60e06\" (UID: \"51d9cc0e-7fe3-4764-8937-b8bbeae60e06\") " Jul 12 20:00:31.783418 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:00:31.783391 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r5c5b\" (UniqueName: \"kubernetes.io/projected/51d9cc0e-7fe3-4764-8937-b8bbeae60e06-kube-api-access-r5c5b\") pod \"51d9cc0e-7fe3-4764-8937-b8bbeae60e06\" (UID: \"51d9cc0e-7fe3-4764-8937-b8bbeae60e06\") " Jul 12 20:00:31.783625 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:00:31.783570 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51d9cc0e-7fe3-4764-8937-b8bbeae60e06-model-cache" (OuterVolumeSpecName: "model-cache") pod "51d9cc0e-7fe3-4764-8937-b8bbeae60e06" (UID: "51d9cc0e-7fe3-4764-8937-b8bbeae60e06"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 20:00:31.783625 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:00:31.783586 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51d9cc0e-7fe3-4764-8937-b8bbeae60e06-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "51d9cc0e-7fe3-4764-8937-b8bbeae60e06" (UID: "51d9cc0e-7fe3-4764-8937-b8bbeae60e06"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 20:00:31.783720 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:00:31.783688 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51d9cc0e-7fe3-4764-8937-b8bbeae60e06-home" (OuterVolumeSpecName: "home") pod "51d9cc0e-7fe3-4764-8937-b8bbeae60e06" (UID: "51d9cc0e-7fe3-4764-8937-b8bbeae60e06"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 20:00:31.783768 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:00:31.783738 2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/51d9cc0e-7fe3-4764-8937-b8bbeae60e06-tmp-dir\") on node \"ip-10-0-139-170.ec2.internal\" DevicePath \"\"" Jul 12 20:00:31.783768 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:00:31.783758 2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/51d9cc0e-7fe3-4764-8937-b8bbeae60e06-model-cache\") on node \"ip-10-0-139-170.ec2.internal\" DevicePath \"\"" Jul 12 20:00:31.785432 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:00:31.785411 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51d9cc0e-7fe3-4764-8937-b8bbeae60e06-dshm" (OuterVolumeSpecName: "dshm") pod "51d9cc0e-7fe3-4764-8937-b8bbeae60e06" (UID: "51d9cc0e-7fe3-4764-8937-b8bbeae60e06"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 20:00:31.785432 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:00:31.785422 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51d9cc0e-7fe3-4764-8937-b8bbeae60e06-kube-api-access-r5c5b" (OuterVolumeSpecName: "kube-api-access-r5c5b") pod "51d9cc0e-7fe3-4764-8937-b8bbeae60e06" (UID: "51d9cc0e-7fe3-4764-8937-b8bbeae60e06"). InnerVolumeSpecName "kube-api-access-r5c5b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 12 20:00:31.785698 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:00:31.785677 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/51d9cc0e-7fe3-4764-8937-b8bbeae60e06-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "51d9cc0e-7fe3-4764-8937-b8bbeae60e06" (UID: "51d9cc0e-7fe3-4764-8937-b8bbeae60e06"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 12 20:00:31.859334 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:00:31.859302 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="62771a54-22e2-4344-882a-27adb79d46c1" path="/var/lib/kubelet/pods/62771a54-22e2-4344-882a-27adb79d46c1/volumes" Jul 12 20:00:31.884590 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:00:31.884565 2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/51d9cc0e-7fe3-4764-8937-b8bbeae60e06-home\") on node \"ip-10-0-139-170.ec2.internal\" DevicePath \"\"" Jul 12 20:00:31.884590 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:00:31.884588 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r5c5b\" (UniqueName: \"kubernetes.io/projected/51d9cc0e-7fe3-4764-8937-b8bbeae60e06-kube-api-access-r5c5b\") on node \"ip-10-0-139-170.ec2.internal\" DevicePath \"\"" Jul 12 20:00:31.884748 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:00:31.884601 2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/51d9cc0e-7fe3-4764-8937-b8bbeae60e06-tls-certs\") on node \"ip-10-0-139-170.ec2.internal\" DevicePath \"\"" Jul 12 20:00:31.884748 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:00:31.884613 2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/51d9cc0e-7fe3-4764-8937-b8bbeae60e06-dshm\") on node \"ip-10-0-139-170.ec2.internal\" DevicePath \"\"" Jul 12 20:00:32.741961 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:00:32.741930 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-5865e5/auth-test-model-5865e5-kserve-76c8db95ff-6tcbr" Jul 12 20:00:32.742376 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:00:32.741928 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-5865e5/auth-test-model-5865e5-kserve-76c8db95ff-6tcbr" event={"ID":"51d9cc0e-7fe3-4764-8937-b8bbeae60e06","Type":"ContainerDied","Data":"dc50e9b0b907f53d6d156697bf1eaac5da753ce94fcd48967c9f427b712b8d55"} Jul 12 20:00:32.742376 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:00:32.742053 2565 scope.go:117] "RemoveContainer" containerID="90832ce716c456d9e0092f4a84ec9210eb387135e29507c0c7038e39c35637ef" Jul 12 20:00:32.757660 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:00:32.757547 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-5865e5/auth-test-model-5865e5-kserve-76c8db95ff-6tcbr"] Jul 12 20:00:32.759543 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:00:32.759519 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-5865e5/auth-test-model-5865e5-kserve-76c8db95ff-6tcbr"] Jul 12 20:00:33.743496 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:00:33.743461 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-e68438/sub-test-model-e68438-kserve-746bdc4495-j59gm" Jul 12 20:00:33.743930 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:00:33.743513 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-e68438/sub-test-model-e68438-kserve-746bdc4495-j59gm" Jul 12 20:00:33.756298 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:00:33.756267 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-e68438/sub-test-model-e68438-kserve-746bdc4495-j59gm" Jul 12 20:00:33.858609 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:00:33.858579 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="51d9cc0e-7fe3-4764-8937-b8bbeae60e06" path="/var/lib/kubelet/pods/51d9cc0e-7fe3-4764-8937-b8bbeae60e06/volumes" Jul 12 20:00:34.763634 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:00:34.763599 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-e68438/sub-test-model-e68438-kserve-746bdc4495-j59gm" Jul 12 20:00:52.060047 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:00:52.060012 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 20:00:54.649653 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:00:54.649615 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 20:01:24.317528 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:01:24.317486 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-e68438/sub-test-model-e68438-kserve-746bdc4495-j59gm"] Jul 12 20:01:24.340145 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:01:24.340119 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 20:01:25.164357 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:01:25.164316 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 20:01:39.239852 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:01:39.239804 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-e68438/rate-test-model-e68438-kserve-59d6c5554c-6t54g"] Jul 12 20:01:39.240234 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:01:39.240141 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62771a54-22e2-4344-882a-27adb79d46c1" containerName="main" Jul 12 20:01:39.240234 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:01:39.240151 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="62771a54-22e2-4344-882a-27adb79d46c1" containerName="main" Jul 12 20:01:39.240234 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:01:39.240158 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="51d9cc0e-7fe3-4764-8937-b8bbeae60e06" containerName="main" Jul 12 20:01:39.240234 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:01:39.240164 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="51d9cc0e-7fe3-4764-8937-b8bbeae60e06" containerName="main" Jul 12 20:01:39.240234 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:01:39.240209 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="51d9cc0e-7fe3-4764-8937-b8bbeae60e06" containerName="main" Jul 12 20:01:39.240234 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:01:39.240217 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="62771a54-22e2-4344-882a-27adb79d46c1" containerName="main" Jul 12 20:01:39.243214 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:01:39.243198 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-e68438/rate-test-model-e68438-kserve-59d6c5554c-6t54g" Jul 12 20:01:39.245775 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:01:39.245749 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-e68438\"/\"rate-test-model-e68438-kserve-self-signed-certs\"" Jul 12 20:01:39.252350 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:01:39.252327 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-e68438/rate-test-model-e68438-kserve-59d6c5554c-6t54g"] Jul 12 20:01:39.260135 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:01:39.260110 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 20:01:39.337101 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:01:39.337044 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-e68438/sub-test-model-e68438-kserve-746bdc4495-j59gm" podUID="1e15a170-2d30-4d03-b422-3d8114845933" containerName="main" containerID="cri-o://dca20b7633e3a653a7f6096fa311f201ba7eecb68f66804d7099167d4f053738" gracePeriod=45 Jul 12 20:01:39.406208 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:01:39.406169 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0c65a059-596e-4da4-af4b-d394c7b02699-home\") pod \"rate-test-model-e68438-kserve-59d6c5554c-6t54g\" (UID: \"0c65a059-596e-4da4-af4b-d394c7b02699\") " pod="ai-tenant-e2e-shared-a-e68438/rate-test-model-e68438-kserve-59d6c5554c-6t54g" Jul 12 20:01:39.406389 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:01:39.406226 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c65a059-596e-4da4-af4b-d394c7b02699-tls-certs\") pod \"rate-test-model-e68438-kserve-59d6c5554c-6t54g\" (UID: \"0c65a059-596e-4da4-af4b-d394c7b02699\") " pod="ai-tenant-e2e-shared-a-e68438/rate-test-model-e68438-kserve-59d6c5554c-6t54g" Jul 12 20:01:39.406389 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:01:39.406252 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c65a059-596e-4da4-af4b-d394c7b02699-tmp-dir\") pod \"rate-test-model-e68438-kserve-59d6c5554c-6t54g\" (UID: \"0c65a059-596e-4da4-af4b-d394c7b02699\") " pod="ai-tenant-e2e-shared-a-e68438/rate-test-model-e68438-kserve-59d6c5554c-6t54g" Jul 12 20:01:39.406389 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:01:39.406289 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ks2n8\" (UniqueName: \"kubernetes.io/projected/0c65a059-596e-4da4-af4b-d394c7b02699-kube-api-access-ks2n8\") pod \"rate-test-model-e68438-kserve-59d6c5554c-6t54g\" (UID: \"0c65a059-596e-4da4-af4b-d394c7b02699\") " pod="ai-tenant-e2e-shared-a-e68438/rate-test-model-e68438-kserve-59d6c5554c-6t54g" Jul 12 20:01:39.406389 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:01:39.406312 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c65a059-596e-4da4-af4b-d394c7b02699-dshm\") pod \"rate-test-model-e68438-kserve-59d6c5554c-6t54g\" (UID: \"0c65a059-596e-4da4-af4b-d394c7b02699\") " pod="ai-tenant-e2e-shared-a-e68438/rate-test-model-e68438-kserve-59d6c5554c-6t54g" Jul 12 20:01:39.406389 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:01:39.406328 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c65a059-596e-4da4-af4b-d394c7b02699-model-cache\") pod \"rate-test-model-e68438-kserve-59d6c5554c-6t54g\" (UID: \"0c65a059-596e-4da4-af4b-d394c7b02699\") " pod="ai-tenant-e2e-shared-a-e68438/rate-test-model-e68438-kserve-59d6c5554c-6t54g" Jul 12 20:01:39.507711 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:01:39.507607 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ks2n8\" (UniqueName: \"kubernetes.io/projected/0c65a059-596e-4da4-af4b-d394c7b02699-kube-api-access-ks2n8\") pod \"rate-test-model-e68438-kserve-59d6c5554c-6t54g\" (UID: \"0c65a059-596e-4da4-af4b-d394c7b02699\") " pod="ai-tenant-e2e-shared-a-e68438/rate-test-model-e68438-kserve-59d6c5554c-6t54g" Jul 12 20:01:39.507711 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:01:39.507663 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c65a059-596e-4da4-af4b-d394c7b02699-dshm\") pod \"rate-test-model-e68438-kserve-59d6c5554c-6t54g\" (UID: \"0c65a059-596e-4da4-af4b-d394c7b02699\") " pod="ai-tenant-e2e-shared-a-e68438/rate-test-model-e68438-kserve-59d6c5554c-6t54g" Jul 12 20:01:39.507711 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:01:39.507693 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c65a059-596e-4da4-af4b-d394c7b02699-model-cache\") pod \"rate-test-model-e68438-kserve-59d6c5554c-6t54g\" (UID: \"0c65a059-596e-4da4-af4b-d394c7b02699\") " pod="ai-tenant-e2e-shared-a-e68438/rate-test-model-e68438-kserve-59d6c5554c-6t54g" Jul 12 20:01:39.508019 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:01:39.507762 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0c65a059-596e-4da4-af4b-d394c7b02699-home\") pod \"rate-test-model-e68438-kserve-59d6c5554c-6t54g\" (UID: \"0c65a059-596e-4da4-af4b-d394c7b02699\") " pod="ai-tenant-e2e-shared-a-e68438/rate-test-model-e68438-kserve-59d6c5554c-6t54g" Jul 12 20:01:39.508019 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:01:39.507822 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c65a059-596e-4da4-af4b-d394c7b02699-tls-certs\") pod \"rate-test-model-e68438-kserve-59d6c5554c-6t54g\" (UID: \"0c65a059-596e-4da4-af4b-d394c7b02699\") " pod="ai-tenant-e2e-shared-a-e68438/rate-test-model-e68438-kserve-59d6c5554c-6t54g" Jul 12 20:01:39.508019 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:01:39.507844 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c65a059-596e-4da4-af4b-d394c7b02699-tmp-dir\") pod \"rate-test-model-e68438-kserve-59d6c5554c-6t54g\" (UID: \"0c65a059-596e-4da4-af4b-d394c7b02699\") " pod="ai-tenant-e2e-shared-a-e68438/rate-test-model-e68438-kserve-59d6c5554c-6t54g" Jul 12 20:01:39.508514 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:01:39.508353 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c65a059-596e-4da4-af4b-d394c7b02699-tmp-dir\") pod \"rate-test-model-e68438-kserve-59d6c5554c-6t54g\" (UID: \"0c65a059-596e-4da4-af4b-d394c7b02699\") " pod="ai-tenant-e2e-shared-a-e68438/rate-test-model-e68438-kserve-59d6c5554c-6t54g" Jul 12 20:01:39.508514 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:01:39.508483 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0c65a059-596e-4da4-af4b-d394c7b02699-home\") pod \"rate-test-model-e68438-kserve-59d6c5554c-6t54g\" (UID: \"0c65a059-596e-4da4-af4b-d394c7b02699\") " pod="ai-tenant-e2e-shared-a-e68438/rate-test-model-e68438-kserve-59d6c5554c-6t54g" Jul 12 20:01:39.508670 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:01:39.508576 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c65a059-596e-4da4-af4b-d394c7b02699-model-cache\") pod \"rate-test-model-e68438-kserve-59d6c5554c-6t54g\" (UID: \"0c65a059-596e-4da4-af4b-d394c7b02699\") " pod="ai-tenant-e2e-shared-a-e68438/rate-test-model-e68438-kserve-59d6c5554c-6t54g" Jul 12 20:01:39.510901 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:01:39.510860 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c65a059-596e-4da4-af4b-d394c7b02699-dshm\") pod \"rate-test-model-e68438-kserve-59d6c5554c-6t54g\" (UID: \"0c65a059-596e-4da4-af4b-d394c7b02699\") " pod="ai-tenant-e2e-shared-a-e68438/rate-test-model-e68438-kserve-59d6c5554c-6t54g" Jul 12 20:01:39.511519 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:01:39.511494 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c65a059-596e-4da4-af4b-d394c7b02699-tls-certs\") pod \"rate-test-model-e68438-kserve-59d6c5554c-6t54g\" (UID: \"0c65a059-596e-4da4-af4b-d394c7b02699\") " pod="ai-tenant-e2e-shared-a-e68438/rate-test-model-e68438-kserve-59d6c5554c-6t54g" Jul 12 20:01:39.516713 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:01:39.516685 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ks2n8\" (UniqueName: \"kubernetes.io/projected/0c65a059-596e-4da4-af4b-d394c7b02699-kube-api-access-ks2n8\") pod \"rate-test-model-e68438-kserve-59d6c5554c-6t54g\" (UID: \"0c65a059-596e-4da4-af4b-d394c7b02699\") " pod="ai-tenant-e2e-shared-a-e68438/rate-test-model-e68438-kserve-59d6c5554c-6t54g" Jul 12 20:01:39.552459 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:01:39.552414 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-e68438/rate-test-model-e68438-kserve-59d6c5554c-6t54g" Jul 12 20:01:39.583235 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:01:39.583212 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-e68438/sub-test-model-e68438-kserve-746bdc4495-j59gm" Jul 12 20:01:39.692796 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:01:39.692767 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-e68438/rate-test-model-e68438-kserve-59d6c5554c-6t54g"] Jul 12 20:01:39.695019 ip-10-0-139-170 kubenswrapper[2565]: W0712 20:01:39.694993 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0c65a059_596e_4da4_af4b_d394c7b02699.slice/crio-43f6a2bb5696b05b74c22e142c0622ea1c10ac53c755ea1300b4c07181ef119c WatchSource:0}: Error finding container 43f6a2bb5696b05b74c22e142c0622ea1c10ac53c755ea1300b4c07181ef119c: Status 404 returned error can't find the container with id 43f6a2bb5696b05b74c22e142c0622ea1c10ac53c755ea1300b4c07181ef119c Jul 12 20:01:39.709675 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:01:39.709645 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e15a170-2d30-4d03-b422-3d8114845933-tmp-dir\") pod \"1e15a170-2d30-4d03-b422-3d8114845933\" (UID: \"1e15a170-2d30-4d03-b422-3d8114845933\") " Jul 12 20:01:39.709768 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:01:39.709708 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1e15a170-2d30-4d03-b422-3d8114845933-tls-certs\") pod \"1e15a170-2d30-4d03-b422-3d8114845933\" (UID: \"1e15a170-2d30-4d03-b422-3d8114845933\") " Jul 12 20:01:39.709768 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:01:39.709733 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1e15a170-2d30-4d03-b422-3d8114845933-dshm\") pod \"1e15a170-2d30-4d03-b422-3d8114845933\" (UID: \"1e15a170-2d30-4d03-b422-3d8114845933\") " Jul 12 20:01:39.709768 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:01:39.709750 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1e15a170-2d30-4d03-b422-3d8114845933-home\") pod \"1e15a170-2d30-4d03-b422-3d8114845933\" (UID: \"1e15a170-2d30-4d03-b422-3d8114845933\") " Jul 12 20:01:39.710023 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:01:39.709841 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6knxs\" (UniqueName: \"kubernetes.io/projected/1e15a170-2d30-4d03-b422-3d8114845933-kube-api-access-6knxs\") pod \"1e15a170-2d30-4d03-b422-3d8114845933\" (UID: \"1e15a170-2d30-4d03-b422-3d8114845933\") " Jul 12 20:01:39.710023 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:01:39.709907 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1e15a170-2d30-4d03-b422-3d8114845933-model-cache\") pod \"1e15a170-2d30-4d03-b422-3d8114845933\" (UID: \"1e15a170-2d30-4d03-b422-3d8114845933\") " Jul 12 20:01:39.710023 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:01:39.709922 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e15a170-2d30-4d03-b422-3d8114845933-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1e15a170-2d30-4d03-b422-3d8114845933" (UID: "1e15a170-2d30-4d03-b422-3d8114845933"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 20:01:39.710187 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:01:39.710048 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e15a170-2d30-4d03-b422-3d8114845933-home" (OuterVolumeSpecName: "home") pod "1e15a170-2d30-4d03-b422-3d8114845933" (UID: "1e15a170-2d30-4d03-b422-3d8114845933"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 20:01:39.710263 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:01:39.710237 2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e15a170-2d30-4d03-b422-3d8114845933-tmp-dir\") on node \"ip-10-0-139-170.ec2.internal\" DevicePath \"\"" Jul 12 20:01:39.710318 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:01:39.710278 2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1e15a170-2d30-4d03-b422-3d8114845933-home\") on node \"ip-10-0-139-170.ec2.internal\" DevicePath \"\"" Jul 12 20:01:39.710318 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:01:39.710238 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e15a170-2d30-4d03-b422-3d8114845933-model-cache" (OuterVolumeSpecName: "model-cache") pod "1e15a170-2d30-4d03-b422-3d8114845933" (UID: "1e15a170-2d30-4d03-b422-3d8114845933"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 20:01:39.712008 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:01:39.711985 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e15a170-2d30-4d03-b422-3d8114845933-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1e15a170-2d30-4d03-b422-3d8114845933" (UID: "1e15a170-2d30-4d03-b422-3d8114845933"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 12 20:01:39.712119 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:01:39.712035 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e15a170-2d30-4d03-b422-3d8114845933-kube-api-access-6knxs" (OuterVolumeSpecName: "kube-api-access-6knxs") pod "1e15a170-2d30-4d03-b422-3d8114845933" (UID: "1e15a170-2d30-4d03-b422-3d8114845933"). InnerVolumeSpecName "kube-api-access-6knxs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 12 20:01:39.712189 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:01:39.712170 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e15a170-2d30-4d03-b422-3d8114845933-dshm" (OuterVolumeSpecName: "dshm") pod "1e15a170-2d30-4d03-b422-3d8114845933" (UID: "1e15a170-2d30-4d03-b422-3d8114845933"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 20:01:39.811661 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:01:39.811561 2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1e15a170-2d30-4d03-b422-3d8114845933-tls-certs\") on node \"ip-10-0-139-170.ec2.internal\" DevicePath \"\"" Jul 12 20:01:39.811661 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:01:39.811595 2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1e15a170-2d30-4d03-b422-3d8114845933-dshm\") on node \"ip-10-0-139-170.ec2.internal\" DevicePath \"\"" Jul 12 20:01:39.811661 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:01:39.811606 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6knxs\" (UniqueName: \"kubernetes.io/projected/1e15a170-2d30-4d03-b422-3d8114845933-kube-api-access-6knxs\") on node \"ip-10-0-139-170.ec2.internal\" DevicePath \"\"" Jul 12 20:01:39.811661 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:01:39.811619 2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1e15a170-2d30-4d03-b422-3d8114845933-model-cache\") on node \"ip-10-0-139-170.ec2.internal\" DevicePath \"\"" Jul 12 20:01:40.039103 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:01:40.039069 2565 generic.go:356] "Generic (PLEG): container finished" podID="1e15a170-2d30-4d03-b422-3d8114845933" containerID="dca20b7633e3a653a7f6096fa311f201ba7eecb68f66804d7099167d4f053738" exitCode=0 Jul 12 20:01:40.039298 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:01:40.039115 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-e68438/sub-test-model-e68438-kserve-746bdc4495-j59gm" Jul 12 20:01:40.039298 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:01:40.039159 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-e68438/sub-test-model-e68438-kserve-746bdc4495-j59gm" event={"ID":"1e15a170-2d30-4d03-b422-3d8114845933","Type":"ContainerDied","Data":"dca20b7633e3a653a7f6096fa311f201ba7eecb68f66804d7099167d4f053738"} Jul 12 20:01:40.039298 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:01:40.039202 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-e68438/sub-test-model-e68438-kserve-746bdc4495-j59gm" event={"ID":"1e15a170-2d30-4d03-b422-3d8114845933","Type":"ContainerDied","Data":"84177b6a11994862c7a6de1da1a238f87ef99efb3d0c0a30a86402aeda2bc84b"} Jul 12 20:01:40.039298 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:01:40.039243 2565 scope.go:117] "RemoveContainer" containerID="dca20b7633e3a653a7f6096fa311f201ba7eecb68f66804d7099167d4f053738" Jul 12 20:01:40.040780 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:01:40.040753 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-e68438/rate-test-model-e68438-kserve-59d6c5554c-6t54g" event={"ID":"0c65a059-596e-4da4-af4b-d394c7b02699","Type":"ContainerStarted","Data":"dcbece61070a467c4ddce696dd90b6655a63ec330dcb16c219e68dddfb86fc9a"} Jul 12 20:01:40.040912 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:01:40.040787 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-e68438/rate-test-model-e68438-kserve-59d6c5554c-6t54g" event={"ID":"0c65a059-596e-4da4-af4b-d394c7b02699","Type":"ContainerStarted","Data":"43f6a2bb5696b05b74c22e142c0622ea1c10ac53c755ea1300b4c07181ef119c"} Jul 12 20:01:40.047506 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:01:40.047476 2565 scope.go:117] "RemoveContainer" containerID="dca20b7633e3a653a7f6096fa311f201ba7eecb68f66804d7099167d4f053738" Jul 12 20:01:40.047752 ip-10-0-139-170 kubenswrapper[2565]: E0712 20:01:40.047735 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dca20b7633e3a653a7f6096fa311f201ba7eecb68f66804d7099167d4f053738\": container with ID starting with dca20b7633e3a653a7f6096fa311f201ba7eecb68f66804d7099167d4f053738 not found: ID does not exist" containerID="dca20b7633e3a653a7f6096fa311f201ba7eecb68f66804d7099167d4f053738" Jul 12 20:01:40.047832 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:01:40.047759 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dca20b7633e3a653a7f6096fa311f201ba7eecb68f66804d7099167d4f053738"} err="failed to get container status \"dca20b7633e3a653a7f6096fa311f201ba7eecb68f66804d7099167d4f053738\": rpc error: code = NotFound desc = could not find container \"dca20b7633e3a653a7f6096fa311f201ba7eecb68f66804d7099167d4f053738\": container with ID starting with dca20b7633e3a653a7f6096fa311f201ba7eecb68f66804d7099167d4f053738 not found: ID does not exist" Jul 12 20:01:40.055139 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:01:40.055116 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-e68438/sub-test-model-e68438-kserve-746bdc4495-j59gm"] Jul 12 20:01:40.058660 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:01:40.058639 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-e68438/sub-test-model-e68438-kserve-746bdc4495-j59gm"] Jul 12 20:01:40.072198 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:01:40.072124 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-e68438/rate-test-model-e68438-kserve-59d6c5554c-6t54g" podStartSLOduration=1.072110205 podStartE2EDuration="1.072110205s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 20:01:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 20:01:40.070938 +0000 UTC m=+2364.803510491" watchObservedRunningTime="2026-07-12 20:01:40.072110205 +0000 UTC m=+2364.804682694" Jul 12 20:01:41.857239 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:01:41.857202 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1e15a170-2d30-4d03-b422-3d8114845933" path="/var/lib/kubelet/pods/1e15a170-2d30-4d03-b422-3d8114845933/volumes" Jul 12 20:01:45.606868 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:01:45.606833 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 20:01:49.553432 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:01:49.553389 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-e68438/rate-test-model-e68438-kserve-59d6c5554c-6t54g" Jul 12 20:01:49.553838 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:01:49.553542 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-e68438/rate-test-model-e68438-kserve-59d6c5554c-6t54g" Jul 12 20:01:49.565844 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:01:49.565817 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-e68438/rate-test-model-e68438-kserve-59d6c5554c-6t54g" Jul 12 20:01:50.097078 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:01:50.097052 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-e68438/rate-test-model-e68438-kserve-59d6c5554c-6t54g" Jul 12 20:02:02.515085 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:02:02.515045 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 20:02:02.613757 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:02:02.613721 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 20:02:03.213542 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:02:03.213510 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-5865e5/rate-test-model-5865e5-kserve-55f6f65947-s7c94"] Jul 12 20:02:03.213851 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:02:03.213839 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1e15a170-2d30-4d03-b422-3d8114845933" containerName="main" Jul 12 20:02:03.213915 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:02:03.213853 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="1e15a170-2d30-4d03-b422-3d8114845933" containerName="main" Jul 12 20:02:03.213956 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:02:03.213942 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="1e15a170-2d30-4d03-b422-3d8114845933" containerName="main" Jul 12 20:02:03.217216 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:02:03.217192 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-5865e5/rate-test-model-5865e5-kserve-55f6f65947-s7c94" Jul 12 20:02:03.219525 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:02:03.219492 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-5865e5\"/\"openshift-service-ca.crt\"" Jul 12 20:02:03.220630 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:02:03.220611 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-5865e5\"/\"rate-test-model-5865e5-kserve-self-signed-certs\"" Jul 12 20:02:03.220744 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:02:03.220659 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-5865e5\"/\"default-dockercfg-t2pbn\"" Jul 12 20:02:03.220744 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:02:03.220726 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-5865e5\"/\"kube-root-ca.crt\"" Jul 12 20:02:03.228537 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:02:03.228507 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-5865e5/rate-test-model-5865e5-kserve-55f6f65947-s7c94"] Jul 12 20:02:03.237261 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:02:03.237225 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 20:02:03.328237 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:02:03.328202 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ccb37f0a-ea64-4cd6-a238-9b012bcd7e03-home\") pod \"rate-test-model-5865e5-kserve-55f6f65947-s7c94\" (UID: \"ccb37f0a-ea64-4cd6-a238-9b012bcd7e03\") " pod="ai-tenant-e2e-shared-b-5865e5/rate-test-model-5865e5-kserve-55f6f65947-s7c94" Jul 12 20:02:03.328425 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:02:03.328252 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ccb37f0a-ea64-4cd6-a238-9b012bcd7e03-tls-certs\") pod \"rate-test-model-5865e5-kserve-55f6f65947-s7c94\" (UID: \"ccb37f0a-ea64-4cd6-a238-9b012bcd7e03\") " pod="ai-tenant-e2e-shared-b-5865e5/rate-test-model-5865e5-kserve-55f6f65947-s7c94" Jul 12 20:02:03.328425 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:02:03.328326 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ccb37f0a-ea64-4cd6-a238-9b012bcd7e03-model-cache\") pod \"rate-test-model-5865e5-kserve-55f6f65947-s7c94\" (UID: \"ccb37f0a-ea64-4cd6-a238-9b012bcd7e03\") " pod="ai-tenant-e2e-shared-b-5865e5/rate-test-model-5865e5-kserve-55f6f65947-s7c94" Jul 12 20:02:03.328425 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:02:03.328383 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ccb37f0a-ea64-4cd6-a238-9b012bcd7e03-dshm\") pod \"rate-test-model-5865e5-kserve-55f6f65947-s7c94\" (UID: \"ccb37f0a-ea64-4cd6-a238-9b012bcd7e03\") " pod="ai-tenant-e2e-shared-b-5865e5/rate-test-model-5865e5-kserve-55f6f65947-s7c94" Jul 12 20:02:03.328425 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:02:03.328410 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ccb37f0a-ea64-4cd6-a238-9b012bcd7e03-tmp-dir\") pod \"rate-test-model-5865e5-kserve-55f6f65947-s7c94\" (UID: \"ccb37f0a-ea64-4cd6-a238-9b012bcd7e03\") " pod="ai-tenant-e2e-shared-b-5865e5/rate-test-model-5865e5-kserve-55f6f65947-s7c94" Jul 12 20:02:03.328609 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:02:03.328452 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fq972\" (UniqueName: \"kubernetes.io/projected/ccb37f0a-ea64-4cd6-a238-9b012bcd7e03-kube-api-access-fq972\") pod \"rate-test-model-5865e5-kserve-55f6f65947-s7c94\" (UID: \"ccb37f0a-ea64-4cd6-a238-9b012bcd7e03\") " pod="ai-tenant-e2e-shared-b-5865e5/rate-test-model-5865e5-kserve-55f6f65947-s7c94" Jul 12 20:02:03.429376 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:02:03.429327 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ccb37f0a-ea64-4cd6-a238-9b012bcd7e03-dshm\") pod \"rate-test-model-5865e5-kserve-55f6f65947-s7c94\" (UID: \"ccb37f0a-ea64-4cd6-a238-9b012bcd7e03\") " pod="ai-tenant-e2e-shared-b-5865e5/rate-test-model-5865e5-kserve-55f6f65947-s7c94" Jul 12 20:02:03.429376 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:02:03.429381 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ccb37f0a-ea64-4cd6-a238-9b012bcd7e03-tmp-dir\") pod \"rate-test-model-5865e5-kserve-55f6f65947-s7c94\" (UID: \"ccb37f0a-ea64-4cd6-a238-9b012bcd7e03\") " pod="ai-tenant-e2e-shared-b-5865e5/rate-test-model-5865e5-kserve-55f6f65947-s7c94" Jul 12 20:02:03.429618 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:02:03.429427 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fq972\" (UniqueName: \"kubernetes.io/projected/ccb37f0a-ea64-4cd6-a238-9b012bcd7e03-kube-api-access-fq972\") pod \"rate-test-model-5865e5-kserve-55f6f65947-s7c94\" (UID: \"ccb37f0a-ea64-4cd6-a238-9b012bcd7e03\") " pod="ai-tenant-e2e-shared-b-5865e5/rate-test-model-5865e5-kserve-55f6f65947-s7c94" Jul 12 20:02:03.429618 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:02:03.429518 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ccb37f0a-ea64-4cd6-a238-9b012bcd7e03-home\") pod \"rate-test-model-5865e5-kserve-55f6f65947-s7c94\" (UID: \"ccb37f0a-ea64-4cd6-a238-9b012bcd7e03\") " pod="ai-tenant-e2e-shared-b-5865e5/rate-test-model-5865e5-kserve-55f6f65947-s7c94" Jul 12 20:02:03.429618 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:02:03.429542 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ccb37f0a-ea64-4cd6-a238-9b012bcd7e03-tls-certs\") pod \"rate-test-model-5865e5-kserve-55f6f65947-s7c94\" (UID: \"ccb37f0a-ea64-4cd6-a238-9b012bcd7e03\") " pod="ai-tenant-e2e-shared-b-5865e5/rate-test-model-5865e5-kserve-55f6f65947-s7c94" Jul 12 20:02:03.429618 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:02:03.429582 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ccb37f0a-ea64-4cd6-a238-9b012bcd7e03-model-cache\") pod \"rate-test-model-5865e5-kserve-55f6f65947-s7c94\" (UID: \"ccb37f0a-ea64-4cd6-a238-9b012bcd7e03\") " pod="ai-tenant-e2e-shared-b-5865e5/rate-test-model-5865e5-kserve-55f6f65947-s7c94" Jul 12 20:02:03.429962 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:02:03.429939 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ccb37f0a-ea64-4cd6-a238-9b012bcd7e03-tmp-dir\") pod \"rate-test-model-5865e5-kserve-55f6f65947-s7c94\" (UID: \"ccb37f0a-ea64-4cd6-a238-9b012bcd7e03\") " pod="ai-tenant-e2e-shared-b-5865e5/rate-test-model-5865e5-kserve-55f6f65947-s7c94" Jul 12 20:02:03.430037 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:02:03.429987 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ccb37f0a-ea64-4cd6-a238-9b012bcd7e03-home\") pod \"rate-test-model-5865e5-kserve-55f6f65947-s7c94\" (UID: \"ccb37f0a-ea64-4cd6-a238-9b012bcd7e03\") " pod="ai-tenant-e2e-shared-b-5865e5/rate-test-model-5865e5-kserve-55f6f65947-s7c94" Jul 12 20:02:03.430037 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:02:03.430004 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ccb37f0a-ea64-4cd6-a238-9b012bcd7e03-model-cache\") pod \"rate-test-model-5865e5-kserve-55f6f65947-s7c94\" (UID: \"ccb37f0a-ea64-4cd6-a238-9b012bcd7e03\") " pod="ai-tenant-e2e-shared-b-5865e5/rate-test-model-5865e5-kserve-55f6f65947-s7c94" Jul 12 20:02:03.431741 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:02:03.431716 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ccb37f0a-ea64-4cd6-a238-9b012bcd7e03-dshm\") pod \"rate-test-model-5865e5-kserve-55f6f65947-s7c94\" (UID: \"ccb37f0a-ea64-4cd6-a238-9b012bcd7e03\") " pod="ai-tenant-e2e-shared-b-5865e5/rate-test-model-5865e5-kserve-55f6f65947-s7c94" Jul 12 20:02:03.432012 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:02:03.431996 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ccb37f0a-ea64-4cd6-a238-9b012bcd7e03-tls-certs\") pod \"rate-test-model-5865e5-kserve-55f6f65947-s7c94\" (UID: \"ccb37f0a-ea64-4cd6-a238-9b012bcd7e03\") " pod="ai-tenant-e2e-shared-b-5865e5/rate-test-model-5865e5-kserve-55f6f65947-s7c94" Jul 12 20:02:03.436541 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:02:03.436516 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fq972\" (UniqueName: \"kubernetes.io/projected/ccb37f0a-ea64-4cd6-a238-9b012bcd7e03-kube-api-access-fq972\") pod \"rate-test-model-5865e5-kserve-55f6f65947-s7c94\" (UID: \"ccb37f0a-ea64-4cd6-a238-9b012bcd7e03\") " pod="ai-tenant-e2e-shared-b-5865e5/rate-test-model-5865e5-kserve-55f6f65947-s7c94" Jul 12 20:02:03.527496 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:02:03.527405 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-5865e5/rate-test-model-5865e5-kserve-55f6f65947-s7c94" Jul 12 20:02:03.662140 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:02:03.662103 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-5865e5/rate-test-model-5865e5-kserve-55f6f65947-s7c94"] Jul 12 20:02:03.665435 ip-10-0-139-170 kubenswrapper[2565]: W0712 20:02:03.665407 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podccb37f0a_ea64_4cd6_a238_9b012bcd7e03.slice/crio-64fa2677b921cce593944f36b327bdd3909816cc73bc563b3d37dc013958770a WatchSource:0}: Error finding container 64fa2677b921cce593944f36b327bdd3909816cc73bc563b3d37dc013958770a: Status 404 returned error can't find the container with id 64fa2677b921cce593944f36b327bdd3909816cc73bc563b3d37dc013958770a Jul 12 20:02:04.154253 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:02:04.154217 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-5865e5/rate-test-model-5865e5-kserve-55f6f65947-s7c94" event={"ID":"ccb37f0a-ea64-4cd6-a238-9b012bcd7e03","Type":"ContainerStarted","Data":"9c6af4dbcc240160dd14dedf6f331e13923d13b43f15ad9429a8d3883f975119"} Jul 12 20:02:04.154415 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:02:04.154263 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-5865e5/rate-test-model-5865e5-kserve-55f6f65947-s7c94" event={"ID":"ccb37f0a-ea64-4cd6-a238-9b012bcd7e03","Type":"ContainerStarted","Data":"64fa2677b921cce593944f36b327bdd3909816cc73bc563b3d37dc013958770a"} Jul 12 20:02:04.172040 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:02:04.171988 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-5865e5/rate-test-model-5865e5-kserve-55f6f65947-s7c94" podStartSLOduration=1.171972508 podStartE2EDuration="1.171972508s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 20:02:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 20:02:04.170132808 +0000 UTC m=+2388.902705299" watchObservedRunningTime="2026-07-12 20:02:04.171972508 +0000 UTC m=+2388.904544997" Jul 12 20:02:13.528312 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:02:13.528271 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-5865e5/rate-test-model-5865e5-kserve-55f6f65947-s7c94" Jul 12 20:02:13.528713 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:02:13.528336 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-5865e5/rate-test-model-5865e5-kserve-55f6f65947-s7c94" Jul 12 20:02:13.540301 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:02:13.540271 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-5865e5/rate-test-model-5865e5-kserve-55f6f65947-s7c94" Jul 12 20:02:14.209825 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:02:14.209796 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-5865e5/rate-test-model-5865e5-kserve-55f6f65947-s7c94" Jul 12 20:02:15.876276 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:02:15.876247 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k5h9m_8e016c2e-0c3a-49da-83d4-f526738c1cbd/ovn-acl-logging/0.log" Jul 12 20:02:15.880454 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:02:15.880431 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k5h9m_8e016c2e-0c3a-49da-83d4-f526738c1cbd/ovn-acl-logging/0.log" Jul 12 20:02:48.721615 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:02:48.721578 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-e68438/rate-test-model-e68438-kserve-59d6c5554c-6t54g"] Jul 12 20:02:48.739126 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:02:48.739069 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 20:02:49.524842 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:02:49.524805 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-5865e5/rate-test-model-5865e5-kserve-55f6f65947-s7c94"] Jul 12 20:02:49.543011 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:02:49.542973 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 20:02:58.685118 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:02:58.685080 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 20:02:58.786229 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:02:58.786192 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 20:03:03.741216 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:03:03.741152 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-e68438/rate-test-model-e68438-kserve-59d6c5554c-6t54g" podUID="0c65a059-596e-4da4-af4b-d394c7b02699" containerName="main" containerID="cri-o://dcbece61070a467c4ddce696dd90b6655a63ec330dcb16c219e68dddfb86fc9a" gracePeriod=45 Jul 12 20:03:03.986132 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:03:03.986105 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-e68438/rate-test-model-e68438-kserve-59d6c5554c-6t54g" Jul 12 20:03:04.092276 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:03:04.092190 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c65a059-596e-4da4-af4b-d394c7b02699-dshm\") pod \"0c65a059-596e-4da4-af4b-d394c7b02699\" (UID: \"0c65a059-596e-4da4-af4b-d394c7b02699\") " Jul 12 20:03:04.092276 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:03:04.092243 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ks2n8\" (UniqueName: \"kubernetes.io/projected/0c65a059-596e-4da4-af4b-d394c7b02699-kube-api-access-ks2n8\") pod \"0c65a059-596e-4da4-af4b-d394c7b02699\" (UID: \"0c65a059-596e-4da4-af4b-d394c7b02699\") " Jul 12 20:03:04.092276 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:03:04.092260 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c65a059-596e-4da4-af4b-d394c7b02699-model-cache\") pod \"0c65a059-596e-4da4-af4b-d394c7b02699\" (UID: \"0c65a059-596e-4da4-af4b-d394c7b02699\") " Jul 12 20:03:04.092553 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:03:04.092288 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c65a059-596e-4da4-af4b-d394c7b02699-tls-certs\") pod \"0c65a059-596e-4da4-af4b-d394c7b02699\" (UID: \"0c65a059-596e-4da4-af4b-d394c7b02699\") " Jul 12 20:03:04.092553 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:03:04.092312 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0c65a059-596e-4da4-af4b-d394c7b02699-home\") pod \"0c65a059-596e-4da4-af4b-d394c7b02699\" (UID: \"0c65a059-596e-4da4-af4b-d394c7b02699\") " Jul 12 20:03:04.092553 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:03:04.092327 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c65a059-596e-4da4-af4b-d394c7b02699-tmp-dir\") pod \"0c65a059-596e-4da4-af4b-d394c7b02699\" (UID: \"0c65a059-596e-4da4-af4b-d394c7b02699\") " Jul 12 20:03:04.092712 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:03:04.092591 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c65a059-596e-4da4-af4b-d394c7b02699-home" (OuterVolumeSpecName: "home") pod "0c65a059-596e-4da4-af4b-d394c7b02699" (UID: "0c65a059-596e-4da4-af4b-d394c7b02699"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 20:03:04.092712 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:03:04.092624 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c65a059-596e-4da4-af4b-d394c7b02699-model-cache" (OuterVolumeSpecName: "model-cache") pod "0c65a059-596e-4da4-af4b-d394c7b02699" (UID: "0c65a059-596e-4da4-af4b-d394c7b02699"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 20:03:04.092795 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:03:04.092710 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c65a059-596e-4da4-af4b-d394c7b02699-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0c65a059-596e-4da4-af4b-d394c7b02699" (UID: "0c65a059-596e-4da4-af4b-d394c7b02699"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 20:03:04.092795 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:03:04.092765 2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c65a059-596e-4da4-af4b-d394c7b02699-model-cache\") on node \"ip-10-0-139-170.ec2.internal\" DevicePath \"\"" Jul 12 20:03:04.092795 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:03:04.092785 2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0c65a059-596e-4da4-af4b-d394c7b02699-home\") on node \"ip-10-0-139-170.ec2.internal\" DevicePath \"\"" Jul 12 20:03:04.094423 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:03:04.094398 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c65a059-596e-4da4-af4b-d394c7b02699-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0c65a059-596e-4da4-af4b-d394c7b02699" (UID: "0c65a059-596e-4da4-af4b-d394c7b02699"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 12 20:03:04.094514 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:03:04.094448 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c65a059-596e-4da4-af4b-d394c7b02699-dshm" (OuterVolumeSpecName: "dshm") pod "0c65a059-596e-4da4-af4b-d394c7b02699" (UID: "0c65a059-596e-4da4-af4b-d394c7b02699"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 20:03:04.094514 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:03:04.094450 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c65a059-596e-4da4-af4b-d394c7b02699-kube-api-access-ks2n8" (OuterVolumeSpecName: "kube-api-access-ks2n8") pod "0c65a059-596e-4da4-af4b-d394c7b02699" (UID: "0c65a059-596e-4da4-af4b-d394c7b02699"). InnerVolumeSpecName "kube-api-access-ks2n8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 12 20:03:04.193503 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:03:04.193471 2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c65a059-596e-4da4-af4b-d394c7b02699-dshm\") on node \"ip-10-0-139-170.ec2.internal\" DevicePath \"\"" Jul 12 20:03:04.193503 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:03:04.193501 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ks2n8\" (UniqueName: \"kubernetes.io/projected/0c65a059-596e-4da4-af4b-d394c7b02699-kube-api-access-ks2n8\") on node \"ip-10-0-139-170.ec2.internal\" DevicePath \"\"" Jul 12 20:03:04.193701 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:03:04.193518 2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c65a059-596e-4da4-af4b-d394c7b02699-tls-certs\") on node \"ip-10-0-139-170.ec2.internal\" DevicePath \"\"" Jul 12 20:03:04.193701 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:03:04.193530 2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c65a059-596e-4da4-af4b-d394c7b02699-tmp-dir\") on node \"ip-10-0-139-170.ec2.internal\" DevicePath \"\"" Jul 12 20:03:04.424079 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:03:04.424048 2565 generic.go:356] "Generic (PLEG): container finished" podID="0c65a059-596e-4da4-af4b-d394c7b02699" containerID="dcbece61070a467c4ddce696dd90b6655a63ec330dcb16c219e68dddfb86fc9a" exitCode=0 Jul 12 20:03:04.424267 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:03:04.424090 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-e68438/rate-test-model-e68438-kserve-59d6c5554c-6t54g" Jul 12 20:03:04.424267 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:03:04.424135 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-e68438/rate-test-model-e68438-kserve-59d6c5554c-6t54g" event={"ID":"0c65a059-596e-4da4-af4b-d394c7b02699","Type":"ContainerDied","Data":"dcbece61070a467c4ddce696dd90b6655a63ec330dcb16c219e68dddfb86fc9a"} Jul 12 20:03:04.424267 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:03:04.424178 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-e68438/rate-test-model-e68438-kserve-59d6c5554c-6t54g" event={"ID":"0c65a059-596e-4da4-af4b-d394c7b02699","Type":"ContainerDied","Data":"43f6a2bb5696b05b74c22e142c0622ea1c10ac53c755ea1300b4c07181ef119c"} Jul 12 20:03:04.424267 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:03:04.424196 2565 scope.go:117] "RemoveContainer" containerID="dcbece61070a467c4ddce696dd90b6655a63ec330dcb16c219e68dddfb86fc9a" Jul 12 20:03:04.432922 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:03:04.432904 2565 scope.go:117] "RemoveContainer" containerID="dcbece61070a467c4ddce696dd90b6655a63ec330dcb16c219e68dddfb86fc9a" Jul 12 20:03:04.433182 ip-10-0-139-170 kubenswrapper[2565]: E0712 20:03:04.433162 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dcbece61070a467c4ddce696dd90b6655a63ec330dcb16c219e68dddfb86fc9a\": container with ID starting with dcbece61070a467c4ddce696dd90b6655a63ec330dcb16c219e68dddfb86fc9a not found: ID does not exist" containerID="dcbece61070a467c4ddce696dd90b6655a63ec330dcb16c219e68dddfb86fc9a" Jul 12 20:03:04.433226 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:03:04.433195 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dcbece61070a467c4ddce696dd90b6655a63ec330dcb16c219e68dddfb86fc9a"} err="failed to get container status \"dcbece61070a467c4ddce696dd90b6655a63ec330dcb16c219e68dddfb86fc9a\": rpc error: code = NotFound desc = could not find container \"dcbece61070a467c4ddce696dd90b6655a63ec330dcb16c219e68dddfb86fc9a\": container with ID starting with dcbece61070a467c4ddce696dd90b6655a63ec330dcb16c219e68dddfb86fc9a not found: ID does not exist" Jul 12 20:03:04.444415 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:03:04.444390 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-e68438/rate-test-model-e68438-kserve-59d6c5554c-6t54g"] Jul 12 20:03:04.447806 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:03:04.447786 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-e68438/rate-test-model-e68438-kserve-59d6c5554c-6t54g"] Jul 12 20:03:04.545087 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:03:04.545043 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-5865e5/rate-test-model-5865e5-kserve-55f6f65947-s7c94" podUID="ccb37f0a-ea64-4cd6-a238-9b012bcd7e03" containerName="main" containerID="cri-o://9c6af4dbcc240160dd14dedf6f331e13923d13b43f15ad9429a8d3883f975119" gracePeriod=45 Jul 12 20:03:04.784836 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:03:04.784812 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-5865e5/rate-test-model-5865e5-kserve-55f6f65947-s7c94" Jul 12 20:03:04.900109 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:03:04.900069 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ccb37f0a-ea64-4cd6-a238-9b012bcd7e03-model-cache\") pod \"ccb37f0a-ea64-4cd6-a238-9b012bcd7e03\" (UID: \"ccb37f0a-ea64-4cd6-a238-9b012bcd7e03\") " Jul 12 20:03:04.900109 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:03:04.900109 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ccb37f0a-ea64-4cd6-a238-9b012bcd7e03-dshm\") pod \"ccb37f0a-ea64-4cd6-a238-9b012bcd7e03\" (UID: \"ccb37f0a-ea64-4cd6-a238-9b012bcd7e03\") " Jul 12 20:03:04.900369 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:03:04.900166 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ccb37f0a-ea64-4cd6-a238-9b012bcd7e03-home\") pod \"ccb37f0a-ea64-4cd6-a238-9b012bcd7e03\" (UID: \"ccb37f0a-ea64-4cd6-a238-9b012bcd7e03\") " Jul 12 20:03:04.900369 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:03:04.900189 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fq972\" (UniqueName: \"kubernetes.io/projected/ccb37f0a-ea64-4cd6-a238-9b012bcd7e03-kube-api-access-fq972\") pod \"ccb37f0a-ea64-4cd6-a238-9b012bcd7e03\" (UID: \"ccb37f0a-ea64-4cd6-a238-9b012bcd7e03\") " Jul 12 20:03:04.900369 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:03:04.900213 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ccb37f0a-ea64-4cd6-a238-9b012bcd7e03-tls-certs\") pod \"ccb37f0a-ea64-4cd6-a238-9b012bcd7e03\" (UID: \"ccb37f0a-ea64-4cd6-a238-9b012bcd7e03\") " Jul 12 20:03:04.900369 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:03:04.900303 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ccb37f0a-ea64-4cd6-a238-9b012bcd7e03-tmp-dir\") pod \"ccb37f0a-ea64-4cd6-a238-9b012bcd7e03\" (UID: \"ccb37f0a-ea64-4cd6-a238-9b012bcd7e03\") " Jul 12 20:03:04.900532 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:03:04.900397 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccb37f0a-ea64-4cd6-a238-9b012bcd7e03-model-cache" (OuterVolumeSpecName: "model-cache") pod "ccb37f0a-ea64-4cd6-a238-9b012bcd7e03" (UID: "ccb37f0a-ea64-4cd6-a238-9b012bcd7e03"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 20:03:04.900532 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:03:04.900423 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccb37f0a-ea64-4cd6-a238-9b012bcd7e03-home" (OuterVolumeSpecName: "home") pod "ccb37f0a-ea64-4cd6-a238-9b012bcd7e03" (UID: "ccb37f0a-ea64-4cd6-a238-9b012bcd7e03"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 20:03:04.900630 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:03:04.900593 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccb37f0a-ea64-4cd6-a238-9b012bcd7e03-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ccb37f0a-ea64-4cd6-a238-9b012bcd7e03" (UID: "ccb37f0a-ea64-4cd6-a238-9b012bcd7e03"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 20:03:04.900670 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:03:04.900639 2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ccb37f0a-ea64-4cd6-a238-9b012bcd7e03-model-cache\") on node \"ip-10-0-139-170.ec2.internal\" DevicePath \"\"" Jul 12 20:03:04.900670 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:03:04.900659 2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ccb37f0a-ea64-4cd6-a238-9b012bcd7e03-home\") on node \"ip-10-0-139-170.ec2.internal\" DevicePath \"\"" Jul 12 20:03:04.902376 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:03:04.902352 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ccb37f0a-ea64-4cd6-a238-9b012bcd7e03-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ccb37f0a-ea64-4cd6-a238-9b012bcd7e03" (UID: "ccb37f0a-ea64-4cd6-a238-9b012bcd7e03"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 12 20:03:04.902505 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:03:04.902413 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ccb37f0a-ea64-4cd6-a238-9b012bcd7e03-kube-api-access-fq972" (OuterVolumeSpecName: "kube-api-access-fq972") pod "ccb37f0a-ea64-4cd6-a238-9b012bcd7e03" (UID: "ccb37f0a-ea64-4cd6-a238-9b012bcd7e03"). InnerVolumeSpecName "kube-api-access-fq972". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 12 20:03:04.902505 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:03:04.902439 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccb37f0a-ea64-4cd6-a238-9b012bcd7e03-dshm" (OuterVolumeSpecName: "dshm") pod "ccb37f0a-ea64-4cd6-a238-9b012bcd7e03" (UID: "ccb37f0a-ea64-4cd6-a238-9b012bcd7e03"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 20:03:05.001545 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:03:05.001463 2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ccb37f0a-ea64-4cd6-a238-9b012bcd7e03-dshm\") on node \"ip-10-0-139-170.ec2.internal\" DevicePath \"\"" Jul 12 20:03:05.001545 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:03:05.001491 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fq972\" (UniqueName: \"kubernetes.io/projected/ccb37f0a-ea64-4cd6-a238-9b012bcd7e03-kube-api-access-fq972\") on node \"ip-10-0-139-170.ec2.internal\" DevicePath \"\"" Jul 12 20:03:05.001545 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:03:05.001501 2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ccb37f0a-ea64-4cd6-a238-9b012bcd7e03-tls-certs\") on node \"ip-10-0-139-170.ec2.internal\" DevicePath \"\"" Jul 12 20:03:05.001545 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:03:05.001509 2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ccb37f0a-ea64-4cd6-a238-9b012bcd7e03-tmp-dir\") on node \"ip-10-0-139-170.ec2.internal\" DevicePath \"\"" Jul 12 20:03:05.430155 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:03:05.430113 2565 generic.go:356] "Generic (PLEG): container finished" podID="ccb37f0a-ea64-4cd6-a238-9b012bcd7e03" containerID="9c6af4dbcc240160dd14dedf6f331e13923d13b43f15ad9429a8d3883f975119" exitCode=0 Jul 12 20:03:05.430357 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:03:05.430168 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-5865e5/rate-test-model-5865e5-kserve-55f6f65947-s7c94" Jul 12 20:03:05.430357 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:03:05.430204 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-5865e5/rate-test-model-5865e5-kserve-55f6f65947-s7c94" event={"ID":"ccb37f0a-ea64-4cd6-a238-9b012bcd7e03","Type":"ContainerDied","Data":"9c6af4dbcc240160dd14dedf6f331e13923d13b43f15ad9429a8d3883f975119"} Jul 12 20:03:05.430357 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:03:05.430251 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-5865e5/rate-test-model-5865e5-kserve-55f6f65947-s7c94" event={"ID":"ccb37f0a-ea64-4cd6-a238-9b012bcd7e03","Type":"ContainerDied","Data":"64fa2677b921cce593944f36b327bdd3909816cc73bc563b3d37dc013958770a"} Jul 12 20:03:05.430357 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:03:05.430269 2565 scope.go:117] "RemoveContainer" containerID="9c6af4dbcc240160dd14dedf6f331e13923d13b43f15ad9429a8d3883f975119" Jul 12 20:03:05.439057 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:03:05.439040 2565 scope.go:117] "RemoveContainer" containerID="9c6af4dbcc240160dd14dedf6f331e13923d13b43f15ad9429a8d3883f975119" Jul 12 20:03:05.439289 ip-10-0-139-170 kubenswrapper[2565]: E0712 20:03:05.439271 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c6af4dbcc240160dd14dedf6f331e13923d13b43f15ad9429a8d3883f975119\": container with ID starting with 9c6af4dbcc240160dd14dedf6f331e13923d13b43f15ad9429a8d3883f975119 not found: ID does not exist" containerID="9c6af4dbcc240160dd14dedf6f331e13923d13b43f15ad9429a8d3883f975119" Jul 12 20:03:05.439332 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:03:05.439298 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c6af4dbcc240160dd14dedf6f331e13923d13b43f15ad9429a8d3883f975119"} err="failed to get container status \"9c6af4dbcc240160dd14dedf6f331e13923d13b43f15ad9429a8d3883f975119\": rpc error: code = NotFound desc = could not find container \"9c6af4dbcc240160dd14dedf6f331e13923d13b43f15ad9429a8d3883f975119\": container with ID starting with 9c6af4dbcc240160dd14dedf6f331e13923d13b43f15ad9429a8d3883f975119 not found: ID does not exist" Jul 12 20:03:05.450171 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:03:05.450144 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-5865e5/rate-test-model-5865e5-kserve-55f6f65947-s7c94"] Jul 12 20:03:05.455579 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:03:05.455555 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-5865e5/rate-test-model-5865e5-kserve-55f6f65947-s7c94"] Jul 12 20:03:05.857947 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:03:05.857852 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0c65a059-596e-4da4-af4b-d394c7b02699" path="/var/lib/kubelet/pods/0c65a059-596e-4da4-af4b-d394c7b02699/volumes" Jul 12 20:03:05.858294 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:03:05.858223 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ccb37f0a-ea64-4cd6-a238-9b012bcd7e03" path="/var/lib/kubelet/pods/ccb37f0a-ea64-4cd6-a238-9b012bcd7e03/volumes" Jul 12 20:03:06.242678 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:03:06.242648 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 20:03:06.343354 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:03:06.343319 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 20:03:15.985247 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:03:15.985214 2565 scope.go:117] "RemoveContainer" containerID="995397c55681d05fb78ccf031ea83d895e378ff20912bbf3be6811ed0e106426" Jul 12 20:03:39.125842 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:03:39.125808 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 20:03:39.173239 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:03:39.173208 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 20:03:58.036790 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:03:58.036753 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 20:03:58.478455 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:03:58.478420 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 20:03:58.578540 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:03:58.578505 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 20:03:58.678340 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:03:58.678307 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 20:03:58.743138 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:03:58.743060 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 20:04:04.205231 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:04:04.205192 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 20:04:04.304942 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:04:04.304901 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 20:04:04.439916 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:04:04.439868 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-e68438-openshift-default-d95d995-95pg5"] Jul 12 20:04:04.440150 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:04:04.440080 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-e68438-openshift-default-d95d995-95pg5" podUID="1e0fbb77-4b94-4dc0-9cb1-60ec64311952" containerName="istio-proxy" containerID="cri-o://d51c41bda76230c0458c14ced26824f00a2b93066d1a8469f022f60a20ed25a1" gracePeriod=30 Jul 12 20:04:04.462789 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:04:04.462711 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 20:04:04.754722 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:04:04.754637 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 20:04:04.869033 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:04:04.869005 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 20:04:05.271273 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:04:05.271238 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 20:04:05.371206 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:04:05.371163 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 20:04:05.486121 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:04:05.486091 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 20:04:09.500438 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:04:09.500405 2565 patch_prober.go:28] interesting pod/e2e-shared-a-e68438-openshift-default-d95d995-95pg5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Jul 12 20:04:09.500818 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:04:09.500463 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-e68438-openshift-default-d95d995-95pg5" podUID="1e0fbb77-4b94-4dc0-9cb1-60ec64311952" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Jul 12 20:04:09.691912 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:04:09.691874 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-e68438-openshift-default-d95d995-95pg5" Jul 12 20:04:09.716384 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:04:09.716301 2565 generic.go:356] "Generic (PLEG): container finished" podID="1e0fbb77-4b94-4dc0-9cb1-60ec64311952" containerID="d51c41bda76230c0458c14ced26824f00a2b93066d1a8469f022f60a20ed25a1" exitCode=0 Jul 12 20:04:09.716384 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:04:09.716349 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-e68438-openshift-default-d95d995-95pg5" Jul 12 20:04:09.716617 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:04:09.716379 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-e68438-openshift-default-d95d995-95pg5" event={"ID":"1e0fbb77-4b94-4dc0-9cb1-60ec64311952","Type":"ContainerDied","Data":"d51c41bda76230c0458c14ced26824f00a2b93066d1a8469f022f60a20ed25a1"} Jul 12 20:04:09.716617 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:04:09.716441 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-e68438-openshift-default-d95d995-95pg5" event={"ID":"1e0fbb77-4b94-4dc0-9cb1-60ec64311952","Type":"ContainerDied","Data":"5923e9754b57be4f2cf88c6ae5b7ad2363fde8f79f5dc90bb7a1e361fae43f6f"} Jul 12 20:04:09.716617 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:04:09.716473 2565 scope.go:117] "RemoveContainer" containerID="d51c41bda76230c0458c14ced26824f00a2b93066d1a8469f022f60a20ed25a1" Jul 12 20:04:09.724858 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:04:09.724840 2565 scope.go:117] "RemoveContainer" containerID="8faf52477a5aa92e322f8cec9fac6834497d87684719560c25c30d91af2c43b0" Jul 12 20:04:09.734178 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:04:09.734143 2565 scope.go:117] "RemoveContainer" containerID="d51c41bda76230c0458c14ced26824f00a2b93066d1a8469f022f60a20ed25a1" Jul 12 20:04:09.734522 ip-10-0-139-170 kubenswrapper[2565]: E0712 20:04:09.734500 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d51c41bda76230c0458c14ced26824f00a2b93066d1a8469f022f60a20ed25a1\": container with ID starting with d51c41bda76230c0458c14ced26824f00a2b93066d1a8469f022f60a20ed25a1 not found: ID does not exist" containerID="d51c41bda76230c0458c14ced26824f00a2b93066d1a8469f022f60a20ed25a1" Jul 12 20:04:09.734627 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:04:09.734530 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d51c41bda76230c0458c14ced26824f00a2b93066d1a8469f022f60a20ed25a1"} err="failed to get container status \"d51c41bda76230c0458c14ced26824f00a2b93066d1a8469f022f60a20ed25a1\": rpc error: code = NotFound desc = could not find container \"d51c41bda76230c0458c14ced26824f00a2b93066d1a8469f022f60a20ed25a1\": container with ID starting with d51c41bda76230c0458c14ced26824f00a2b93066d1a8469f022f60a20ed25a1 not found: ID does not exist" Jul 12 20:04:09.734627 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:04:09.734550 2565 scope.go:117] "RemoveContainer" containerID="8faf52477a5aa92e322f8cec9fac6834497d87684719560c25c30d91af2c43b0" Jul 12 20:04:09.734834 ip-10-0-139-170 kubenswrapper[2565]: E0712 20:04:09.734813 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8faf52477a5aa92e322f8cec9fac6834497d87684719560c25c30d91af2c43b0\": container with ID starting with 8faf52477a5aa92e322f8cec9fac6834497d87684719560c25c30d91af2c43b0 not found: ID does not exist" containerID="8faf52477a5aa92e322f8cec9fac6834497d87684719560c25c30d91af2c43b0" Jul 12 20:04:09.734925 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:04:09.734847 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8faf52477a5aa92e322f8cec9fac6834497d87684719560c25c30d91af2c43b0"} err="failed to get container status \"8faf52477a5aa92e322f8cec9fac6834497d87684719560c25c30d91af2c43b0\": rpc error: code = NotFound desc = could not find container \"8faf52477a5aa92e322f8cec9fac6834497d87684719560c25c30d91af2c43b0\": container with ID starting with 8faf52477a5aa92e322f8cec9fac6834497d87684719560c25c30d91af2c43b0 not found: ID does not exist" Jul 12 20:04:09.771230 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:04:09.771208 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e0fbb77-4b94-4dc0-9cb1-60ec64311952-istio-data\") pod \"1e0fbb77-4b94-4dc0-9cb1-60ec64311952\" (UID: \"1e0fbb77-4b94-4dc0-9cb1-60ec64311952\") " Jul 12 20:04:09.771351 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:04:09.771242 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e0fbb77-4b94-4dc0-9cb1-60ec64311952-workload-socket\") pod \"1e0fbb77-4b94-4dc0-9cb1-60ec64311952\" (UID: \"1e0fbb77-4b94-4dc0-9cb1-60ec64311952\") " Jul 12 20:04:09.771351 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:04:09.771271 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e0fbb77-4b94-4dc0-9cb1-60ec64311952-istio-envoy\") pod \"1e0fbb77-4b94-4dc0-9cb1-60ec64311952\" (UID: \"1e0fbb77-4b94-4dc0-9cb1-60ec64311952\") " Jul 12 20:04:09.771351 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:04:09.771294 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6kh7f\" (UniqueName: \"kubernetes.io/projected/1e0fbb77-4b94-4dc0-9cb1-60ec64311952-kube-api-access-6kh7f\") pod \"1e0fbb77-4b94-4dc0-9cb1-60ec64311952\" (UID: \"1e0fbb77-4b94-4dc0-9cb1-60ec64311952\") " Jul 12 20:04:09.771351 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:04:09.771331 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e0fbb77-4b94-4dc0-9cb1-60ec64311952-istio-podinfo\") pod \"1e0fbb77-4b94-4dc0-9cb1-60ec64311952\" (UID: \"1e0fbb77-4b94-4dc0-9cb1-60ec64311952\") " Jul 12 20:04:09.771555 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:04:09.771359 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e0fbb77-4b94-4dc0-9cb1-60ec64311952-credential-socket\") pod \"1e0fbb77-4b94-4dc0-9cb1-60ec64311952\" (UID: \"1e0fbb77-4b94-4dc0-9cb1-60ec64311952\") " Jul 12 20:04:09.771555 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:04:09.771393 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e0fbb77-4b94-4dc0-9cb1-60ec64311952-istio-token\") pod \"1e0fbb77-4b94-4dc0-9cb1-60ec64311952\" (UID: \"1e0fbb77-4b94-4dc0-9cb1-60ec64311952\") " Jul 12 20:04:09.771555 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:04:09.771417 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e0fbb77-4b94-4dc0-9cb1-60ec64311952-istiod-ca-cert\") pod \"1e0fbb77-4b94-4dc0-9cb1-60ec64311952\" (UID: \"1e0fbb77-4b94-4dc0-9cb1-60ec64311952\") " Jul 12 20:04:09.771555 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:04:09.771500 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e0fbb77-4b94-4dc0-9cb1-60ec64311952-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "1e0fbb77-4b94-4dc0-9cb1-60ec64311952" (UID: "1e0fbb77-4b94-4dc0-9cb1-60ec64311952"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 20:04:09.771555 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:04:09.771503 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e0fbb77-4b94-4dc0-9cb1-60ec64311952-workload-certs\") pod \"1e0fbb77-4b94-4dc0-9cb1-60ec64311952\" (UID: \"1e0fbb77-4b94-4dc0-9cb1-60ec64311952\") " Jul 12 20:04:09.771803 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:04:09.771737 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e0fbb77-4b94-4dc0-9cb1-60ec64311952-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "1e0fbb77-4b94-4dc0-9cb1-60ec64311952" (UID: "1e0fbb77-4b94-4dc0-9cb1-60ec64311952"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 20:04:09.771877 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:04:09.771851 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e0fbb77-4b94-4dc0-9cb1-60ec64311952-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "1e0fbb77-4b94-4dc0-9cb1-60ec64311952" (UID: "1e0fbb77-4b94-4dc0-9cb1-60ec64311952"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 12 20:04:09.772022 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:04:09.771954 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e0fbb77-4b94-4dc0-9cb1-60ec64311952-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "1e0fbb77-4b94-4dc0-9cb1-60ec64311952" (UID: "1e0fbb77-4b94-4dc0-9cb1-60ec64311952"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 20:04:09.772022 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:04:09.771953 2565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e0fbb77-4b94-4dc0-9cb1-60ec64311952-workload-certs\") on node \"ip-10-0-139-170.ec2.internal\" DevicePath \"\"" Jul 12 20:04:09.772022 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:04:09.772016 2565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e0fbb77-4b94-4dc0-9cb1-60ec64311952-workload-socket\") on node \"ip-10-0-139-170.ec2.internal\" DevicePath \"\"" Jul 12 20:04:09.772652 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:04:09.772624 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e0fbb77-4b94-4dc0-9cb1-60ec64311952-istio-data" (OuterVolumeSpecName: "istio-data") pod "1e0fbb77-4b94-4dc0-9cb1-60ec64311952" (UID: "1e0fbb77-4b94-4dc0-9cb1-60ec64311952"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 20:04:09.773553 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:04:09.773531 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e0fbb77-4b94-4dc0-9cb1-60ec64311952-kube-api-access-6kh7f" (OuterVolumeSpecName: "kube-api-access-6kh7f") pod "1e0fbb77-4b94-4dc0-9cb1-60ec64311952" (UID: "1e0fbb77-4b94-4dc0-9cb1-60ec64311952"). InnerVolumeSpecName "kube-api-access-6kh7f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 12 20:04:09.773707 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:04:09.773685 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e0fbb77-4b94-4dc0-9cb1-60ec64311952-istio-token" (OuterVolumeSpecName: "istio-token") pod "1e0fbb77-4b94-4dc0-9cb1-60ec64311952" (UID: "1e0fbb77-4b94-4dc0-9cb1-60ec64311952"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 12 20:04:09.773821 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:04:09.773802 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e0fbb77-4b94-4dc0-9cb1-60ec64311952-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "1e0fbb77-4b94-4dc0-9cb1-60ec64311952" (UID: "1e0fbb77-4b94-4dc0-9cb1-60ec64311952"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 20:04:09.773876 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:04:09.773808 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1e0fbb77-4b94-4dc0-9cb1-60ec64311952-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "1e0fbb77-4b94-4dc0-9cb1-60ec64311952" (UID: "1e0fbb77-4b94-4dc0-9cb1-60ec64311952"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 12 20:04:09.872996 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:04:09.872975 2565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e0fbb77-4b94-4dc0-9cb1-60ec64311952-istio-data\") on node \"ip-10-0-139-170.ec2.internal\" DevicePath \"\"" Jul 12 20:04:09.872996 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:04:09.872996 2565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e0fbb77-4b94-4dc0-9cb1-60ec64311952-istio-envoy\") on node \"ip-10-0-139-170.ec2.internal\" DevicePath \"\"" Jul 12 20:04:09.873127 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:04:09.873005 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6kh7f\" (UniqueName: \"kubernetes.io/projected/1e0fbb77-4b94-4dc0-9cb1-60ec64311952-kube-api-access-6kh7f\") on node \"ip-10-0-139-170.ec2.internal\" DevicePath \"\"" Jul 12 20:04:09.873127 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:04:09.873014 2565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e0fbb77-4b94-4dc0-9cb1-60ec64311952-istio-podinfo\") on node \"ip-10-0-139-170.ec2.internal\" DevicePath \"\"" Jul 12 20:04:09.873127 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:04:09.873022 2565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e0fbb77-4b94-4dc0-9cb1-60ec64311952-credential-socket\") on node \"ip-10-0-139-170.ec2.internal\" DevicePath \"\"" Jul 12 20:04:09.873127 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:04:09.873030 2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e0fbb77-4b94-4dc0-9cb1-60ec64311952-istio-token\") on node \"ip-10-0-139-170.ec2.internal\" DevicePath \"\"" Jul 12 20:04:09.873127 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:04:09.873038 2565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e0fbb77-4b94-4dc0-9cb1-60ec64311952-istiod-ca-cert\") on node \"ip-10-0-139-170.ec2.internal\" DevicePath \"\"" Jul 12 20:04:10.031467 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:04:10.031390 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-e68438-openshift-default-d95d995-95pg5"] Jul 12 20:04:10.036803 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:04:10.036771 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-e68438-openshift-default-d95d995-95pg5"] Jul 12 20:04:11.192683 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:04:11.192645 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 20:04:11.256939 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:04:11.256879 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 20:04:11.442430 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:04:11.442394 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 20:04:11.657595 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:04:11.657556 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 20:04:11.857061 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:04:11.857026 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1e0fbb77-4b94-4dc0-9cb1-60ec64311952" path="/var/lib/kubelet/pods/1e0fbb77-4b94-4dc0-9cb1-60ec64311952/volumes" Jul 12 20:04:17.108907 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:04:17.108855 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-4jntx_352106d4-8ce4-442d-a232-416311e4b0a1/server/0.log" Jul 12 20:04:17.336351 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:04:17.336317 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5449d7969f-kr7sd_c46c3317-ba7f-493b-a751-961cfa421b5c/copy-manifests/0.log" Jul 12 20:04:17.342629 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:04:17.342605 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5449d7969f-kr7sd_c46c3317-ba7f-493b-a751-961cfa421b5c/manager/0.log" Jul 12 20:04:20.060983 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:04:20.060954 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-7f964fb678c9jcd_d445ad9c-2f45-4808-989a-d988ef1dbaff/istio-proxy/0.log" Jul 12 20:04:20.170509 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:04:20.170476 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-vgbw9_4f668c0d-a883-4283-bea4-fb5f56cd1481/discovery/0.log" Jul 12 20:04:20.280214 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:04:20.280185 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-654967888b-nbxf9_f94b7de3-7cb4-4346-87eb-1ac4dfc83cc0/kube-auth-proxy/0.log" Jul 12 20:04:21.600169 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:04:21.600137 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-sshcb_41da0a81-53dc-45a9-a44e-4b3b1acdd405/main/0.log" Jul 12 20:04:26.082787 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:04:26.082747 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 20:04:26.149290 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:04:26.149263 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" limit=1024 Jul 12 20:04:28.996236 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:04:28.996204 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-288bq_a7a0abe6-09ba-429e-a7f2-8de5b3f5d620/global-pull-secret-syncer/0.log" Jul 12 20:04:29.120481 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:04:29.120445 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-7vvbp_26ba017f-506c-4e00-9b24-4932c97ad489/konnectivity-agent/0.log" Jul 12 20:04:29.231903 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:04:29.231864 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-139-170.ec2.internal_2a4584a3f0966be0c3f50f97c2a5e31a/haproxy/0.log" Jul 12 20:04:36.798421 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:04:36.798345 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-785778ff74-vdl5x_2b24f43c-e196-4d40-a20b-469489818be0/kube-state-metrics/0.log" Jul 12 20:04:36.820261 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:04:36.820242 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-785778ff74-vdl5x_2b24f43c-e196-4d40-a20b-469489818be0/kube-rbac-proxy-main/0.log" Jul 12 20:04:36.851382 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:04:36.851357 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-785778ff74-vdl5x_2b24f43c-e196-4d40-a20b-469489818be0/kube-rbac-proxy-self/0.log" Jul 12 20:04:36.877711 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:04:36.877687 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-75f45dd54d-p5rzj_ef3fe7d8-3ba3-453d-b939-ba590b2c24a9/metrics-server/0.log" Jul 12 20:04:36.909067 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:04:36.909048 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-5f68df7f58-l5mf4_502d5209-08a9-431f-a3f0-ab55309635e1/monitoring-plugin/0.log" Jul 12 20:04:37.011559 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:04:37.011529 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-r5rxf_1d52b8f7-a1db-4ab3-bd17-7462950f76ae/node-exporter/0.log" Jul 12 20:04:37.032732 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:04:37.032704 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-r5rxf_1d52b8f7-a1db-4ab3-bd17-7462950f76ae/kube-rbac-proxy/0.log" Jul 12 20:04:37.052232 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:04:37.052168 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-r5rxf_1d52b8f7-a1db-4ab3-bd17-7462950f76ae/init-textfile/0.log" Jul 12 20:04:37.340791 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:04:37.340712 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_1588d7eb-6773-4bcb-bce3-cc83d47485b5/prometheus/0.log" Jul 12 20:04:37.360183 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:04:37.360156 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_1588d7eb-6773-4bcb-bce3-cc83d47485b5/config-reloader/0.log" Jul 12 20:04:37.387452 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:04:37.387429 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_1588d7eb-6773-4bcb-bce3-cc83d47485b5/thanos-sidecar/0.log" Jul 12 20:04:37.408086 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:04:37.408064 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_1588d7eb-6773-4bcb-bce3-cc83d47485b5/kube-rbac-proxy-web/0.log" Jul 12 20:04:37.430756 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:04:37.430736 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_1588d7eb-6773-4bcb-bce3-cc83d47485b5/kube-rbac-proxy/0.log" Jul 12 20:04:37.452094 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:04:37.452069 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_1588d7eb-6773-4bcb-bce3-cc83d47485b5/kube-rbac-proxy-thanos/0.log" Jul 12 20:04:37.473382 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:04:37.473358 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_1588d7eb-6773-4bcb-bce3-cc83d47485b5/init-config-reloader/0.log" Jul 12 20:04:37.502975 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:04:37.502948 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-64785b6675-vcm7j_c0d417af-5977-4ea9-b7d7-53532cf3f29c/prometheus-operator/0.log" Jul 12 20:04:37.520737 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:04:37.520709 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-64785b6675-vcm7j_c0d417af-5977-4ea9-b7d7-53532cf3f29c/kube-rbac-proxy/0.log" Jul 12 20:04:39.119053 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:04:39.119024 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-wqrhv/perf-node-gather-daemonset-zjpl5"] Jul 12 20:04:39.119430 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:04:39.119345 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0c65a059-596e-4da4-af4b-d394c7b02699" containerName="main" Jul 12 20:04:39.119430 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:04:39.119361 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="0c65a059-596e-4da4-af4b-d394c7b02699" containerName="main" Jul 12 20:04:39.119430 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:04:39.119389 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1e0fbb77-4b94-4dc0-9cb1-60ec64311952" containerName="istio-proxy" Jul 12 20:04:39.119430 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:04:39.119398 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="1e0fbb77-4b94-4dc0-9cb1-60ec64311952" containerName="istio-proxy" Jul 12 20:04:39.119430 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:04:39.119414 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ccb37f0a-ea64-4cd6-a238-9b012bcd7e03" containerName="main" Jul 12 20:04:39.119430 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:04:39.119422 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="ccb37f0a-ea64-4cd6-a238-9b012bcd7e03" containerName="main" Jul 12 20:04:39.119430 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:04:39.119435 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1e0fbb77-4b94-4dc0-9cb1-60ec64311952" containerName="istio-proxy" Jul 12 20:04:39.119761 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:04:39.119440 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="1e0fbb77-4b94-4dc0-9cb1-60ec64311952" containerName="istio-proxy" Jul 12 20:04:39.119761 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:04:39.119524 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="1e0fbb77-4b94-4dc0-9cb1-60ec64311952" containerName="istio-proxy" Jul 12 20:04:39.119761 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:04:39.119538 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="1e0fbb77-4b94-4dc0-9cb1-60ec64311952" containerName="istio-proxy" Jul 12 20:04:39.119761 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:04:39.119549 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="ccb37f0a-ea64-4cd6-a238-9b012bcd7e03" containerName="main" Jul 12 20:04:39.119761 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:04:39.119557 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="0c65a059-596e-4da4-af4b-d394c7b02699" containerName="main" Jul 12 20:04:39.122480 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:04:39.122461 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-wqrhv/perf-node-gather-daemonset-zjpl5" Jul 12 20:04:39.125051 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:04:39.125030 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-wqrhv\"/\"default-dockercfg-jbd6q\"" Jul 12 20:04:39.125995 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:04:39.125975 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-wqrhv\"/\"kube-root-ca.crt\"" Jul 12 20:04:39.126098 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:04:39.126018 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-wqrhv\"/\"openshift-service-ca.crt\"" Jul 12 20:04:39.128687 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:04:39.128668 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-wqrhv/perf-node-gather-daemonset-zjpl5"] Jul 12 20:04:39.219381 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:04:39.219347 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/3683d0cf-7bf2-45f9-b6ee-5fa4e52c020d-proc\") pod \"perf-node-gather-daemonset-zjpl5\" (UID: \"3683d0cf-7bf2-45f9-b6ee-5fa4e52c020d\") " pod="openshift-must-gather-wqrhv/perf-node-gather-daemonset-zjpl5" Jul 12 20:04:39.219555 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:04:39.219406 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66gt4\" (UniqueName: \"kubernetes.io/projected/3683d0cf-7bf2-45f9-b6ee-5fa4e52c020d-kube-api-access-66gt4\") pod \"perf-node-gather-daemonset-zjpl5\" (UID: \"3683d0cf-7bf2-45f9-b6ee-5fa4e52c020d\") " pod="openshift-must-gather-wqrhv/perf-node-gather-daemonset-zjpl5" Jul 12 20:04:39.219555 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:04:39.219481 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3683d0cf-7bf2-45f9-b6ee-5fa4e52c020d-sys\") pod \"perf-node-gather-daemonset-zjpl5\" (UID: \"3683d0cf-7bf2-45f9-b6ee-5fa4e52c020d\") " pod="openshift-must-gather-wqrhv/perf-node-gather-daemonset-zjpl5" Jul 12 20:04:39.219555 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:04:39.219549 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3683d0cf-7bf2-45f9-b6ee-5fa4e52c020d-lib-modules\") pod \"perf-node-gather-daemonset-zjpl5\" (UID: \"3683d0cf-7bf2-45f9-b6ee-5fa4e52c020d\") " pod="openshift-must-gather-wqrhv/perf-node-gather-daemonset-zjpl5" Jul 12 20:04:39.219667 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:04:39.219570 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/3683d0cf-7bf2-45f9-b6ee-5fa4e52c020d-podres\") pod \"perf-node-gather-daemonset-zjpl5\" (UID: \"3683d0cf-7bf2-45f9-b6ee-5fa4e52c020d\") " pod="openshift-must-gather-wqrhv/perf-node-gather-daemonset-zjpl5" Jul 12 20:04:39.320670 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:04:39.320637 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3683d0cf-7bf2-45f9-b6ee-5fa4e52c020d-lib-modules\") pod \"perf-node-gather-daemonset-zjpl5\" (UID: \"3683d0cf-7bf2-45f9-b6ee-5fa4e52c020d\") " pod="openshift-must-gather-wqrhv/perf-node-gather-daemonset-zjpl5" Jul 12 20:04:39.320670 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:04:39.320672 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/3683d0cf-7bf2-45f9-b6ee-5fa4e52c020d-podres\") pod \"perf-node-gather-daemonset-zjpl5\" (UID: \"3683d0cf-7bf2-45f9-b6ee-5fa4e52c020d\") " pod="openshift-must-gather-wqrhv/perf-node-gather-daemonset-zjpl5" Jul 12 20:04:39.320869 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:04:39.320794 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/3683d0cf-7bf2-45f9-b6ee-5fa4e52c020d-proc\") pod \"perf-node-gather-daemonset-zjpl5\" (UID: \"3683d0cf-7bf2-45f9-b6ee-5fa4e52c020d\") " pod="openshift-must-gather-wqrhv/perf-node-gather-daemonset-zjpl5" Jul 12 20:04:39.320869 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:04:39.320830 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/3683d0cf-7bf2-45f9-b6ee-5fa4e52c020d-proc\") pod \"perf-node-gather-daemonset-zjpl5\" (UID: \"3683d0cf-7bf2-45f9-b6ee-5fa4e52c020d\") " pod="openshift-must-gather-wqrhv/perf-node-gather-daemonset-zjpl5" Jul 12 20:04:39.320869 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:04:39.320799 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/3683d0cf-7bf2-45f9-b6ee-5fa4e52c020d-podres\") pod \"perf-node-gather-daemonset-zjpl5\" (UID: \"3683d0cf-7bf2-45f9-b6ee-5fa4e52c020d\") " pod="openshift-must-gather-wqrhv/perf-node-gather-daemonset-zjpl5" Jul 12 20:04:39.320869 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:04:39.320800 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3683d0cf-7bf2-45f9-b6ee-5fa4e52c020d-lib-modules\") pod \"perf-node-gather-daemonset-zjpl5\" (UID: \"3683d0cf-7bf2-45f9-b6ee-5fa4e52c020d\") " pod="openshift-must-gather-wqrhv/perf-node-gather-daemonset-zjpl5" Jul 12 20:04:39.320869 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:04:39.320861 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66gt4\" (UniqueName: \"kubernetes.io/projected/3683d0cf-7bf2-45f9-b6ee-5fa4e52c020d-kube-api-access-66gt4\") pod \"perf-node-gather-daemonset-zjpl5\" (UID: \"3683d0cf-7bf2-45f9-b6ee-5fa4e52c020d\") " pod="openshift-must-gather-wqrhv/perf-node-gather-daemonset-zjpl5" Jul 12 20:04:39.321067 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:04:39.320915 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3683d0cf-7bf2-45f9-b6ee-5fa4e52c020d-sys\") pod \"perf-node-gather-daemonset-zjpl5\" (UID: \"3683d0cf-7bf2-45f9-b6ee-5fa4e52c020d\") " pod="openshift-must-gather-wqrhv/perf-node-gather-daemonset-zjpl5" Jul 12 20:04:39.321067 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:04:39.320976 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3683d0cf-7bf2-45f9-b6ee-5fa4e52c020d-sys\") pod \"perf-node-gather-daemonset-zjpl5\" (UID: \"3683d0cf-7bf2-45f9-b6ee-5fa4e52c020d\") " pod="openshift-must-gather-wqrhv/perf-node-gather-daemonset-zjpl5" Jul 12 20:04:39.328081 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:04:39.328053 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-66gt4\" (UniqueName: \"kubernetes.io/projected/3683d0cf-7bf2-45f9-b6ee-5fa4e52c020d-kube-api-access-66gt4\") pod \"perf-node-gather-daemonset-zjpl5\" (UID: \"3683d0cf-7bf2-45f9-b6ee-5fa4e52c020d\") " pod="openshift-must-gather-wqrhv/perf-node-gather-daemonset-zjpl5" Jul 12 20:04:39.432512 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:04:39.432477 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-wqrhv/perf-node-gather-daemonset-zjpl5" Jul 12 20:04:39.555047 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:04:39.555019 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-wqrhv/perf-node-gather-daemonset-zjpl5"] Jul 12 20:04:39.557442 ip-10-0-139-170 kubenswrapper[2565]: W0712 20:04:39.557415 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod3683d0cf_7bf2_45f9_b6ee_5fa4e52c020d.slice/crio-918bca0588a68d4668ab3dc13a8d46f2b2c45eb32717466e9a4f0e302ef91d6c WatchSource:0}: Error finding container 918bca0588a68d4668ab3dc13a8d46f2b2c45eb32717466e9a4f0e302ef91d6c: Status 404 returned error can't find the container with id 918bca0588a68d4668ab3dc13a8d46f2b2c45eb32717466e9a4f0e302ef91d6c Jul 12 20:04:39.850576 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:04:39.850482 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-wqrhv/perf-node-gather-daemonset-zjpl5" event={"ID":"3683d0cf-7bf2-45f9-b6ee-5fa4e52c020d","Type":"ContainerStarted","Data":"a5c182fa3762bd2a57f8a55755feb1cd856d4c21eeb8a705c009e2d85e18f355"} Jul 12 20:04:39.850576 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:04:39.850533 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-wqrhv/perf-node-gather-daemonset-zjpl5" event={"ID":"3683d0cf-7bf2-45f9-b6ee-5fa4e52c020d","Type":"ContainerStarted","Data":"918bca0588a68d4668ab3dc13a8d46f2b2c45eb32717466e9a4f0e302ef91d6c"} Jul 12 20:04:39.850576 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:04:39.850562 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-wqrhv/perf-node-gather-daemonset-zjpl5" Jul 12 20:04:39.865413 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:04:39.865366 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-wqrhv/perf-node-gather-daemonset-zjpl5" podStartSLOduration=0.865351668 podStartE2EDuration="865.351668ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-12 20:04:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 20:04:39.864310517 +0000 UTC m=+2544.596883031" watchObservedRunningTime="2026-07-12 20:04:39.865351668 +0000 UTC m=+2544.597924157" Jul 12 20:04:40.015783 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:04:40.015754 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-7d4d5f7c4f-d5vdz_ac3c9fb2-cbee-4fc5-91da-448c7482fe6c/console/0.log" Jul 12 20:04:41.680987 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:04:41.680947 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-zn4n8_53b79340-2472-4ee4-ab93-4eecee72fbfb/dns/0.log" Jul 12 20:04:41.699212 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:04:41.699183 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-zn4n8_53b79340-2472-4ee4-ab93-4eecee72fbfb/kube-rbac-proxy/0.log" Jul 12 20:04:41.766372 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:04:41.766337 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-zs4lw_8e832009-b138-4442-b9db-10e73957481e/dns-node-resolver/0.log" Jul 12 20:04:42.300138 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:04:42.300102 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-c89jb_b8cca381-48d7-43ac-ae6a-15f96d99345d/node-ca/0.log" Jul 12 20:04:43.233103 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:04:43.233074 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-7f964fb678c9jcd_d445ad9c-2f45-4808-989a-d988ef1dbaff/istio-proxy/0.log" Jul 12 20:04:43.335089 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:04:43.335057 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-vgbw9_4f668c0d-a883-4283-bea4-fb5f56cd1481/discovery/0.log" Jul 12 20:04:43.353660 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:04:43.353634 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-654967888b-nbxf9_f94b7de3-7cb4-4346-87eb-1ac4dfc83cc0/kube-auth-proxy/0.log" Jul 12 20:04:44.294061 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:04:44.294035 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-vcdcf_ea94aee7-5bcb-41be-9513-c66834d390e3/serve-healthcheck-canary/0.log" Jul 12 20:04:44.833021 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:04:44.832986 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-qhx89_a7e56501-c8eb-4615-8bf8-1598f8f9fe39/kube-rbac-proxy/0.log" Jul 12 20:04:44.851668 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:04:44.851647 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-qhx89_a7e56501-c8eb-4615-8bf8-1598f8f9fe39/exporter/0.log" Jul 12 20:04:44.871355 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:04:44.871334 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-qhx89_a7e56501-c8eb-4615-8bf8-1598f8f9fe39/extractor/0.log" Jul 12 20:04:45.864184 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:04:45.864159 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-wqrhv/perf-node-gather-daemonset-zjpl5" Jul 12 20:04:47.309542 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:04:47.309514 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-4jntx_352106d4-8ce4-442d-a232-416311e4b0a1/server/0.log" Jul 12 20:04:47.443818 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:04:47.443785 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5449d7969f-kr7sd_c46c3317-ba7f-493b-a751-961cfa421b5c/manager/0.log" Jul 12 20:04:47.461609 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:04:47.461583 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5449d7969f-kr7sd_c46c3317-ba7f-493b-a751-961cfa421b5c/copy-manifests/0.log" Jul 12 20:04:49.377529 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:04:49.377452 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-79f8975998-xbfx6_87e72f83-5c44-408f-bfa5-24e34721c580/manager/0.log" Jul 12 20:04:50.078816 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:04:50.078783 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-7f964fb678c9jcd_d445ad9c-2f45-4808-989a-d988ef1dbaff/istio-proxy/0.log" Jul 12 20:04:50.179822 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:04:50.179740 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-vgbw9_4f668c0d-a883-4283-bea4-fb5f56cd1481/discovery/0.log" Jul 12 20:04:50.200206 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:04:50.200183 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-654967888b-nbxf9_f94b7de3-7cb4-4346-87eb-1ac4dfc83cc0/kube-auth-proxy/0.log" Jul 12 20:04:51.585925 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:04:51.585864 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-7f964fb678c9jcd_d445ad9c-2f45-4808-989a-d988ef1dbaff/istio-proxy/0.log" Jul 12 20:04:51.690240 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:04:51.690212 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-vgbw9_4f668c0d-a883-4283-bea4-fb5f56cd1481/discovery/0.log" Jul 12 20:04:51.701047 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:04:51.701024 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-654967888b-nbxf9_f94b7de3-7cb4-4346-87eb-1ac4dfc83cc0/kube-auth-proxy/0.log" Jul 12 20:04:55.659210 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:04:55.659175 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-9klzd_bdb521c9-c545-445b-aee4-872355e9eb78/kube-multus/0.log" Jul 12 20:04:55.969803 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:04:55.969719 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-prfcd_51dee771-9663-4366-90ca-9b1c286e35f8/kube-multus-additional-cni-plugins/0.log" Jul 12 20:04:55.988757 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:04:55.988731 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-prfcd_51dee771-9663-4366-90ca-9b1c286e35f8/egress-router-binary-copy/0.log" Jul 12 20:04:56.007264 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:04:56.007235 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-prfcd_51dee771-9663-4366-90ca-9b1c286e35f8/cni-plugins/0.log" Jul 12 20:04:56.025337 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:04:56.025312 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-prfcd_51dee771-9663-4366-90ca-9b1c286e35f8/bond-cni-plugin/0.log" Jul 12 20:04:56.043697 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:04:56.043674 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-prfcd_51dee771-9663-4366-90ca-9b1c286e35f8/routeoverride-cni/0.log" Jul 12 20:04:56.062649 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:04:56.062629 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-prfcd_51dee771-9663-4366-90ca-9b1c286e35f8/whereabouts-cni-bincopy/0.log" Jul 12 20:04:56.080422 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:04:56.080399 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-prfcd_51dee771-9663-4366-90ca-9b1c286e35f8/whereabouts-cni/0.log" Jul 12 20:04:56.102382 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:04:56.102360 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-2xmfq_e7722ae2-e293-478e-9ac8-ea376f8d583f/network-metrics-daemon/0.log" Jul 12 20:04:56.119063 ip-10-0-139-170 kubenswrapper[2565]: I0712 20:04:56.119042 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-2xmfq_e7722ae2-e293-478e-9ac8-ea376f8d583f/kube-rbac-proxy/0.log"