Jul 15 23:36:24.519242 ip-10-0-128-197 systemd[1]: Starting Kubernetes Kubelet... Jul 15 23:36:24.876206 ip-10-0-128-197 kubenswrapper[2563]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 15 23:36:24.876206 ip-10-0-128-197 kubenswrapper[2563]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 15 23:36:24.876206 ip-10-0-128-197 kubenswrapper[2563]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 15 23:36:24.876206 ip-10-0-128-197 kubenswrapper[2563]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 15 23:36:24.876206 ip-10-0-128-197 kubenswrapper[2563]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 15 23:36:24.876206 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:24.866727 2563 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 15 23:36:24.876206 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.873193 2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 15 23:36:24.876206 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.873206 2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 15 23:36:24.876206 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.873211 2563 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 15 23:36:24.876206 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.873214 2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 15 23:36:24.876206 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.873217 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 15 23:36:24.876206 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.873221 2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 15 23:36:24.876206 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.873224 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 15 23:36:24.876206 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.873227 2563 feature_gate.go:328] unrecognized feature gate: Example2 Jul 15 23:36:24.883411 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.873230 2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 15 23:36:24.883411 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.873233 2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 15 23:36:24.883411 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.873235 2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 15 23:36:24.883411 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.873238 2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 15 23:36:24.883411 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.873241 2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 15 23:36:24.883411 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.873243 2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 15 23:36:24.883411 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.873246 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 15 23:36:24.883411 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.873249 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 15 23:36:24.883411 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.873251 2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 15 23:36:24.883411 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.873254 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 15 23:36:24.883411 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.873257 2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 15 23:36:24.883411 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.873259 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 15 23:36:24.883411 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.873262 2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 15 23:36:24.883411 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.873273 2563 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 15 23:36:24.883411 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.873276 2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 15 23:36:24.883411 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.873279 2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 15 23:36:24.883411 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.873281 2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 15 23:36:24.883411 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.873285 2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 15 23:36:24.883411 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.873288 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 15 23:36:24.883411 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.873291 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 15 23:36:24.884037 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.873293 2563 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 15 23:36:24.884037 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.873296 2563 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 15 23:36:24.884037 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.873298 2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 15 23:36:24.884037 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.873301 2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 15 23:36:24.884037 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.873304 2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 15 23:36:24.884037 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.873307 2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 15 23:36:24.884037 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.873309 2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 15 23:36:24.884037 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.873314 2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 15 23:36:24.884037 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.873318 2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 15 23:36:24.884037 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.873321 2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 15 23:36:24.884037 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.873323 2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 15 23:36:24.884037 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.873328 2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 15 23:36:24.884037 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.873331 2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 15 23:36:24.884037 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.873334 2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 15 23:36:24.884037 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.873336 2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 15 23:36:24.884037 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.873339 2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 15 23:36:24.884037 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.873341 2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 15 23:36:24.884037 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.873344 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 15 23:36:24.884037 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.873347 2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 15 23:36:24.884037 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.873350 2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 15 23:36:24.884549 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.873352 2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 15 23:36:24.884549 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.873355 2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 15 23:36:24.884549 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.873358 2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 15 23:36:24.884549 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.873361 2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 15 23:36:24.884549 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.873363 2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 15 23:36:24.884549 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.873366 2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 15 23:36:24.884549 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.873369 2563 feature_gate.go:328] unrecognized feature gate: Example Jul 15 23:36:24.884549 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.873372 2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 15 23:36:24.884549 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.873374 2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 15 23:36:24.884549 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.873377 2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 15 23:36:24.884549 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.873380 2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 15 23:36:24.884549 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.873382 2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 15 23:36:24.884549 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.873385 2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 15 23:36:24.884549 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.873388 2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 15 23:36:24.884549 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.873390 2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 15 23:36:24.884549 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.873393 2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 15 23:36:24.884549 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.873395 2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 15 23:36:24.884549 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.873398 2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 15 23:36:24.884549 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.873400 2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 15 23:36:24.884549 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.873403 2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 15 23:36:24.885154 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.873406 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 15 23:36:24.885154 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.873408 2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 15 23:36:24.885154 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.873411 2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 15 23:36:24.885154 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.873414 2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 15 23:36:24.885154 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.873418 2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 15 23:36:24.885154 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.873420 2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 15 23:36:24.885154 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.873423 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 15 23:36:24.885154 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.873425 2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 15 23:36:24.885154 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.873428 2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 15 23:36:24.885154 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.873430 2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 15 23:36:24.885154 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.873433 2563 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 15 23:36:24.885154 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.873437 2563 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 15 23:36:24.885154 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.873440 2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 15 23:36:24.885154 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.873442 2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 15 23:36:24.885154 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.873445 2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 15 23:36:24.885154 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.873448 2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 15 23:36:24.885154 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.873450 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 15 23:36:24.885154 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.873453 2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 15 23:36:24.885154 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.873456 2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 15 23:36:24.885154 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.873459 2563 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 15 23:36:24.885684 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.873462 2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 15 23:36:24.885684 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.873464 2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 15 23:36:24.885684 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.873467 2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 15 23:36:24.885684 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.873470 2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 15 23:36:24.885684 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.873472 2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 15 23:36:24.885684 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.873475 2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 15 23:36:24.885684 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.873477 2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 15 23:36:24.885684 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.873481 2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 15 23:36:24.885684 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.873486 2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 15 23:36:24.885684 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.873489 2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 15 23:36:24.885684 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.873493 2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 15 23:36:24.885684 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.876259 2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 15 23:36:24.885684 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.876270 2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 15 23:36:24.885684 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.876275 2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 15 23:36:24.885684 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.876278 2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 15 23:36:24.885684 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.876281 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 15 23:36:24.885684 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.876284 2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 15 23:36:24.885684 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.876287 2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 15 23:36:24.885684 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.876290 2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 15 23:36:24.886216 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.876293 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 15 23:36:24.886216 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.876295 2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 15 23:36:24.886216 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.876298 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 15 23:36:24.886216 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.876301 2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 15 23:36:24.886216 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.876303 2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 15 23:36:24.886216 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.876306 2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 15 23:36:24.886216 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.876308 2563 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 15 23:36:24.886216 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.876311 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 15 23:36:24.886216 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.876314 2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 15 23:36:24.886216 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.876316 2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 15 23:36:24.886216 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.876319 2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 15 23:36:24.886216 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.876322 2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 15 23:36:24.886216 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.876324 2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 15 23:36:24.886216 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.876327 2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 15 23:36:24.886216 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.876330 2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 15 23:36:24.886216 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.876332 2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 15 23:36:24.886216 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.876335 2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 15 23:36:24.886216 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.876338 2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 15 23:36:24.886216 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.876340 2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 15 23:36:24.886216 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.876343 2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 15 23:36:24.886743 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.876345 2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 15 23:36:24.886743 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.876348 2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 15 23:36:24.886743 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.876351 2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 15 23:36:24.886743 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.876354 2563 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 15 23:36:24.886743 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.876356 2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 15 23:36:24.886743 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.876359 2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 15 23:36:24.886743 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.876362 2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 15 23:36:24.886743 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.876364 2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 15 23:36:24.886743 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.876367 2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 15 23:36:24.886743 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.876370 2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 15 23:36:24.886743 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.876372 2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 15 23:36:24.886743 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.876375 2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 15 23:36:24.886743 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.876378 2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 15 23:36:24.886743 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.876380 2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 15 23:36:24.886743 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.876383 2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 15 23:36:24.886743 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.876386 2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 15 23:36:24.886743 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.876388 2563 feature_gate.go:328] unrecognized feature gate: Example Jul 15 23:36:24.886743 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.876391 2563 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 15 23:36:24.886743 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.876393 2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 15 23:36:24.887289 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.876396 2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 15 23:36:24.887289 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.876398 2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 15 23:36:24.887289 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.876402 2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 15 23:36:24.887289 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.876404 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 15 23:36:24.887289 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.876407 2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 15 23:36:24.887289 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.876411 2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 15 23:36:24.887289 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.876416 2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 15 23:36:24.887289 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.876420 2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 15 23:36:24.887289 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.876423 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 15 23:36:24.887289 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.876426 2563 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 15 23:36:24.887289 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.876428 2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 15 23:36:24.887289 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.876431 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 15 23:36:24.887289 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.876434 2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 15 23:36:24.887289 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.876436 2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 15 23:36:24.887289 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.876439 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 15 23:36:24.887289 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.876444 2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 15 23:36:24.887289 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.876451 2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 15 23:36:24.887289 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.876454 2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 15 23:36:24.887289 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.876457 2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 15 23:36:24.887898 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.876461 2563 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 15 23:36:24.887898 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.876464 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 15 23:36:24.887898 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.876467 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 15 23:36:24.887898 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.876470 2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 15 23:36:24.887898 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.876473 2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 15 23:36:24.887898 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.876476 2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 15 23:36:24.887898 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.876479 2563 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 15 23:36:24.887898 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.876481 2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 15 23:36:24.887898 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.876484 2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 15 23:36:24.887898 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.876487 2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 15 23:36:24.887898 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.876489 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 15 23:36:24.887898 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.876492 2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 15 23:36:24.887898 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.876495 2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 15 23:36:24.887898 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.876497 2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 15 23:36:24.887898 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.876500 2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 15 23:36:24.887898 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.876502 2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 15 23:36:24.887898 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.876505 2563 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 15 23:36:24.887898 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.876508 2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 15 23:36:24.887898 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.876511 2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 15 23:36:24.887898 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.876513 2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 15 23:36:24.888527 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.876516 2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 15 23:36:24.888527 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.876519 2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 15 23:36:24.888527 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.876521 2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 15 23:36:24.888527 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.876524 2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 15 23:36:24.888527 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.876526 2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 15 23:36:24.888527 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.876529 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 15 23:36:24.888527 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.876531 2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 15 23:36:24.888527 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.876534 2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 15 23:36:24.888527 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.876537 2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 15 23:36:24.888527 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.876541 2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 15 23:36:24.888527 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.876543 2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 15 23:36:24.888527 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.876546 2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 15 23:36:24.888527 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.876548 2563 feature_gate.go:328] unrecognized feature gate: Example2 Jul 15 23:36:24.888527 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.876931 2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 15 23:36:24.888527 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.876937 2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 15 23:36:24.888527 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.876941 2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 15 23:36:24.888527 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.876944 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 15 23:36:24.888527 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.876947 2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 15 23:36:24.888527 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.876950 2563 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 15 23:36:24.888527 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.876953 2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 15 23:36:24.889078 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.876956 2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 15 23:36:24.889078 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.876959 2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 15 23:36:24.889078 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.876961 2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 15 23:36:24.889078 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.876981 2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 15 23:36:24.889078 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.876985 2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 15 23:36:24.889078 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.876988 2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 15 23:36:24.889078 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.876991 2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 15 23:36:24.889078 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.876993 2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 15 23:36:24.889078 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.876996 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 15 23:36:24.889078 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.877000 2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 15 23:36:24.889078 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.877003 2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 15 23:36:24.889078 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.877005 2563 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 15 23:36:24.889078 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.877009 2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 15 23:36:24.889078 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.877013 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 15 23:36:24.889078 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.877016 2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 15 23:36:24.889078 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.877019 2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 15 23:36:24.889078 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.877022 2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 15 23:36:24.889078 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.877025 2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 15 23:36:24.889078 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.877027 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 15 23:36:24.889632 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.877030 2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 15 23:36:24.889632 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.877034 2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 15 23:36:24.889632 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.877038 2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 15 23:36:24.889632 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.877041 2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 15 23:36:24.889632 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.877043 2563 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 15 23:36:24.889632 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.877046 2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 15 23:36:24.889632 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.877048 2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 15 23:36:24.889632 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.877051 2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 15 23:36:24.889632 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.877054 2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 15 23:36:24.889632 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.877057 2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 15 23:36:24.889632 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.877065 2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 15 23:36:24.889632 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.877069 2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 15 23:36:24.889632 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.877072 2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 15 23:36:24.889632 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.877075 2563 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 15 23:36:24.889632 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.877078 2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 15 23:36:24.889632 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.877081 2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 15 23:36:24.889632 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.877083 2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 15 23:36:24.889632 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.877086 2563 feature_gate.go:328] unrecognized feature gate: Example2 Jul 15 23:36:24.889632 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.877089 2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 15 23:36:24.890163 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.877092 2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 15 23:36:24.890163 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.877094 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 15 23:36:24.890163 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.877097 2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 15 23:36:24.890163 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.877100 2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 15 23:36:24.890163 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.877103 2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 15 23:36:24.890163 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.877105 2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 15 23:36:24.890163 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.877108 2563 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 15 23:36:24.890163 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.877111 2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 15 23:36:24.890163 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.877114 2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 15 23:36:24.890163 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.877116 2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 15 23:36:24.890163 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.877118 2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 15 23:36:24.890163 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.877121 2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 15 23:36:24.890163 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.877124 2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 15 23:36:24.890163 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.877127 2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 15 23:36:24.890163 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.877131 2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 15 23:36:24.890163 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.877135 2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 15 23:36:24.890163 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.877138 2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 15 23:36:24.890163 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.877140 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 15 23:36:24.890163 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.877143 2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 15 23:36:24.890680 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.877146 2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 15 23:36:24.890680 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.877148 2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 15 23:36:24.890680 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.877151 2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 15 23:36:24.890680 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.877154 2563 feature_gate.go:328] unrecognized feature gate: Example Jul 15 23:36:24.890680 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.877156 2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 15 23:36:24.890680 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.877159 2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 15 23:36:24.890680 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.877162 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 15 23:36:24.890680 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.877164 2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 15 23:36:24.890680 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.877166 2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 15 23:36:24.890680 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.877170 2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 15 23:36:24.890680 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.877172 2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 15 23:36:24.890680 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.877175 2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 15 23:36:24.890680 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.877177 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 15 23:36:24.890680 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.877180 2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 15 23:36:24.890680 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.877183 2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 15 23:36:24.890680 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.877190 2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 15 23:36:24.890680 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.877192 2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 15 23:36:24.890680 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.877195 2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 15 23:36:24.890680 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.877199 2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 15 23:36:24.890680 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.877201 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 15 23:36:24.891227 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.877204 2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 15 23:36:24.891227 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.877207 2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 15 23:36:24.891227 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.877210 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 15 23:36:24.891227 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.877212 2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 15 23:36:24.891227 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.877215 2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 15 23:36:24.891227 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.877217 2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 15 23:36:24.891227 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.877219 2563 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 15 23:36:24.891227 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.877223 2563 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 15 23:36:24.891227 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.877226 2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 15 23:36:24.891227 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.877229 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 15 23:36:24.891227 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.877231 2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 15 23:36:24.891227 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.877235 2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 15 23:36:24.891227 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.877237 2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 15 23:36:24.891227 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.877240 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 15 23:36:24.891227 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.877242 2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 15 23:36:24.891227 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:24.877319 2563 flags.go:64] FLAG: --address="0.0.0.0" Jul 15 23:36:24.891227 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:24.877326 2563 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 15 23:36:24.891227 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:24.877333 2563 flags.go:64] FLAG: --anonymous-auth="true" Jul 15 23:36:24.891227 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:24.877339 2563 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 15 23:36:24.891227 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:24.877345 2563 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 15 23:36:24.891227 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:24.877348 2563 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 15 23:36:24.891227 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:24.877353 2563 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 15 23:36:24.891842 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:24.877356 2563 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 15 23:36:24.891842 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:24.877359 2563 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 15 23:36:24.891842 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:24.877362 2563 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 15 23:36:24.891842 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:24.877366 2563 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 15 23:36:24.891842 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:24.877369 2563 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 15 23:36:24.891842 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:24.877372 2563 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 15 23:36:24.891842 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:24.877375 2563 flags.go:64] FLAG: --cgroup-root="" Jul 15 23:36:24.891842 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:24.877378 2563 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 15 23:36:24.891842 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:24.877381 2563 flags.go:64] FLAG: --client-ca-file="" Jul 15 23:36:24.891842 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:24.877385 2563 flags.go:64] FLAG: --cloud-provider="external" Jul 15 23:36:24.891842 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:24.877388 2563 flags.go:64] FLAG: --cluster-dns="[]" Jul 15 23:36:24.891842 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:24.877393 2563 flags.go:64] FLAG: --cluster-domain="" Jul 15 23:36:24.891842 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:24.877395 2563 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 15 23:36:24.891842 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:24.877399 2563 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 15 23:36:24.891842 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:24.877401 2563 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 15 23:36:24.891842 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:24.877405 2563 flags.go:64] FLAG: --container-log-max-files="5" Jul 15 23:36:24.891842 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:24.877415 2563 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 15 23:36:24.891842 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:24.877418 2563 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 15 23:36:24.891842 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:24.877423 2563 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 15 23:36:24.891842 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:24.877426 2563 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 15 23:36:24.891842 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:24.877429 2563 flags.go:64] FLAG: --contention-profiling="false" Jul 15 23:36:24.891842 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:24.877433 2563 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 15 23:36:24.891842 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:24.877436 2563 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 15 23:36:24.891842 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:24.877439 2563 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 15 23:36:24.892466 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:24.877442 2563 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 15 23:36:24.892466 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:24.877446 2563 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 15 23:36:24.892466 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:24.877449 2563 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 15 23:36:24.892466 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:24.877452 2563 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 15 23:36:24.892466 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:24.877454 2563 flags.go:64] FLAG: --enable-load-reader="false" Jul 15 23:36:24.892466 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:24.877457 2563 flags.go:64] FLAG: --enable-server="true" Jul 15 23:36:24.892466 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:24.877460 2563 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 15 23:36:24.892466 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:24.877466 2563 flags.go:64] FLAG: --event-burst="100" Jul 15 23:36:24.892466 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:24.877469 2563 flags.go:64] FLAG: --event-qps="50" Jul 15 23:36:24.892466 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:24.877472 2563 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 15 23:36:24.892466 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:24.877475 2563 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 15 23:36:24.892466 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:24.877478 2563 flags.go:64] FLAG: --eviction-hard="" Jul 15 23:36:24.892466 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:24.877481 2563 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 15 23:36:24.892466 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:24.877484 2563 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 15 23:36:24.892466 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:24.877487 2563 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 15 23:36:24.892466 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:24.877490 2563 flags.go:64] FLAG: --eviction-soft="" Jul 15 23:36:24.892466 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:24.877493 2563 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 15 23:36:24.892466 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:24.877496 2563 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 15 23:36:24.892466 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:24.877499 2563 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 15 23:36:24.892466 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:24.877502 2563 flags.go:64] FLAG: --experimental-mounter-path="" Jul 15 23:36:24.892466 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:24.877505 2563 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 15 23:36:24.892466 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:24.877507 2563 flags.go:64] FLAG: --fail-swap-on="true" Jul 15 23:36:24.892466 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:24.877510 2563 flags.go:64] FLAG: --feature-gates="" Jul 15 23:36:24.892466 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:24.877514 2563 flags.go:64] FLAG: --file-check-frequency="20s" Jul 15 23:36:24.892466 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:24.877517 2563 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 15 23:36:24.900002 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:24.877520 2563 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 15 23:36:24.900002 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:24.877524 2563 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 15 23:36:24.900002 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:24.877527 2563 flags.go:64] FLAG: --healthz-port="10248" Jul 15 23:36:24.900002 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:24.877530 2563 flags.go:64] FLAG: --help="false" Jul 15 23:36:24.900002 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:24.877533 2563 flags.go:64] FLAG: --hostname-override="ip-10-0-128-197.ec2.internal" Jul 15 23:36:24.900002 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:24.877536 2563 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 15 23:36:24.900002 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:24.877539 2563 flags.go:64] FLAG: --http-check-frequency="20s" Jul 15 23:36:24.900002 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:24.877541 2563 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 15 23:36:24.900002 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:24.877545 2563 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 15 23:36:24.900002 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:24.877548 2563 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 15 23:36:24.900002 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:24.877551 2563 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 15 23:36:24.900002 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:24.877554 2563 flags.go:64] FLAG: --image-service-endpoint="" Jul 15 23:36:24.900002 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:24.877556 2563 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 15 23:36:24.900002 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:24.877559 2563 flags.go:64] FLAG: --kube-api-burst="100" Jul 15 23:36:24.900002 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:24.877563 2563 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 15 23:36:24.900002 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:24.877566 2563 flags.go:64] FLAG: --kube-api-qps="50" Jul 15 23:36:24.900002 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:24.877569 2563 flags.go:64] FLAG: --kube-reserved="" Jul 15 23:36:24.900002 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:24.877571 2563 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 15 23:36:24.900002 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:24.877574 2563 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 15 23:36:24.900002 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:24.877577 2563 flags.go:64] FLAG: --kubelet-cgroups="" Jul 15 23:36:24.900002 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:24.877580 2563 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 15 23:36:24.900002 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:24.877583 2563 flags.go:64] FLAG: --lock-file="" Jul 15 23:36:24.900002 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:24.877586 2563 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 15 23:36:24.900002 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:24.877589 2563 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 15 23:36:24.900658 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:24.877592 2563 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 15 23:36:24.900658 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:24.877597 2563 flags.go:64] FLAG: --log-json-split-stream="false" Jul 15 23:36:24.900658 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:24.877600 2563 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 15 23:36:24.900658 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:24.877603 2563 flags.go:64] FLAG: --log-text-split-stream="false" Jul 15 23:36:24.900658 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:24.877606 2563 flags.go:64] FLAG: --logging-format="text" Jul 15 23:36:24.900658 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:24.877608 2563 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 15 23:36:24.900658 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:24.877612 2563 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 15 23:36:24.900658 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:24.877614 2563 flags.go:64] FLAG: --manifest-url="" Jul 15 23:36:24.900658 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:24.877617 2563 flags.go:64] FLAG: --manifest-url-header="" Jul 15 23:36:24.900658 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:24.877622 2563 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 15 23:36:24.900658 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:24.877625 2563 flags.go:64] FLAG: --max-open-files="1000000" Jul 15 23:36:24.900658 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:24.877629 2563 flags.go:64] FLAG: --max-pods="110" Jul 15 23:36:24.900658 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:24.877632 2563 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 15 23:36:24.900658 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:24.877635 2563 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 15 23:36:24.900658 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:24.877638 2563 flags.go:64] FLAG: --memory-manager-policy="None" Jul 15 23:36:24.900658 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:24.877641 2563 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 15 23:36:24.900658 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:24.877644 2563 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 15 23:36:24.900658 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:24.877658 2563 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 15 23:36:24.900658 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:24.877662 2563 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 15 23:36:24.900658 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:24.877670 2563 flags.go:64] FLAG: --node-status-max-images="50" Jul 15 23:36:24.900658 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:24.877674 2563 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 15 23:36:24.900658 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:24.877677 2563 flags.go:64] FLAG: --oom-score-adj="-999" Jul 15 23:36:24.900658 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:24.877684 2563 flags.go:64] FLAG: --pod-cidr="" Jul 15 23:36:24.901516 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:24.877687 2563 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b04c0b1f73af4d021dd8f0b51bfeb302c24482c093de5f2c43654ab56c506926" Jul 15 23:36:24.901516 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:24.877692 2563 flags.go:64] FLAG: --pod-manifest-path="" Jul 15 23:36:24.901516 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:24.877695 2563 flags.go:64] FLAG: --pod-max-pids="-1" Jul 15 23:36:24.901516 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:24.877698 2563 flags.go:64] FLAG: --pods-per-core="0" Jul 15 23:36:24.901516 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:24.877701 2563 flags.go:64] FLAG: --port="10250" Jul 15 23:36:24.901516 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:24.877704 2563 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 15 23:36:24.901516 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:24.877707 2563 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-07f5b524d6e22ac73" Jul 15 23:36:24.901516 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:24.877710 2563 flags.go:64] FLAG: --qos-reserved="" Jul 15 23:36:24.901516 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:24.877713 2563 flags.go:64] FLAG: --read-only-port="10255" Jul 15 23:36:24.901516 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:24.877716 2563 flags.go:64] FLAG: --register-node="true" Jul 15 23:36:24.901516 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:24.877718 2563 flags.go:64] FLAG: --register-with-taints="" Jul 15 23:36:24.901516 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:24.877722 2563 flags.go:64] FLAG: --registry-burst="10" Jul 15 23:36:24.901516 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:24.877726 2563 flags.go:64] FLAG: --registry-qps="5" Jul 15 23:36:24.901516 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:24.877729 2563 flags.go:64] FLAG: --reserved-cpus="" Jul 15 23:36:24.901516 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:24.877732 2563 flags.go:64] FLAG: --reserved-memory="" Jul 15 23:36:24.901516 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:24.877735 2563 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 15 23:36:24.901516 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:24.877738 2563 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 15 23:36:24.901516 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:24.877741 2563 flags.go:64] FLAG: --rotate-certificates="false" Jul 15 23:36:24.901516 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:24.877744 2563 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 15 23:36:24.901516 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:24.877748 2563 flags.go:64] FLAG: --runonce="false" Jul 15 23:36:24.901516 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:24.877751 2563 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 15 23:36:24.901516 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:24.877754 2563 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 15 23:36:24.901516 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:24.877757 2563 flags.go:64] FLAG: --seccomp-default="false" Jul 15 23:36:24.901516 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:24.877760 2563 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 15 23:36:24.901516 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:24.877763 2563 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 15 23:36:25.007281 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:24.877765 2563 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 15 23:36:25.007281 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:24.877768 2563 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 15 23:36:25.007281 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:24.877771 2563 flags.go:64] FLAG: --storage-driver-password="root" Jul 15 23:36:25.007281 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:24.877774 2563 flags.go:64] FLAG: --storage-driver-secure="false" Jul 15 23:36:25.007281 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:24.877776 2563 flags.go:64] FLAG: --storage-driver-table="stats" Jul 15 23:36:25.007281 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:24.877779 2563 flags.go:64] FLAG: --storage-driver-user="root" Jul 15 23:36:25.007281 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:24.877783 2563 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 15 23:36:25.007281 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:24.877786 2563 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 15 23:36:25.007281 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:24.877789 2563 flags.go:64] FLAG: --system-cgroups="" Jul 15 23:36:25.007281 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:24.877791 2563 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 15 23:36:25.007281 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:24.877797 2563 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 15 23:36:25.007281 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:24.877800 2563 flags.go:64] FLAG: --tls-cert-file="" Jul 15 23:36:25.007281 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:24.877802 2563 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 15 23:36:25.007281 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:24.877807 2563 flags.go:64] FLAG: --tls-min-version="" Jul 15 23:36:25.007281 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:24.877809 2563 flags.go:64] FLAG: --tls-private-key-file="" Jul 15 23:36:25.007281 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:24.877812 2563 flags.go:64] FLAG: --topology-manager-policy="none" Jul 15 23:36:25.007281 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:24.877815 2563 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 15 23:36:25.007281 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:24.877818 2563 flags.go:64] FLAG: --topology-manager-scope="container" Jul 15 23:36:25.007281 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:24.877821 2563 flags.go:64] FLAG: --v="2" Jul 15 23:36:25.007281 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:24.877825 2563 flags.go:64] FLAG: --version="false" Jul 15 23:36:25.007281 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:24.877832 2563 flags.go:64] FLAG: --vmodule="" Jul 15 23:36:25.007281 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:24.877836 2563 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 15 23:36:25.007281 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:24.877839 2563 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 15 23:36:25.007281 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.877982 2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 15 23:36:25.007281 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.877987 2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 15 23:36:24.944814 ip-10-0-128-197 systemd[1]: Started Kubernetes Kubelet. Jul 15 23:36:25.008159 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.877990 2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 15 23:36:25.008159 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.877994 2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 15 23:36:25.008159 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.877998 2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 15 23:36:25.008159 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.878000 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 15 23:36:25.008159 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.878003 2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 15 23:36:25.008159 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.878006 2563 feature_gate.go:328] unrecognized feature gate: Example Jul 15 23:36:25.008159 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.878008 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 15 23:36:25.008159 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.878011 2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 15 23:36:25.008159 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.878014 2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 15 23:36:25.008159 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.878016 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 15 23:36:25.008159 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.878019 2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 15 23:36:25.008159 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.878022 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 15 23:36:25.008159 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.878024 2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 15 23:36:25.008159 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.878028 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 15 23:36:25.008159 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.878031 2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 15 23:36:25.008159 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.878034 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 15 23:36:25.008159 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.878036 2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 15 23:36:25.008159 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.878041 2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 15 23:36:25.008159 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.878044 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 15 23:36:25.008159 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.878047 2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 15 23:36:25.009765 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.878050 2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 15 23:36:25.009765 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.878053 2563 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 15 23:36:25.009765 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.878055 2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 15 23:36:25.009765 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.878058 2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 15 23:36:25.009765 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.878060 2563 feature_gate.go:328] unrecognized feature gate: Example2 Jul 15 23:36:25.009765 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.878063 2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 15 23:36:25.009765 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.878065 2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 15 23:36:25.009765 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.878068 2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 15 23:36:25.009765 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.878071 2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 15 23:36:25.009765 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.878074 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 15 23:36:25.009765 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.878076 2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 15 23:36:25.009765 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.878079 2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 15 23:36:25.009765 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.878082 2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 15 23:36:25.009765 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.878086 2563 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 15 23:36:25.009765 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.878089 2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 15 23:36:25.009765 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.878091 2563 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 15 23:36:25.009765 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.878094 2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 15 23:36:25.009765 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.878096 2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 15 23:36:25.009765 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.878099 2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 15 23:36:25.009765 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.878102 2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 15 23:36:25.010463 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.878106 2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 15 23:36:25.010463 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.878109 2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 15 23:36:25.010463 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.878112 2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 15 23:36:25.010463 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.878115 2563 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 15 23:36:25.010463 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.878117 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 15 23:36:25.010463 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.878121 2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 15 23:36:25.010463 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.878124 2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 15 23:36:25.010463 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.878127 2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 15 23:36:25.010463 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.878129 2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 15 23:36:25.010463 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.878132 2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 15 23:36:25.010463 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.878135 2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 15 23:36:25.010463 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.878137 2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 15 23:36:25.010463 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.878140 2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 15 23:36:25.010463 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.878142 2563 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 15 23:36:25.010463 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.878145 2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 15 23:36:25.010463 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.878147 2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 15 23:36:25.010463 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.878149 2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 15 23:36:25.010463 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.878152 2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 15 23:36:25.010463 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.878154 2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 15 23:36:25.011073 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.878157 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 15 23:36:25.011073 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.878159 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 15 23:36:25.011073 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.878163 2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 15 23:36:25.011073 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.878165 2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 15 23:36:25.011073 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.878168 2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 15 23:36:25.011073 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.878170 2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 15 23:36:25.011073 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.878174 2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 15 23:36:25.011073 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.878177 2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 15 23:36:25.011073 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.878179 2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 15 23:36:25.011073 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.878182 2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 15 23:36:25.011073 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.878185 2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 15 23:36:25.011073 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.878187 2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 15 23:36:25.011073 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.878189 2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 15 23:36:25.011073 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.878192 2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 15 23:36:25.011073 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.878194 2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 15 23:36:25.011073 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.878197 2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 15 23:36:25.011073 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.878200 2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 15 23:36:25.011073 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.878204 2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 15 23:36:25.011073 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.878208 2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 15 23:36:25.011862 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.878211 2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 15 23:36:25.011862 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.878214 2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 15 23:36:25.011862 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.878217 2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 15 23:36:25.011862 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.878219 2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 15 23:36:25.011862 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.878222 2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 15 23:36:25.011862 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.878224 2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 15 23:36:25.011862 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.878227 2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 15 23:36:25.011862 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.878230 2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 15 23:36:25.011862 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.878232 2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 15 23:36:25.011862 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.878235 2563 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 15 23:36:25.011862 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.878237 2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 15 23:36:25.011862 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.878240 2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 15 23:36:25.011862 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.878242 2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 15 23:36:25.011862 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.878245 2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 15 23:36:25.011862 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.878247 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 15 23:36:25.011862 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.878250 2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 15 23:36:25.011862 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.878253 2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 15 23:36:25.011862 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.878256 2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 15 23:36:25.011862 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.878259 2563 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 15 23:36:25.012462 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:24.878265 2563 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 15 23:36:25.012462 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:24.884870 2563 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 15 23:36:25.012462 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:24.884886 2563 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 15 23:36:25.012462 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:24.884905 2563 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 15 23:36:25.012462 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:24.884910 2563 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 15 23:36:25.012462 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.884988 2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 15 23:36:25.012462 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.884994 2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 15 23:36:25.012462 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.884998 2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 15 23:36:25.012462 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.885001 2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 15 23:36:25.012462 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.885004 2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 15 23:36:25.012462 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.885008 2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 15 23:36:25.012462 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.885011 2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 15 23:36:25.012462 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.885013 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 15 23:36:25.012462 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.885016 2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 15 23:36:25.013023 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.885020 2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 15 23:36:25.013023 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.885024 2563 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 15 23:36:25.013023 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.885027 2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 15 23:36:25.013023 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.885030 2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 15 23:36:25.013023 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.885032 2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 15 23:36:25.013023 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.885035 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 15 23:36:25.013023 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.885038 2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 15 23:36:25.013023 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.885040 2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 15 23:36:25.013023 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.885043 2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 15 23:36:25.013023 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.885045 2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 15 23:36:25.013023 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.885048 2563 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 15 23:36:25.013023 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.885051 2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 15 23:36:25.013023 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.885053 2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 15 23:36:25.013023 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.885056 2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 15 23:36:25.013023 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.885059 2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 15 23:36:25.013023 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.885063 2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 15 23:36:25.013023 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.885066 2563 feature_gate.go:328] unrecognized feature gate: Example Jul 15 23:36:25.013023 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.885069 2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 15 23:36:25.013023 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.885072 2563 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 15 23:36:25.013023 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.885075 2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 15 23:36:25.165362 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.885078 2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 15 23:36:25.165362 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.885080 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 15 23:36:25.165362 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.885083 2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 15 23:36:25.165362 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.885085 2563 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 15 23:36:25.165362 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.885088 2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 15 23:36:25.165362 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.885091 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 15 23:36:25.165362 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.885094 2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 15 23:36:25.165362 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.885096 2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 15 23:36:25.165362 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.885099 2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 15 23:36:25.165362 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.885101 2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 15 23:36:25.165362 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.885105 2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 15 23:36:25.165362 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.885109 2563 feature_gate.go:328] unrecognized feature gate: Example2 Jul 15 23:36:25.165362 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.885112 2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 15 23:36:25.165362 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.885116 2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 15 23:36:25.165362 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.885118 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 15 23:36:25.165362 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.885121 2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 15 23:36:25.165362 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.885124 2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 15 23:36:25.165362 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.885127 2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 15 23:36:25.165362 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.885130 2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 15 23:36:25.166214 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.885132 2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 15 23:36:25.166214 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.885135 2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 15 23:36:25.166214 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.885137 2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 15 23:36:25.166214 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.885141 2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 15 23:36:25.166214 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.885143 2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 15 23:36:25.166214 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.885146 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 15 23:36:25.166214 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.885149 2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 15 23:36:25.166214 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.885151 2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 15 23:36:25.166214 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.885154 2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 15 23:36:25.166214 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.885157 2563 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 15 23:36:25.166214 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.885160 2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 15 23:36:25.166214 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.885162 2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 15 23:36:25.166214 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.885165 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 15 23:36:25.166214 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.885168 2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 15 23:36:25.166214 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.885171 2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 15 23:36:25.166214 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.885173 2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 15 23:36:25.166214 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.885176 2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 15 23:36:25.166214 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.885179 2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 15 23:36:25.166214 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.885181 2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 15 23:36:25.167365 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.885184 2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 15 23:36:25.167365 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.885187 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 15 23:36:25.167365 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.885189 2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 15 23:36:25.167365 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.885192 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 15 23:36:25.167365 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.885194 2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 15 23:36:25.167365 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.885197 2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 15 23:36:25.167365 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.885200 2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 15 23:36:25.167365 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.885203 2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 15 23:36:25.167365 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.885205 2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 15 23:36:25.167365 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.885208 2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 15 23:36:25.167365 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.885210 2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 15 23:36:25.167365 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.885213 2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 15 23:36:25.167365 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.885215 2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 15 23:36:25.167365 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.885218 2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 15 23:36:25.167365 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.885220 2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 15 23:36:25.167365 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.885223 2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 15 23:36:25.167365 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.885226 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 15 23:36:25.167365 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.885229 2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 15 23:36:25.167365 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.885231 2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 15 23:36:25.167365 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.885234 2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 15 23:36:25.168466 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.885236 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 15 23:36:25.168466 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.885239 2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 15 23:36:25.168466 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.885241 2563 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 15 23:36:25.168466 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.885244 2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 15 23:36:25.168466 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.885247 2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 15 23:36:25.168466 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.885251 2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 15 23:36:25.168466 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.885254 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 15 23:36:25.168466 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.885257 2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 15 23:36:25.168466 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.885260 2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 15 23:36:25.168466 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.885269 2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 15 23:36:25.168466 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.885273 2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 15 23:36:25.168466 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.885276 2563 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 15 23:36:25.168466 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:24.885281 2563 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 15 23:36:25.168466 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.885391 2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 15 23:36:25.168466 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.885396 2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 15 23:36:25.169239 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.885398 2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 15 23:36:25.169239 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.885401 2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 15 23:36:25.169239 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.885404 2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 15 23:36:25.169239 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.885407 2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 15 23:36:25.169239 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.885411 2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 15 23:36:25.169239 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.885414 2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 15 23:36:25.169239 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.885417 2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 15 23:36:25.169239 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.885420 2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 15 23:36:25.169239 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.885422 2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 15 23:36:25.169239 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.885425 2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 15 23:36:25.169239 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.885430 2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 15 23:36:25.169239 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.885433 2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 15 23:36:25.169239 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.885436 2563 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 15 23:36:25.169239 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.885438 2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 15 23:36:25.169239 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.885441 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 15 23:36:25.169239 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.885443 2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 15 23:36:25.169239 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.885446 2563 feature_gate.go:328] unrecognized feature gate: Example Jul 15 23:36:25.169239 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.885449 2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 15 23:36:25.169239 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.885451 2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 15 23:36:25.170274 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.885454 2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 15 23:36:25.170274 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.885457 2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 15 23:36:25.170274 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.885460 2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 15 23:36:25.170274 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.885462 2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 15 23:36:25.170274 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.885466 2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 15 23:36:25.170274 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.885469 2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 15 23:36:25.170274 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.885472 2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 15 23:36:25.170274 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.885475 2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 15 23:36:25.170274 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.885477 2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 15 23:36:25.170274 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.885480 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 15 23:36:25.170274 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.885483 2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 15 23:36:25.170274 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.885485 2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 15 23:36:25.170274 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.885488 2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 15 23:36:25.170274 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.885491 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 15 23:36:25.170274 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.885495 2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 15 23:36:25.170274 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.885497 2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 15 23:36:25.170274 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.885500 2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 15 23:36:25.170274 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.885502 2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 15 23:36:25.170274 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.885505 2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 15 23:36:25.171160 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.885507 2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 15 23:36:25.171160 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.885510 2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 15 23:36:25.171160 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.885513 2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 15 23:36:25.171160 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.885515 2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 15 23:36:25.171160 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.885519 2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 15 23:36:25.171160 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.885521 2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 15 23:36:25.171160 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.885524 2563 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 15 23:36:25.171160 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.885526 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 15 23:36:25.171160 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.885529 2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 15 23:36:25.171160 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.885531 2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 15 23:36:25.171160 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.885534 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 15 23:36:25.171160 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.885537 2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 15 23:36:25.171160 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.885539 2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 15 23:36:25.171160 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.885542 2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 15 23:36:25.171160 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.885544 2563 feature_gate.go:328] unrecognized feature gate: Example2 Jul 15 23:36:25.171160 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.885547 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 15 23:36:25.171160 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.885550 2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 15 23:36:25.171160 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.885552 2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 15 23:36:25.171160 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.885555 2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 15 23:36:25.171160 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.885558 2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 15 23:36:25.172252 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.885560 2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 15 23:36:25.172252 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.885563 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 15 23:36:25.172252 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.885565 2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 15 23:36:25.172252 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.885568 2563 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 15 23:36:25.172252 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.885570 2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 15 23:36:25.172252 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.885573 2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 15 23:36:25.172252 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.885575 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 15 23:36:25.172252 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.885579 2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 15 23:36:25.172252 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.885582 2563 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 15 23:36:25.172252 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.885584 2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 15 23:36:25.172252 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.885587 2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 15 23:36:25.172252 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.885589 2563 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 15 23:36:25.172252 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.885593 2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 15 23:36:25.172252 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.885596 2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 15 23:36:25.172252 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.885599 2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 15 23:36:25.172252 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.885602 2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 15 23:36:25.172252 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.885606 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 15 23:36:25.172252 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.885608 2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 15 23:36:25.172252 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.885611 2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 15 23:36:25.172252 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.885613 2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 15 23:36:25.173157 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.885616 2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 15 23:36:25.173157 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.885618 2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 15 23:36:25.173157 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.885621 2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 15 23:36:25.173157 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.885623 2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 15 23:36:25.173157 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.885626 2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 15 23:36:25.173157 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.885630 2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 15 23:36:25.173157 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.885633 2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 15 23:36:25.173157 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.885636 2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 15 23:36:25.173157 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.885639 2563 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 15 23:36:25.173157 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.885642 2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 15 23:36:25.173157 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.885645 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 15 23:36:25.173157 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.885648 2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 15 23:36:25.173157 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.885651 2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 15 23:36:25.173157 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.885654 2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 15 23:36:25.173157 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.885656 2563 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 15 23:36:25.173157 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.885659 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 15 23:36:25.173157 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.885661 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 15 23:36:25.173157 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.885664 2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 15 23:36:25.173157 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:24.885666 2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 15 23:36:25.174234 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:24.885673 2563 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 15 23:36:25.174234 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:24.885780 2563 server.go:962] "Client rotation is on, will bootstrap in background" Jul 15 23:36:25.174234 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:24.888136 2563 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 15 23:36:25.174234 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:24.889351 2563 server.go:1019] "Starting client certificate rotation" Jul 15 23:36:25.174234 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:24.889444 2563 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 15 23:36:25.174234 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:24.889477 2563 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 15 23:36:25.174234 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:24.891647 2563 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 15 23:36:25.174234 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:24.892745 2563 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 15 23:36:25.174234 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:24.904852 2563 log.go:25] "Validated CRI v1 runtime API" Jul 15 23:36:25.174234 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:24.907866 2563 log.go:25] "Validated CRI v1 image API" Jul 15 23:36:25.174234 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:24.910307 2563 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 15 23:36:25.174234 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:24.920961 2563 fs.go:135] Filesystem UUIDs: map[4eb27903-4f34-4421-9ab6-c5e96572c02c:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 c458213f-5889-4dce-ab45-241770c657b3:/dev/nvme0n1p4] Jul 15 23:36:25.174889 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:24.920992 2563 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 15 23:36:25.174889 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:24.924014 2563 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 15 23:36:25.175315 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:24.925937 2563 manager.go:216] Machine: {Timestamp:2026-07-15 23:36:24.92463866 +0000 UTC m=+0.298096614 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100180 MemoryCapacity:33164500992 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2ada1df17812c5e967401d8c8f5cd2 SystemUUID:ec2ada1d-f178-12c5-e967-401d8c8f5cd2 BootID:9f729b47-b5bc-4525-86da-dab7d4ee99f2 Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582250496 Type:vfs Inodes:4048401 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:f1:18:a1:65:f1 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:f1:18:a1:65:f1 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:c6:65:6c:55:f2:8e Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164500992 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 15 23:36:25.175315 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:24.926054 2563 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 15 23:36:25.175315 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:24.926158 2563 manager.go:232] Version: {KernelVersion:5.14.0-570.126.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260702-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 15 23:36:25.175315 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:24.928579 2563 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 15 23:36:25.175315 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:24.928603 2563 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-128-197.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 15 23:36:25.175315 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:24.929630 2563 topology_manager.go:138] "Creating topology manager with none policy" Jul 15 23:36:25.175315 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:24.929639 2563 container_manager_linux.go:309] "Creating device plugin manager" Jul 15 23:36:25.175315 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:24.929647 2563 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 15 23:36:25.175315 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:24.930267 2563 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 15 23:36:25.175315 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:24.931024 2563 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 15 23:36:25.175315 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:24.934879 2563 state_mem.go:36] "Initialized new in-memory state store" Jul 15 23:36:25.175315 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:24.935007 2563 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 15 23:36:25.175315 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:24.935033 2563 kubelet.go:486] "Attempting to sync node with API server" Jul 15 23:36:25.175315 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:24.935046 2563 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 15 23:36:25.175315 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:24.935058 2563 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 15 23:36:25.175315 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:24.935067 2563 kubelet.go:398] "Adding apiserver pod source" Jul 15 23:36:25.175315 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:24.935075 2563 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 15 23:36:25.175315 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:24.935366 2563 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 15 23:36:25.176195 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:24.935387 2563 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 15 23:36:25.176195 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:24.937229 2563 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 15 23:36:25.176195 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:24.938539 2563 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 15 23:36:25.176195 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:24.938580 2563 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 15 23:36:25.176195 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:24.939227 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 15 23:36:25.176195 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:24.939240 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 15 23:36:25.176195 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:24.939246 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 15 23:36:25.176195 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:24.939251 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 15 23:36:25.176195 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:24.939260 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 15 23:36:25.176195 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:24.939266 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 15 23:36:25.176195 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:24.939365 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 15 23:36:25.176195 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:24.939371 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 15 23:36:25.176195 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:24.939377 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 15 23:36:25.176195 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:24.939383 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 15 23:36:25.176195 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:24.939403 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 15 23:36:25.176195 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:24.939411 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 15 23:36:25.176195 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:24.940063 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 15 23:36:25.176195 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:24.940071 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 15 23:36:25.176195 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:24.943854 2563 server.go:1268] "Started kubelet" Jul 15 23:36:25.176195 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:24.944080 2563 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 15 23:36:25.176195 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:24.944093 2563 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 15 23:36:25.176195 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:24.944162 2563 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 15 23:36:25.176195 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:24.944463 2563 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-128-197.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 15 23:36:25.176195 ip-10-0-128-197 kubenswrapper[2563]: E0715 23:36:24.944503 2563 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 15 23:36:25.176195 ip-10-0-128-197 kubenswrapper[2563]: E0715 23:36:24.944575 2563 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-128-197.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 15 23:36:25.176195 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:24.945462 2563 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 15 23:36:25.176195 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:24.946114 2563 server.go:310] "Adding debug handlers to kubelet server" Jul 15 23:36:25.176195 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:24.949342 2563 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-cw28r" Jul 15 23:36:25.176195 ip-10-0-128-197 kubenswrapper[2563]: E0715 23:36:24.949847 2563 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 15 23:36:25.176195 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:24.950769 2563 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 15 23:36:25.176195 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:24.950789 2563 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 15 23:36:25.176195 ip-10-0-128-197 kubenswrapper[2563]: E0715 23:36:24.950879 2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-197.ec2.internal\" not found" Jul 15 23:36:25.176195 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:24.952671 2563 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 15 23:36:25.176195 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:24.952859 2563 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 15 23:36:25.176195 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:24.953039 2563 reconstruct.go:97] "Volume reconstruction finished" Jul 15 23:36:25.176195 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:24.953046 2563 reconciler.go:29] "Reconciler: start to sync state" Jul 15 23:36:25.176195 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:24.953209 2563 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 15 23:36:25.176195 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:24.955442 2563 factory.go:55] Registering systemd factory Jul 15 23:36:25.176195 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:24.955454 2563 factory.go:223] Registration of the systemd container factory successfully Jul 15 23:36:25.176195 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:24.955678 2563 factory.go:153] Registering CRI-O factory Jul 15 23:36:25.176195 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:24.955699 2563 factory.go:223] Registration of the crio container factory successfully Jul 15 23:36:25.176195 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:24.955749 2563 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 15 23:36:25.176195 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:24.955771 2563 factory.go:103] Registering Raw factory Jul 15 23:36:25.176195 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:24.955783 2563 manager.go:1196] Started watching for new ooms in manager Jul 15 23:36:25.176195 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:24.956464 2563 manager.go:349] Starting recovery of all containers Jul 15 23:36:25.176195 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:24.956897 2563 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-cw28r" Jul 15 23:36:25.181561 ip-10-0-128-197 kubenswrapper[2563]: E0715 23:36:24.954243 2563 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-197.ec2.internal.18c29ac7b68a5d35 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-197.ec2.internal,UID:ip-10-0-128-197.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-128-197.ec2.internal,},FirstTimestamp:2026-07-15 23:36:24.943828277 +0000 UTC m=+0.317286231,LastTimestamp:2026-07-15 23:36:24.943828277 +0000 UTC m=+0.317286231,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-197.ec2.internal,}" Jul 15 23:36:25.181561 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:24.967086 2563 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 15 23:36:25.181561 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:24.972087 2563 manager.go:354] Recovery completed Jul 15 23:36:25.181561 ip-10-0-128-197 kubenswrapper[2563]: E0715 23:36:24.972397 2563 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-128-197.ec2.internal\" not found" node="ip-10-0-128-197.ec2.internal" Jul 15 23:36:25.181561 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:24.977518 2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 15 23:36:25.181561 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:24.979883 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-197.ec2.internal" event="NodeHasSufficientMemory" Jul 15 23:36:25.181561 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:24.979909 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-197.ec2.internal" event="NodeHasNoDiskPressure" Jul 15 23:36:25.181561 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:24.979923 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-197.ec2.internal" event="NodeHasSufficientPID" Jul 15 23:36:25.181561 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:24.980544 2563 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 15 23:36:25.181561 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:24.980554 2563 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 15 23:36:25.181561 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:24.980573 2563 state_mem.go:36] "Initialized new in-memory state store" Jul 15 23:36:25.181561 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:24.980577 2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 15 23:36:25.181561 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:24.981362 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-197.ec2.internal" event="NodeHasSufficientMemory" Jul 15 23:36:25.181561 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:24.981384 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-197.ec2.internal" event="NodeHasNoDiskPressure" Jul 15 23:36:25.181561 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:24.981394 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-197.ec2.internal" event="NodeHasSufficientPID" Jul 15 23:36:25.181561 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:24.982652 2563 policy_none.go:49] "None policy: Start" Jul 15 23:36:25.181561 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:24.982665 2563 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 15 23:36:25.181561 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:24.982675 2563 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 15 23:36:25.181561 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:24.984765 2563 policy_none.go:47] "Start" Jul 15 23:36:25.181561 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:25.030468 2563 manager.go:337] "Starting Device Plugin manager" Jul 15 23:36:25.181561 ip-10-0-128-197 kubenswrapper[2563]: E0715 23:36:25.030500 2563 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 15 23:36:25.181561 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:25.030512 2563 server.go:87] "Starting device plugin registration server" Jul 15 23:36:25.181561 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:25.030729 2563 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 15 23:36:25.181561 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:25.030743 2563 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 15 23:36:25.181561 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:25.030842 2563 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 15 23:36:25.181561 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:25.030918 2563 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 15 23:36:25.181561 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:25.030927 2563 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 15 23:36:25.181561 ip-10-0-128-197 kubenswrapper[2563]: E0715 23:36:25.031395 2563 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 15 23:36:25.181561 ip-10-0-128-197 kubenswrapper[2563]: E0715 23:36:25.031448 2563 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-128-197.ec2.internal\" not found" Jul 15 23:36:25.181561 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:25.131471 2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 15 23:36:25.181561 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:25.134698 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-197.ec2.internal" event="NodeHasSufficientMemory" Jul 15 23:36:25.181561 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:25.134729 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-197.ec2.internal" event="NodeHasNoDiskPressure" Jul 15 23:36:25.181561 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:25.134743 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-197.ec2.internal" event="NodeHasSufficientPID" Jul 15 23:36:25.181561 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:25.134769 2563 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-128-197.ec2.internal" Jul 15 23:36:25.181561 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:25.146536 2563 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-128-197.ec2.internal" Jul 15 23:36:25.181561 ip-10-0-128-197 kubenswrapper[2563]: E0715 23:36:25.146555 2563 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-128-197.ec2.internal\": node \"ip-10-0-128-197.ec2.internal\" not found" Jul 15 23:36:25.181561 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:25.161411 2563 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 15 23:36:25.181561 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:25.162656 2563 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 15 23:36:25.181561 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:25.162687 2563 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 15 23:36:25.181561 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:25.162707 2563 kubelet.go:2448] "Starting kubelet main sync loop" Jul 15 23:36:25.181561 ip-10-0-128-197 kubenswrapper[2563]: E0715 23:36:25.162746 2563 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 15 23:36:25.181561 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:25.164488 2563 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 15 23:36:25.183337 ip-10-0-128-197 kubenswrapper[2563]: E0715 23:36:25.166054 2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-197.ec2.internal\" not found" Jul 15 23:36:25.367592 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:25.262827 2563 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-197.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-128-197.ec2.internal"] Jul 15 23:36:25.367592 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:25.262924 2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 15 23:36:25.367592 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:25.263967 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-197.ec2.internal" event="NodeHasSufficientMemory" Jul 15 23:36:25.367592 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:25.264013 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-197.ec2.internal" event="NodeHasNoDiskPressure" Jul 15 23:36:25.367592 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:25.264028 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-197.ec2.internal" event="NodeHasSufficientPID" Jul 15 23:36:25.367592 ip-10-0-128-197 kubenswrapper[2563]: E0715 23:36:25.266497 2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-197.ec2.internal\" not found" Jul 15 23:36:25.367592 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:25.266508 2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 15 23:36:25.367592 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:25.266667 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-197.ec2.internal" Jul 15 23:36:25.367592 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:25.266691 2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 15 23:36:25.367592 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:25.267339 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-197.ec2.internal" event="NodeHasSufficientMemory" Jul 15 23:36:25.367592 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:25.267362 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-197.ec2.internal" event="NodeHasNoDiskPressure" Jul 15 23:36:25.367592 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:25.267363 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-197.ec2.internal" event="NodeHasSufficientMemory" Jul 15 23:36:25.367592 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:25.267374 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-197.ec2.internal" event="NodeHasSufficientPID" Jul 15 23:36:25.367592 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:25.267388 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-197.ec2.internal" event="NodeHasNoDiskPressure" Jul 15 23:36:25.367592 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:25.267401 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-197.ec2.internal" event="NodeHasSufficientPID" Jul 15 23:36:25.367592 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:25.269750 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-197.ec2.internal" Jul 15 23:36:25.367592 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:25.269772 2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 15 23:36:25.367592 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:25.270533 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-197.ec2.internal" event="NodeHasSufficientMemory" Jul 15 23:36:25.367592 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:25.270555 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-197.ec2.internal" event="NodeHasNoDiskPressure" Jul 15 23:36:25.367592 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:25.270565 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-197.ec2.internal" event="NodeHasSufficientPID" Jul 15 23:36:25.367592 ip-10-0-128-197 kubenswrapper[2563]: E0715 23:36:25.307802 2563 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-197.ec2.internal\" not found" node="ip-10-0-128-197.ec2.internal" Jul 15 23:36:25.367592 ip-10-0-128-197 kubenswrapper[2563]: E0715 23:36:25.314154 2563 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-197.ec2.internal\" not found" node="ip-10-0-128-197.ec2.internal" Jul 15 23:36:25.367592 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:25.360809 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2e2be974bacf2b9813c8842cd4ec626c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-197.ec2.internal\" (UID: \"2e2be974bacf2b9813c8842cd4ec626c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-197.ec2.internal" Jul 15 23:36:25.367592 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:25.360836 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2e2be974bacf2b9813c8842cd4ec626c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-197.ec2.internal\" (UID: \"2e2be974bacf2b9813c8842cd4ec626c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-197.ec2.internal" Jul 15 23:36:25.367592 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:25.360855 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/1c95fb996cc0f9a497fcf20794fa4db8-config\") pod \"kube-apiserver-proxy-ip-10-0-128-197.ec2.internal\" (UID: \"1c95fb996cc0f9a497fcf20794fa4db8\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-197.ec2.internal" Jul 15 23:36:25.367592 ip-10-0-128-197 kubenswrapper[2563]: E0715 23:36:25.367406 2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-197.ec2.internal\" not found" Jul 15 23:36:25.498223 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:25.460956 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2e2be974bacf2b9813c8842cd4ec626c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-197.ec2.internal\" (UID: \"2e2be974bacf2b9813c8842cd4ec626c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-197.ec2.internal" Jul 15 23:36:25.498223 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:25.461026 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2e2be974bacf2b9813c8842cd4ec626c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-197.ec2.internal\" (UID: \"2e2be974bacf2b9813c8842cd4ec626c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-197.ec2.internal" Jul 15 23:36:25.498223 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:25.461047 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/1c95fb996cc0f9a497fcf20794fa4db8-config\") pod \"kube-apiserver-proxy-ip-10-0-128-197.ec2.internal\" (UID: \"1c95fb996cc0f9a497fcf20794fa4db8\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-197.ec2.internal" Jul 15 23:36:25.498223 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:25.461049 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2e2be974bacf2b9813c8842cd4ec626c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-197.ec2.internal\" (UID: \"2e2be974bacf2b9813c8842cd4ec626c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-197.ec2.internal" Jul 15 23:36:25.498223 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:25.461088 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/1c95fb996cc0f9a497fcf20794fa4db8-config\") pod \"kube-apiserver-proxy-ip-10-0-128-197.ec2.internal\" (UID: \"1c95fb996cc0f9a497fcf20794fa4db8\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-197.ec2.internal" Jul 15 23:36:25.498223 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:25.461126 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2e2be974bacf2b9813c8842cd4ec626c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-197.ec2.internal\" (UID: \"2e2be974bacf2b9813c8842cd4ec626c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-197.ec2.internal" Jul 15 23:36:25.498223 ip-10-0-128-197 kubenswrapper[2563]: E0715 23:36:25.467496 2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-197.ec2.internal\" not found" Jul 15 23:36:25.721221 ip-10-0-128-197 kubenswrapper[2563]: E0715 23:36:25.568654 2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-197.ec2.internal\" not found" Jul 15 23:36:25.721221 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:25.614274 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-197.ec2.internal" Jul 15 23:36:25.721221 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:25.617677 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-197.ec2.internal" Jul 15 23:36:25.721221 ip-10-0-128-197 kubenswrapper[2563]: E0715 23:36:25.668939 2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-197.ec2.internal\" not found" Jul 15 23:36:25.832720 ip-10-0-128-197 kubenswrapper[2563]: E0715 23:36:25.769403 2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-197.ec2.internal\" not found" Jul 15 23:36:25.955642 ip-10-0-128-197 kubenswrapper[2563]: E0715 23:36:25.870310 2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-197.ec2.internal\" not found" Jul 15 23:36:25.965444 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:25.964898 2563 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-14 23:31:25 +0000 UTC" deadline="2028-01-19 09:26:10.257961008 +0000 UTC" Jul 15 23:36:25.965444 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:25.964927 2563 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13257h49m44.293037822s" Jul 15 23:36:25.972556 ip-10-0-128-197 kubenswrapper[2563]: E0715 23:36:25.970443 2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-197.ec2.internal\" not found" Jul 15 23:36:26.078402 ip-10-0-128-197 kubenswrapper[2563]: E0715 23:36:26.077826 2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-197.ec2.internal\" not found" Jul 15 23:36:26.194167 ip-10-0-128-197 kubenswrapper[2563]: E0715 23:36:26.184448 2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-197.ec2.internal\" not found" Jul 15 23:36:26.317702 ip-10-0-128-197 kubenswrapper[2563]: E0715 23:36:26.284768 2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-197.ec2.internal\" not found" Jul 15 23:36:26.441827 ip-10-0-128-197 kubenswrapper[2563]: E0715 23:36:26.385102 2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-197.ec2.internal\" not found" Jul 15 23:36:26.571706 ip-10-0-128-197 kubenswrapper[2563]: E0715 23:36:26.485350 2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-197.ec2.internal\" not found" Jul 15 23:36:26.704868 ip-10-0-128-197 kubenswrapper[2563]: E0715 23:36:26.614537 2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-197.ec2.internal\" not found" Jul 15 23:36:26.715643 ip-10-0-128-197 kubenswrapper[2563]: E0715 23:36:26.715040 2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-197.ec2.internal\" not found" Jul 15 23:36:26.830469 ip-10-0-128-197 kubenswrapper[2563]: E0715 23:36:26.815270 2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-197.ec2.internal\" not found" Jul 15 23:36:26.830469 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:26.820128 2563 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 15 23:36:26.948839 ip-10-0-128-197 kubenswrapper[2563]: E0715 23:36:26.915763 2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-197.ec2.internal\" not found" Jul 15 23:36:27.016636 ip-10-0-128-197 kubenswrapper[2563]: E0715 23:36:27.016611 2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-197.ec2.internal\" not found" Jul 15 23:36:27.073352 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:27.073332 2563 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 15 23:36:27.076857 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:27.076841 2563 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 15 23:36:27.152483 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:27.152403 2563 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-197.ec2.internal" Jul 15 23:36:27.169071 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:27.169053 2563 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 15 23:36:27.170810 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:27.170796 2563 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-197.ec2.internal" Jul 15 23:36:27.179186 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:27.179165 2563 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 15 23:36:27.480121 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:27.480082 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c95fb996cc0f9a497fcf20794fa4db8.slice/crio-1a5204f763961efb8ae1a33d3eff2178e062af4aeec40f045524b205b4259097 WatchSource:0}: Error finding container 1a5204f763961efb8ae1a33d3eff2178e062af4aeec40f045524b205b4259097: Status 404 returned error can't find the container with id 1a5204f763961efb8ae1a33d3eff2178e062af4aeec40f045524b205b4259097 Jul 15 23:36:27.480808 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:27.480782 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e2be974bacf2b9813c8842cd4ec626c.slice/crio-6630f6027ffd5c4a0d2e2999ec33d48590f9f0f317874292adf301122658b85a WatchSource:0}: Error finding container 6630f6027ffd5c4a0d2e2999ec33d48590f9f0f317874292adf301122658b85a: Status 404 returned error can't find the container with id 6630f6027ffd5c4a0d2e2999ec33d48590f9f0f317874292adf301122658b85a Jul 15 23:36:27.487253 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:27.487235 2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 15 23:36:27.825665 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:27.825633 2563 apiserver.go:52] "Watching apiserver" Jul 15 23:36:27.825665 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:27.825648 2563 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 15 23:36:27.828056 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:27.827963 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-197.ec2.internal" event={"ID":"1c95fb996cc0f9a497fcf20794fa4db8","Type":"ContainerStarted","Data":"1a5204f763961efb8ae1a33d3eff2178e062af4aeec40f045524b205b4259097"} Jul 15 23:36:27.829104 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:27.829075 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-197.ec2.internal" event={"ID":"2e2be974bacf2b9813c8842cd4ec626c","Type":"ContainerStarted","Data":"6630f6027ffd5c4a0d2e2999ec33d48590f9f0f317874292adf301122658b85a"} Jul 15 23:36:27.832452 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:27.832428 2563 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 15 23:36:27.833893 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:27.833871 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-6ff59","openshift-cluster-node-tuning-operator/tuned-tm65j","openshift-image-registry/node-ca-khwkr","openshift-multus/multus-v62rk","openshift-multus/network-metrics-daemon-7n4t9","openshift-network-diagnostics/network-check-target-mp4pr","openshift-network-operator/iptables-alerter-v2rfk","kube-system/kube-apiserver-proxy-ip-10-0-128-197.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6pwvf","openshift-dns/node-resolver-z9q4h","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-197.ec2.internal","openshift-multus/multus-additional-cni-plugins-tg24d","openshift-ovn-kubernetes/ovnkube-node-d8f6k","kube-system/global-pull-secret-syncer-4zw2v"] Jul 15 23:36:27.835159 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:27.835135 2563 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 15 23:36:27.837940 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:27.837918 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-v2rfk" Jul 15 23:36:27.841893 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:27.841487 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-d8f6k" Jul 15 23:36:27.841893 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:27.841564 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 15 23:36:27.841893 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:27.841570 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 15 23:36:27.841893 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:27.841588 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-fhdk4\"" Jul 15 23:36:27.842216 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:27.841926 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 15 23:36:27.843906 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:27.843883 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 15 23:36:27.844006 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:27.843908 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 15 23:36:27.844089 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:27.844010 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 15 23:36:27.844155 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:27.844118 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-khwkr" Jul 15 23:36:27.844761 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:27.844740 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-2rjbp\"" Jul 15 23:36:27.844761 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:27.844751 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 15 23:36:27.844903 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:27.844763 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 15 23:36:27.844903 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:27.844750 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 15 23:36:27.846015 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:27.846001 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-nzzzp\"" Jul 15 23:36:27.846239 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:27.846221 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 15 23:36:27.846308 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:27.846221 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 15 23:36:27.846308 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:27.846239 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 15 23:36:27.846503 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:27.846488 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-v62rk" Jul 15 23:36:27.848572 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:27.848556 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 15 23:36:27.848770 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:27.848755 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 15 23:36:27.848994 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:27.848970 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 15 23:36:27.849099 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:27.848973 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 15 23:36:27.849099 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:27.848978 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-zvrpp\"" Jul 15 23:36:27.849099 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:27.849050 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7n4t9" Jul 15 23:36:27.849258 ip-10-0-128-197 kubenswrapper[2563]: E0715 23:36:27.849130 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7n4t9" podUID="d1c182db-bcdc-40a1-bb82-4f8b64173e43" Jul 15 23:36:27.851468 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:27.851449 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mp4pr" Jul 15 23:36:27.851615 ip-10-0-128-197 kubenswrapper[2563]: E0715 23:36:27.851591 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mp4pr" podUID="b0b0bc56-c9c9-4a96-a97b-15f00f3b2099" Jul 15 23:36:27.854057 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:27.854024 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6pwvf" Jul 15 23:36:27.856187 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:27.856165 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 15 23:36:27.856347 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:27.856317 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-rh7fc\"" Jul 15 23:36:27.856429 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:27.856380 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 15 23:36:27.856511 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:27.856492 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-6ff59" Jul 15 23:36:27.856609 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:27.856592 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 15 23:36:27.858586 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:27.858565 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-hc2cn\"" Jul 15 23:36:27.858697 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:27.858593 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 15 23:36:27.858854 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:27.858839 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 15 23:36:27.859039 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:27.859016 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-tm65j" Jul 15 23:36:27.861400 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:27.861372 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-4cxpb\"" Jul 15 23:36:27.861496 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:27.861408 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 15 23:36:27.861496 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:27.861430 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-z9q4h" Jul 15 23:36:27.861496 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:27.861460 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 15 23:36:27.863429 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:27.863411 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 15 23:36:27.863540 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:27.863523 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-x5rvq\"" Jul 15 23:36:27.863858 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:27.863843 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4zw2v" Jul 15 23:36:27.863858 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:27.863849 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 15 23:36:27.863996 ip-10-0-128-197 kubenswrapper[2563]: E0715 23:36:27.863909 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4zw2v" podUID="10a402bd-5fc8-41c2-852a-5df876f74bdb" Jul 15 23:36:27.867933 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:27.867914 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-tg24d" Jul 15 23:36:27.870790 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:27.870762 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 15 23:36:27.870882 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:27.870759 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 15 23:36:27.870882 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:27.870817 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-r6kl2\"" Jul 15 23:36:27.881867 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:27.881849 2563 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-ptp82" Jul 15 23:36:27.889517 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:27.889501 2563 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-ptp82" Jul 15 23:36:27.933027 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:27.933000 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2f64cab3-104a-44f6-b125-ba458f54f058-log-socket\") pod \"ovnkube-node-d8f6k\" (UID: \"2f64cab3-104a-44f6-b125-ba458f54f058\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8f6k" Jul 15 23:36:27.933149 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:27.933040 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2f64cab3-104a-44f6-b125-ba458f54f058-env-overrides\") pod \"ovnkube-node-d8f6k\" (UID: \"2f64cab3-104a-44f6-b125-ba458f54f058\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8f6k" Jul 15 23:36:27.933149 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:27.933072 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2f64cab3-104a-44f6-b125-ba458f54f058-ovn-node-metrics-cert\") pod \"ovnkube-node-d8f6k\" (UID: \"2f64cab3-104a-44f6-b125-ba458f54f058\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8f6k" Jul 15 23:36:27.933297 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:27.933141 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/12202d45-85c8-41de-b95e-f5cf403572ae-multus-cni-dir\") pod \"multus-v62rk\" (UID: \"12202d45-85c8-41de-b95e-f5cf403572ae\") " pod="openshift-multus/multus-v62rk" Jul 15 23:36:27.933297 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:27.933176 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/12202d45-85c8-41de-b95e-f5cf403572ae-cnibin\") pod \"multus-v62rk\" (UID: \"12202d45-85c8-41de-b95e-f5cf403572ae\") " pod="openshift-multus/multus-v62rk" Jul 15 23:36:27.933297 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:27.933203 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/12202d45-85c8-41de-b95e-f5cf403572ae-multus-conf-dir\") pod \"multus-v62rk\" (UID: \"12202d45-85c8-41de-b95e-f5cf403572ae\") " pod="openshift-multus/multus-v62rk" Jul 15 23:36:27.933297 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:27.933230 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k668j\" (UniqueName: \"kubernetes.io/projected/12202d45-85c8-41de-b95e-f5cf403572ae-kube-api-access-k668j\") pod \"multus-v62rk\" (UID: \"12202d45-85c8-41de-b95e-f5cf403572ae\") " pod="openshift-multus/multus-v62rk" Jul 15 23:36:27.933297 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:27.933257 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/abb1e456-40be-492d-9d6b-d10413926119-etc-selinux\") pod \"aws-ebs-csi-driver-node-6pwvf\" (UID: \"abb1e456-40be-492d-9d6b-d10413926119\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6pwvf" Jul 15 23:36:27.933297 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:27.933278 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/abb1e456-40be-492d-9d6b-d10413926119-sys-fs\") pod \"aws-ebs-csi-driver-node-6pwvf\" (UID: \"abb1e456-40be-492d-9d6b-d10413926119\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6pwvf" Jul 15 23:36:27.933621 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:27.933313 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2f64cab3-104a-44f6-b125-ba458f54f058-systemd-units\") pod \"ovnkube-node-d8f6k\" (UID: \"2f64cab3-104a-44f6-b125-ba458f54f058\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8f6k" Jul 15 23:36:27.933621 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:27.933340 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfx42\" (UniqueName: \"kubernetes.io/projected/2f64cab3-104a-44f6-b125-ba458f54f058-kube-api-access-qfx42\") pod \"ovnkube-node-d8f6k\" (UID: \"2f64cab3-104a-44f6-b125-ba458f54f058\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8f6k" Jul 15 23:36:27.933621 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:27.933357 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v47j5\" (UniqueName: \"kubernetes.io/projected/b0b0bc56-c9c9-4a96-a97b-15f00f3b2099-kube-api-access-v47j5\") pod \"network-check-target-mp4pr\" (UID: \"b0b0bc56-c9c9-4a96-a97b-15f00f3b2099\") " pod="openshift-network-diagnostics/network-check-target-mp4pr" Jul 15 23:36:27.933621 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:27.933399 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/12202d45-85c8-41de-b95e-f5cf403572ae-host-run-k8s-cni-cncf-io\") pod \"multus-v62rk\" (UID: \"12202d45-85c8-41de-b95e-f5cf403572ae\") " pod="openshift-multus/multus-v62rk" Jul 15 23:36:27.933621 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:27.933471 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4tzp\" (UniqueName: \"kubernetes.io/projected/d1c182db-bcdc-40a1-bb82-4f8b64173e43-kube-api-access-b4tzp\") pod \"network-metrics-daemon-7n4t9\" (UID: \"d1c182db-bcdc-40a1-bb82-4f8b64173e43\") " pod="openshift-multus/network-metrics-daemon-7n4t9" Jul 15 23:36:27.933621 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:27.933508 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xzlw\" (UniqueName: \"kubernetes.io/projected/498d1e24-acb4-4d75-a5a7-410f00ecc9f9-kube-api-access-7xzlw\") pod \"node-ca-khwkr\" (UID: \"498d1e24-acb4-4d75-a5a7-410f00ecc9f9\") " pod="openshift-image-registry/node-ca-khwkr" Jul 15 23:36:27.933621 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:27.933534 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/12202d45-85c8-41de-b95e-f5cf403572ae-host-run-multus-certs\") pod \"multus-v62rk\" (UID: \"12202d45-85c8-41de-b95e-f5cf403572ae\") " pod="openshift-multus/multus-v62rk" Jul 15 23:36:27.933621 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:27.933559 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/abb1e456-40be-492d-9d6b-d10413926119-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6pwvf\" (UID: \"abb1e456-40be-492d-9d6b-d10413926119\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6pwvf" Jul 15 23:36:27.933621 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:27.933585 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2f64cab3-104a-44f6-b125-ba458f54f058-host-cni-netd\") pod \"ovnkube-node-d8f6k\" (UID: \"2f64cab3-104a-44f6-b125-ba458f54f058\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8f6k" Jul 15 23:36:27.934099 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:27.933636 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2f64cab3-104a-44f6-b125-ba458f54f058-ovnkube-script-lib\") pod \"ovnkube-node-d8f6k\" (UID: \"2f64cab3-104a-44f6-b125-ba458f54f058\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8f6k" Jul 15 23:36:27.934099 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:27.933663 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/498d1e24-acb4-4d75-a5a7-410f00ecc9f9-host\") pod \"node-ca-khwkr\" (UID: \"498d1e24-acb4-4d75-a5a7-410f00ecc9f9\") " pod="openshift-image-registry/node-ca-khwkr" Jul 15 23:36:27.934099 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:27.933691 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/387d6414-0c71-495c-acc0-9d64115d31fe-iptables-alerter-script\") pod \"iptables-alerter-v2rfk\" (UID: \"387d6414-0c71-495c-acc0-9d64115d31fe\") " pod="openshift-network-operator/iptables-alerter-v2rfk" Jul 15 23:36:27.934099 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:27.933717 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/12202d45-85c8-41de-b95e-f5cf403572ae-host-var-lib-cni-multus\") pod \"multus-v62rk\" (UID: \"12202d45-85c8-41de-b95e-f5cf403572ae\") " pod="openshift-multus/multus-v62rk" Jul 15 23:36:27.934099 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:27.933742 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/12202d45-85c8-41de-b95e-f5cf403572ae-host-var-lib-kubelet\") pod \"multus-v62rk\" (UID: \"12202d45-85c8-41de-b95e-f5cf403572ae\") " pod="openshift-multus/multus-v62rk" Jul 15 23:36:27.934099 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:27.933768 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/abb1e456-40be-492d-9d6b-d10413926119-device-dir\") pod \"aws-ebs-csi-driver-node-6pwvf\" (UID: \"abb1e456-40be-492d-9d6b-d10413926119\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6pwvf" Jul 15 23:36:27.934099 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:27.933794 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2f64cab3-104a-44f6-b125-ba458f54f058-run-systemd\") pod \"ovnkube-node-d8f6k\" (UID: \"2f64cab3-104a-44f6-b125-ba458f54f058\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8f6k" Jul 15 23:36:27.934099 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:27.933817 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2f64cab3-104a-44f6-b125-ba458f54f058-host-cni-bin\") pod \"ovnkube-node-d8f6k\" (UID: \"2f64cab3-104a-44f6-b125-ba458f54f058\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8f6k" Jul 15 23:36:27.934099 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:27.933841 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/498d1e24-acb4-4d75-a5a7-410f00ecc9f9-serviceca\") pod \"node-ca-khwkr\" (UID: \"498d1e24-acb4-4d75-a5a7-410f00ecc9f9\") " pod="openshift-image-registry/node-ca-khwkr" Jul 15 23:36:27.934099 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:27.933860 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/12202d45-85c8-41de-b95e-f5cf403572ae-cni-binary-copy\") pod \"multus-v62rk\" (UID: \"12202d45-85c8-41de-b95e-f5cf403572ae\") " pod="openshift-multus/multus-v62rk" Jul 15 23:36:27.934099 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:27.933877 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/12202d45-85c8-41de-b95e-f5cf403572ae-host-run-netns\") pod \"multus-v62rk\" (UID: \"12202d45-85c8-41de-b95e-f5cf403572ae\") " pod="openshift-multus/multus-v62rk" Jul 15 23:36:27.934099 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:27.933896 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/12202d45-85c8-41de-b95e-f5cf403572ae-etc-kubernetes\") pod \"multus-v62rk\" (UID: \"12202d45-85c8-41de-b95e-f5cf403572ae\") " pod="openshift-multus/multus-v62rk" Jul 15 23:36:27.934099 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:27.933922 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d1c182db-bcdc-40a1-bb82-4f8b64173e43-metrics-certs\") pod \"network-metrics-daemon-7n4t9\" (UID: \"d1c182db-bcdc-40a1-bb82-4f8b64173e43\") " pod="openshift-multus/network-metrics-daemon-7n4t9" Jul 15 23:36:27.934099 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:27.933987 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/abb1e456-40be-492d-9d6b-d10413926119-registration-dir\") pod \"aws-ebs-csi-driver-node-6pwvf\" (UID: \"abb1e456-40be-492d-9d6b-d10413926119\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6pwvf" Jul 15 23:36:27.934099 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:27.934021 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2f64cab3-104a-44f6-b125-ba458f54f058-run-openvswitch\") pod \"ovnkube-node-d8f6k\" (UID: \"2f64cab3-104a-44f6-b125-ba458f54f058\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8f6k" Jul 15 23:36:27.934099 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:27.934046 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2f64cab3-104a-44f6-b125-ba458f54f058-node-log\") pod \"ovnkube-node-d8f6k\" (UID: \"2f64cab3-104a-44f6-b125-ba458f54f058\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8f6k" Jul 15 23:36:27.934099 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:27.934072 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/387d6414-0c71-495c-acc0-9d64115d31fe-host-slash\") pod \"iptables-alerter-v2rfk\" (UID: \"387d6414-0c71-495c-acc0-9d64115d31fe\") " pod="openshift-network-operator/iptables-alerter-v2rfk" Jul 15 23:36:27.934937 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:27.934099 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/12202d45-85c8-41de-b95e-f5cf403572ae-os-release\") pod \"multus-v62rk\" (UID: \"12202d45-85c8-41de-b95e-f5cf403572ae\") " pod="openshift-multus/multus-v62rk" Jul 15 23:36:27.934937 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:27.934195 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/12202d45-85c8-41de-b95e-f5cf403572ae-multus-socket-dir-parent\") pod \"multus-v62rk\" (UID: \"12202d45-85c8-41de-b95e-f5cf403572ae\") " pod="openshift-multus/multus-v62rk" Jul 15 23:36:27.934937 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:27.934389 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/12202d45-85c8-41de-b95e-f5cf403572ae-host-var-lib-cni-bin\") pod \"multus-v62rk\" (UID: \"12202d45-85c8-41de-b95e-f5cf403572ae\") " pod="openshift-multus/multus-v62rk" Jul 15 23:36:27.934937 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:27.934416 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/12202d45-85c8-41de-b95e-f5cf403572ae-hostroot\") pod \"multus-v62rk\" (UID: \"12202d45-85c8-41de-b95e-f5cf403572ae\") " pod="openshift-multus/multus-v62rk" Jul 15 23:36:27.934937 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:27.934440 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2f64cab3-104a-44f6-b125-ba458f54f058-host-kubelet\") pod \"ovnkube-node-d8f6k\" (UID: \"2f64cab3-104a-44f6-b125-ba458f54f058\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8f6k" Jul 15 23:36:27.934937 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:27.934461 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2f64cab3-104a-44f6-b125-ba458f54f058-etc-openvswitch\") pod \"ovnkube-node-d8f6k\" (UID: \"2f64cab3-104a-44f6-b125-ba458f54f058\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8f6k" Jul 15 23:36:27.934937 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:27.934506 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/12202d45-85c8-41de-b95e-f5cf403572ae-system-cni-dir\") pod \"multus-v62rk\" (UID: \"12202d45-85c8-41de-b95e-f5cf403572ae\") " pod="openshift-multus/multus-v62rk" Jul 15 23:36:27.934937 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:27.934533 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmm79\" (UniqueName: \"kubernetes.io/projected/abb1e456-40be-492d-9d6b-d10413926119-kube-api-access-cmm79\") pod \"aws-ebs-csi-driver-node-6pwvf\" (UID: \"abb1e456-40be-492d-9d6b-d10413926119\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6pwvf" Jul 15 23:36:27.934937 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:27.934554 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2f64cab3-104a-44f6-b125-ba458f54f058-host-run-netns\") pod \"ovnkube-node-d8f6k\" (UID: \"2f64cab3-104a-44f6-b125-ba458f54f058\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8f6k" Jul 15 23:36:27.934937 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:27.934570 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2f64cab3-104a-44f6-b125-ba458f54f058-var-lib-openvswitch\") pod \"ovnkube-node-d8f6k\" (UID: \"2f64cab3-104a-44f6-b125-ba458f54f058\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8f6k" Jul 15 23:36:27.934937 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:27.934589 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2f64cab3-104a-44f6-b125-ba458f54f058-host-run-ovn-kubernetes\") pod \"ovnkube-node-d8f6k\" (UID: \"2f64cab3-104a-44f6-b125-ba458f54f058\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8f6k" Jul 15 23:36:27.934937 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:27.934614 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2f64cab3-104a-44f6-b125-ba458f54f058-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-d8f6k\" (UID: \"2f64cab3-104a-44f6-b125-ba458f54f058\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8f6k" Jul 15 23:36:27.934937 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:27.934630 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2f64cab3-104a-44f6-b125-ba458f54f058-ovnkube-config\") pod \"ovnkube-node-d8f6k\" (UID: \"2f64cab3-104a-44f6-b125-ba458f54f058\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8f6k" Jul 15 23:36:27.934937 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:27.934653 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zt4pq\" (UniqueName: \"kubernetes.io/projected/387d6414-0c71-495c-acc0-9d64115d31fe-kube-api-access-zt4pq\") pod \"iptables-alerter-v2rfk\" (UID: \"387d6414-0c71-495c-acc0-9d64115d31fe\") " pod="openshift-network-operator/iptables-alerter-v2rfk" Jul 15 23:36:27.934937 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:27.934681 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/12202d45-85c8-41de-b95e-f5cf403572ae-multus-daemon-config\") pod \"multus-v62rk\" (UID: \"12202d45-85c8-41de-b95e-f5cf403572ae\") " pod="openshift-multus/multus-v62rk" Jul 15 23:36:27.934937 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:27.934705 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/abb1e456-40be-492d-9d6b-d10413926119-socket-dir\") pod \"aws-ebs-csi-driver-node-6pwvf\" (UID: \"abb1e456-40be-492d-9d6b-d10413926119\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6pwvf" Jul 15 23:36:27.935748 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:27.934755 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2f64cab3-104a-44f6-b125-ba458f54f058-host-slash\") pod \"ovnkube-node-d8f6k\" (UID: \"2f64cab3-104a-44f6-b125-ba458f54f058\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8f6k" Jul 15 23:36:27.935748 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:27.934787 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2f64cab3-104a-44f6-b125-ba458f54f058-run-ovn\") pod \"ovnkube-node-d8f6k\" (UID: \"2f64cab3-104a-44f6-b125-ba458f54f058\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8f6k" Jul 15 23:36:27.954226 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:27.954204 2563 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 15 23:36:28.035021 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:28.034986 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/abb1e456-40be-492d-9d6b-d10413926119-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6pwvf\" (UID: \"abb1e456-40be-492d-9d6b-d10413926119\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6pwvf" Jul 15 23:36:28.035659 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:28.035035 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2f64cab3-104a-44f6-b125-ba458f54f058-host-cni-netd\") pod \"ovnkube-node-d8f6k\" (UID: \"2f64cab3-104a-44f6-b125-ba458f54f058\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8f6k" Jul 15 23:36:28.035659 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:28.035061 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2f64cab3-104a-44f6-b125-ba458f54f058-ovnkube-script-lib\") pod \"ovnkube-node-d8f6k\" (UID: \"2f64cab3-104a-44f6-b125-ba458f54f058\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8f6k" Jul 15 23:36:28.035659 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:28.035089 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1f66ebce-c0a5-4840-b106-3cedd6448ba5-etc-systemd\") pod \"tuned-tm65j\" (UID: \"1f66ebce-c0a5-4840-b106-3cedd6448ba5\") " pod="openshift-cluster-node-tuning-operator/tuned-tm65j" Jul 15 23:36:28.035659 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:28.035115 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/12202d45-85c8-41de-b95e-f5cf403572ae-host-var-lib-cni-multus\") pod \"multus-v62rk\" (UID: \"12202d45-85c8-41de-b95e-f5cf403572ae\") " pod="openshift-multus/multus-v62rk" Jul 15 23:36:28.035659 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:28.035128 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/abb1e456-40be-492d-9d6b-d10413926119-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6pwvf\" (UID: \"abb1e456-40be-492d-9d6b-d10413926119\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6pwvf" Jul 15 23:36:28.035659 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:28.035146 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2f64cab3-104a-44f6-b125-ba458f54f058-host-cni-bin\") pod \"ovnkube-node-d8f6k\" (UID: \"2f64cab3-104a-44f6-b125-ba458f54f058\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8f6k" Jul 15 23:36:28.035659 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:28.035171 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a87574b4-736b-4e7d-9af6-3b6f89ac1c31-cnibin\") pod \"multus-additional-cni-plugins-tg24d\" (UID: \"a87574b4-736b-4e7d-9af6-3b6f89ac1c31\") " pod="openshift-multus/multus-additional-cni-plugins-tg24d" Jul 15 23:36:28.035659 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:28.035196 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/12202d45-85c8-41de-b95e-f5cf403572ae-host-run-netns\") pod \"multus-v62rk\" (UID: \"12202d45-85c8-41de-b95e-f5cf403572ae\") " pod="openshift-multus/multus-v62rk" Jul 15 23:36:28.035659 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:28.035219 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/12202d45-85c8-41de-b95e-f5cf403572ae-etc-kubernetes\") pod \"multus-v62rk\" (UID: \"12202d45-85c8-41de-b95e-f5cf403572ae\") " pod="openshift-multus/multus-v62rk" Jul 15 23:36:28.035659 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:28.035245 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d1c182db-bcdc-40a1-bb82-4f8b64173e43-metrics-certs\") pod \"network-metrics-daemon-7n4t9\" (UID: \"d1c182db-bcdc-40a1-bb82-4f8b64173e43\") " pod="openshift-multus/network-metrics-daemon-7n4t9" Jul 15 23:36:28.035659 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:28.035271 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/abb1e456-40be-492d-9d6b-d10413926119-registration-dir\") pod \"aws-ebs-csi-driver-node-6pwvf\" (UID: \"abb1e456-40be-492d-9d6b-d10413926119\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6pwvf" Jul 15 23:36:28.035659 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:28.035294 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1f66ebce-c0a5-4840-b106-3cedd6448ba5-etc-sysconfig\") pod \"tuned-tm65j\" (UID: \"1f66ebce-c0a5-4840-b106-3cedd6448ba5\") " pod="openshift-cluster-node-tuning-operator/tuned-tm65j" Jul 15 23:36:28.035659 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:28.035317 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1f66ebce-c0a5-4840-b106-3cedd6448ba5-etc-sysctl-d\") pod \"tuned-tm65j\" (UID: \"1f66ebce-c0a5-4840-b106-3cedd6448ba5\") " pod="openshift-cluster-node-tuning-operator/tuned-tm65j" Jul 15 23:36:28.035659 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:28.035342 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1f66ebce-c0a5-4840-b106-3cedd6448ba5-lib-modules\") pod \"tuned-tm65j\" (UID: \"1f66ebce-c0a5-4840-b106-3cedd6448ba5\") " pod="openshift-cluster-node-tuning-operator/tuned-tm65j" Jul 15 23:36:28.035659 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:28.035369 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/498d1e24-acb4-4d75-a5a7-410f00ecc9f9-serviceca\") pod \"node-ca-khwkr\" (UID: \"498d1e24-acb4-4d75-a5a7-410f00ecc9f9\") " pod="openshift-image-registry/node-ca-khwkr" Jul 15 23:36:28.035659 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:28.035395 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/12202d45-85c8-41de-b95e-f5cf403572ae-multus-socket-dir-parent\") pod \"multus-v62rk\" (UID: \"12202d45-85c8-41de-b95e-f5cf403572ae\") " pod="openshift-multus/multus-v62rk" Jul 15 23:36:28.035659 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:28.035422 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2f64cab3-104a-44f6-b125-ba458f54f058-host-run-ovn-kubernetes\") pod \"ovnkube-node-d8f6k\" (UID: \"2f64cab3-104a-44f6-b125-ba458f54f058\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8f6k" Jul 15 23:36:28.036372 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:28.035451 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1f66ebce-c0a5-4840-b106-3cedd6448ba5-etc-modprobe-d\") pod \"tuned-tm65j\" (UID: \"1f66ebce-c0a5-4840-b106-3cedd6448ba5\") " pod="openshift-cluster-node-tuning-operator/tuned-tm65j" Jul 15 23:36:28.036372 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:28.035478 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-686vw\" (UniqueName: \"kubernetes.io/projected/f9fcf8a8-82dc-4e0b-b211-4611f9436609-kube-api-access-686vw\") pod \"node-resolver-z9q4h\" (UID: \"f9fcf8a8-82dc-4e0b-b211-4611f9436609\") " pod="openshift-dns/node-resolver-z9q4h" Jul 15 23:36:28.036372 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:28.035506 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a87574b4-736b-4e7d-9af6-3b6f89ac1c31-tuning-conf-dir\") pod \"multus-additional-cni-plugins-tg24d\" (UID: \"a87574b4-736b-4e7d-9af6-3b6f89ac1c31\") " pod="openshift-multus/multus-additional-cni-plugins-tg24d" Jul 15 23:36:28.036372 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:28.035533 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmm79\" (UniqueName: \"kubernetes.io/projected/abb1e456-40be-492d-9d6b-d10413926119-kube-api-access-cmm79\") pod \"aws-ebs-csi-driver-node-6pwvf\" (UID: \"abb1e456-40be-492d-9d6b-d10413926119\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6pwvf" Jul 15 23:36:28.036372 ip-10-0-128-197 kubenswrapper[2563]: E0715 23:36:28.035548 2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 23:36:28.036372 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:28.035558 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2f64cab3-104a-44f6-b125-ba458f54f058-host-run-netns\") pod \"ovnkube-node-d8f6k\" (UID: \"2f64cab3-104a-44f6-b125-ba458f54f058\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8f6k" Jul 15 23:36:28.036372 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:28.035583 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2f64cab3-104a-44f6-b125-ba458f54f058-env-overrides\") pod \"ovnkube-node-d8f6k\" (UID: \"2f64cab3-104a-44f6-b125-ba458f54f058\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8f6k" Jul 15 23:36:28.036372 ip-10-0-128-197 kubenswrapper[2563]: E0715 23:36:28.035640 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d1c182db-bcdc-40a1-bb82-4f8b64173e43-metrics-certs podName:d1c182db-bcdc-40a1-bb82-4f8b64173e43 nodeName:}" failed. No retries permitted until 2026-07-15 23:36:28.535597514 +0000 UTC m=+3.909055473 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d1c182db-bcdc-40a1-bb82-4f8b64173e43-metrics-certs") pod "network-metrics-daemon-7n4t9" (UID: "d1c182db-bcdc-40a1-bb82-4f8b64173e43") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 23:36:28.036372 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:28.035667 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f9fcf8a8-82dc-4e0b-b211-4611f9436609-hosts-file\") pod \"node-resolver-z9q4h\" (UID: \"f9fcf8a8-82dc-4e0b-b211-4611f9436609\") " pod="openshift-dns/node-resolver-z9q4h" Jul 15 23:36:28.036372 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:28.035701 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a87574b4-736b-4e7d-9af6-3b6f89ac1c31-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-tg24d\" (UID: \"a87574b4-736b-4e7d-9af6-3b6f89ac1c31\") " pod="openshift-multus/multus-additional-cni-plugins-tg24d" Jul 15 23:36:28.036372 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:28.035735 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfx42\" (UniqueName: \"kubernetes.io/projected/2f64cab3-104a-44f6-b125-ba458f54f058-kube-api-access-qfx42\") pod \"ovnkube-node-d8f6k\" (UID: \"2f64cab3-104a-44f6-b125-ba458f54f058\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8f6k" Jul 15 23:36:28.036372 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:28.035762 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zt4pq\" (UniqueName: \"kubernetes.io/projected/387d6414-0c71-495c-acc0-9d64115d31fe-kube-api-access-zt4pq\") pod \"iptables-alerter-v2rfk\" (UID: \"387d6414-0c71-495c-acc0-9d64115d31fe\") " pod="openshift-network-operator/iptables-alerter-v2rfk" Jul 15 23:36:28.036372 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:28.035801 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2f64cab3-104a-44f6-b125-ba458f54f058-host-slash\") pod \"ovnkube-node-d8f6k\" (UID: \"2f64cab3-104a-44f6-b125-ba458f54f058\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8f6k" Jul 15 23:36:28.036372 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:28.035798 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2f64cab3-104a-44f6-b125-ba458f54f058-host-cni-netd\") pod \"ovnkube-node-d8f6k\" (UID: \"2f64cab3-104a-44f6-b125-ba458f54f058\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8f6k" Jul 15 23:36:28.036372 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:28.035826 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2f64cab3-104a-44f6-b125-ba458f54f058-run-ovn\") pod \"ovnkube-node-d8f6k\" (UID: \"2f64cab3-104a-44f6-b125-ba458f54f058\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8f6k" Jul 15 23:36:28.036372 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:28.035925 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2f64cab3-104a-44f6-b125-ba458f54f058-host-slash\") pod \"ovnkube-node-d8f6k\" (UID: \"2f64cab3-104a-44f6-b125-ba458f54f058\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8f6k" Jul 15 23:36:28.036372 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:28.035957 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2f64cab3-104a-44f6-b125-ba458f54f058-env-overrides\") pod \"ovnkube-node-d8f6k\" (UID: \"2f64cab3-104a-44f6-b125-ba458f54f058\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8f6k" Jul 15 23:36:28.037268 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:28.035968 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2f64cab3-104a-44f6-b125-ba458f54f058-log-socket\") pod \"ovnkube-node-d8f6k\" (UID: \"2f64cab3-104a-44f6-b125-ba458f54f058\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8f6k" Jul 15 23:36:28.037268 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:28.036017 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k668j\" (UniqueName: \"kubernetes.io/projected/12202d45-85c8-41de-b95e-f5cf403572ae-kube-api-access-k668j\") pod \"multus-v62rk\" (UID: \"12202d45-85c8-41de-b95e-f5cf403572ae\") " pod="openshift-multus/multus-v62rk" Jul 15 23:36:28.037268 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:28.036025 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2f64cab3-104a-44f6-b125-ba458f54f058-host-run-ovn-kubernetes\") pod \"ovnkube-node-d8f6k\" (UID: \"2f64cab3-104a-44f6-b125-ba458f54f058\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8f6k" Jul 15 23:36:28.037268 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:28.036046 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/abb1e456-40be-492d-9d6b-d10413926119-etc-selinux\") pod \"aws-ebs-csi-driver-node-6pwvf\" (UID: \"abb1e456-40be-492d-9d6b-d10413926119\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6pwvf" Jul 15 23:36:28.037268 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:28.036075 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/12202d45-85c8-41de-b95e-f5cf403572ae-host-run-netns\") pod \"multus-v62rk\" (UID: \"12202d45-85c8-41de-b95e-f5cf403572ae\") " pod="openshift-multus/multus-v62rk" Jul 15 23:36:28.037268 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:28.036099 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/abb1e456-40be-492d-9d6b-d10413926119-sys-fs\") pod \"aws-ebs-csi-driver-node-6pwvf\" (UID: \"abb1e456-40be-492d-9d6b-d10413926119\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6pwvf" Jul 15 23:36:28.037268 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:28.036131 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2f64cab3-104a-44f6-b125-ba458f54f058-systemd-units\") pod \"ovnkube-node-d8f6k\" (UID: \"2f64cab3-104a-44f6-b125-ba458f54f058\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8f6k" Jul 15 23:36:28.037268 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:28.036148 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/abb1e456-40be-492d-9d6b-d10413926119-etc-selinux\") pod \"aws-ebs-csi-driver-node-6pwvf\" (UID: \"abb1e456-40be-492d-9d6b-d10413926119\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6pwvf" Jul 15 23:36:28.037268 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:28.036160 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xzlw\" (UniqueName: \"kubernetes.io/projected/498d1e24-acb4-4d75-a5a7-410f00ecc9f9-kube-api-access-7xzlw\") pod \"node-ca-khwkr\" (UID: \"498d1e24-acb4-4d75-a5a7-410f00ecc9f9\") " pod="openshift-image-registry/node-ca-khwkr" Jul 15 23:36:28.037268 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:28.036189 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2f64cab3-104a-44f6-b125-ba458f54f058-run-ovn\") pod \"ovnkube-node-d8f6k\" (UID: \"2f64cab3-104a-44f6-b125-ba458f54f058\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8f6k" Jul 15 23:36:28.037268 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:28.036190 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1f66ebce-c0a5-4840-b106-3cedd6448ba5-etc-sysctl-conf\") pod \"tuned-tm65j\" (UID: \"1f66ebce-c0a5-4840-b106-3cedd6448ba5\") " pod="openshift-cluster-node-tuning-operator/tuned-tm65j" Jul 15 23:36:28.037268 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:28.036102 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2f64cab3-104a-44f6-b125-ba458f54f058-host-cni-bin\") pod \"ovnkube-node-d8f6k\" (UID: \"2f64cab3-104a-44f6-b125-ba458f54f058\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8f6k" Jul 15 23:36:28.037268 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:28.036310 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1f66ebce-c0a5-4840-b106-3cedd6448ba5-run\") pod \"tuned-tm65j\" (UID: \"1f66ebce-c0a5-4840-b106-3cedd6448ba5\") " pod="openshift-cluster-node-tuning-operator/tuned-tm65j" Jul 15 23:36:28.037268 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:28.036344 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/12202d45-85c8-41de-b95e-f5cf403572ae-host-run-k8s-cni-cncf-io\") pod \"multus-v62rk\" (UID: \"12202d45-85c8-41de-b95e-f5cf403572ae\") " pod="openshift-multus/multus-v62rk" Jul 15 23:36:28.037268 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:28.036391 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4tzp\" (UniqueName: \"kubernetes.io/projected/d1c182db-bcdc-40a1-bb82-4f8b64173e43-kube-api-access-b4tzp\") pod \"network-metrics-daemon-7n4t9\" (UID: \"d1c182db-bcdc-40a1-bb82-4f8b64173e43\") " pod="openshift-multus/network-metrics-daemon-7n4t9" Jul 15 23:36:28.037268 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:28.036418 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1f66ebce-c0a5-4840-b106-3cedd6448ba5-var-lib-kubelet\") pod \"tuned-tm65j\" (UID: \"1f66ebce-c0a5-4840-b106-3cedd6448ba5\") " pod="openshift-cluster-node-tuning-operator/tuned-tm65j" Jul 15 23:36:28.037268 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:28.036425 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/498d1e24-acb4-4d75-a5a7-410f00ecc9f9-serviceca\") pod \"node-ca-khwkr\" (UID: \"498d1e24-acb4-4d75-a5a7-410f00ecc9f9\") " pod="openshift-image-registry/node-ca-khwkr" Jul 15 23:36:28.038195 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:28.036441 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2deddafa-cbdf-407b-98c2-7c73bd107f1b-agent-certs\") pod \"konnectivity-agent-6ff59\" (UID: \"2deddafa-cbdf-407b-98c2-7c73bd107f1b\") " pod="kube-system/konnectivity-agent-6ff59" Jul 15 23:36:28.038195 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:28.036493 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/12202d45-85c8-41de-b95e-f5cf403572ae-multus-socket-dir-parent\") pod \"multus-v62rk\" (UID: \"12202d45-85c8-41de-b95e-f5cf403572ae\") " pod="openshift-multus/multus-v62rk" Jul 15 23:36:28.038195 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:28.036495 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/12202d45-85c8-41de-b95e-f5cf403572ae-host-run-k8s-cni-cncf-io\") pod \"multus-v62rk\" (UID: \"12202d45-85c8-41de-b95e-f5cf403572ae\") " pod="openshift-multus/multus-v62rk" Jul 15 23:36:28.038195 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:28.036553 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a87574b4-736b-4e7d-9af6-3b6f89ac1c31-system-cni-dir\") pod \"multus-additional-cni-plugins-tg24d\" (UID: \"a87574b4-736b-4e7d-9af6-3b6f89ac1c31\") " pod="openshift-multus/multus-additional-cni-plugins-tg24d" Jul 15 23:36:28.038195 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:28.036562 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2f64cab3-104a-44f6-b125-ba458f54f058-host-run-netns\") pod \"ovnkube-node-d8f6k\" (UID: \"2f64cab3-104a-44f6-b125-ba458f54f058\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8f6k" Jul 15 23:36:28.038195 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:28.036574 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/abb1e456-40be-492d-9d6b-d10413926119-registration-dir\") pod \"aws-ebs-csi-driver-node-6pwvf\" (UID: \"abb1e456-40be-492d-9d6b-d10413926119\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6pwvf" Jul 15 23:36:28.038195 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:28.036586 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/12202d45-85c8-41de-b95e-f5cf403572ae-host-run-multus-certs\") pod \"multus-v62rk\" (UID: \"12202d45-85c8-41de-b95e-f5cf403572ae\") " pod="openshift-multus/multus-v62rk" Jul 15 23:36:28.038195 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:28.036617 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/12202d45-85c8-41de-b95e-f5cf403572ae-etc-kubernetes\") pod \"multus-v62rk\" (UID: \"12202d45-85c8-41de-b95e-f5cf403572ae\") " pod="openshift-multus/multus-v62rk" Jul 15 23:36:28.038195 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:28.036674 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/498d1e24-acb4-4d75-a5a7-410f00ecc9f9-host\") pod \"node-ca-khwkr\" (UID: \"498d1e24-acb4-4d75-a5a7-410f00ecc9f9\") " pod="openshift-image-registry/node-ca-khwkr" Jul 15 23:36:28.038195 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:28.036700 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2f64cab3-104a-44f6-b125-ba458f54f058-systemd-units\") pod \"ovnkube-node-d8f6k\" (UID: \"2f64cab3-104a-44f6-b125-ba458f54f058\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8f6k" Jul 15 23:36:28.038195 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:28.036713 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a87574b4-736b-4e7d-9af6-3b6f89ac1c31-os-release\") pod \"multus-additional-cni-plugins-tg24d\" (UID: \"a87574b4-736b-4e7d-9af6-3b6f89ac1c31\") " pod="openshift-multus/multus-additional-cni-plugins-tg24d" Jul 15 23:36:28.038195 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:28.036741 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2f64cab3-104a-44f6-b125-ba458f54f058-log-socket\") pod \"ovnkube-node-d8f6k\" (UID: \"2f64cab3-104a-44f6-b125-ba458f54f058\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8f6k" Jul 15 23:36:28.038195 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:28.036746 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhwjf\" (UniqueName: \"kubernetes.io/projected/a87574b4-736b-4e7d-9af6-3b6f89ac1c31-kube-api-access-qhwjf\") pod \"multus-additional-cni-plugins-tg24d\" (UID: \"a87574b4-736b-4e7d-9af6-3b6f89ac1c31\") " pod="openshift-multus/multus-additional-cni-plugins-tg24d" Jul 15 23:36:28.038195 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:28.036785 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/12202d45-85c8-41de-b95e-f5cf403572ae-host-var-lib-cni-multus\") pod \"multus-v62rk\" (UID: \"12202d45-85c8-41de-b95e-f5cf403572ae\") " pod="openshift-multus/multus-v62rk" Jul 15 23:36:28.038195 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:28.036792 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/387d6414-0c71-495c-acc0-9d64115d31fe-iptables-alerter-script\") pod \"iptables-alerter-v2rfk\" (UID: \"387d6414-0c71-495c-acc0-9d64115d31fe\") " pod="openshift-network-operator/iptables-alerter-v2rfk" Jul 15 23:36:28.038195 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:28.036852 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/12202d45-85c8-41de-b95e-f5cf403572ae-host-run-multus-certs\") pod \"multus-v62rk\" (UID: \"12202d45-85c8-41de-b95e-f5cf403572ae\") " pod="openshift-multus/multus-v62rk" Jul 15 23:36:28.038195 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:28.036857 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2f64cab3-104a-44f6-b125-ba458f54f058-ovnkube-script-lib\") pod \"ovnkube-node-d8f6k\" (UID: \"2f64cab3-104a-44f6-b125-ba458f54f058\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8f6k" Jul 15 23:36:28.039086 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:28.036889 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/498d1e24-acb4-4d75-a5a7-410f00ecc9f9-host\") pod \"node-ca-khwkr\" (UID: \"498d1e24-acb4-4d75-a5a7-410f00ecc9f9\") " pod="openshift-image-registry/node-ca-khwkr" Jul 15 23:36:28.039086 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:28.036888 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/12202d45-85c8-41de-b95e-f5cf403572ae-host-var-lib-kubelet\") pod \"multus-v62rk\" (UID: \"12202d45-85c8-41de-b95e-f5cf403572ae\") " pod="openshift-multus/multus-v62rk" Jul 15 23:36:28.039086 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:28.036910 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/abb1e456-40be-492d-9d6b-d10413926119-sys-fs\") pod \"aws-ebs-csi-driver-node-6pwvf\" (UID: \"abb1e456-40be-492d-9d6b-d10413926119\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6pwvf" Jul 15 23:36:28.039086 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:28.036930 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/12202d45-85c8-41de-b95e-f5cf403572ae-host-var-lib-kubelet\") pod \"multus-v62rk\" (UID: \"12202d45-85c8-41de-b95e-f5cf403572ae\") " pod="openshift-multus/multus-v62rk" Jul 15 23:36:28.039086 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:28.036932 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/abb1e456-40be-492d-9d6b-d10413926119-device-dir\") pod \"aws-ebs-csi-driver-node-6pwvf\" (UID: \"abb1e456-40be-492d-9d6b-d10413926119\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6pwvf" Jul 15 23:36:28.039086 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:28.036967 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2f64cab3-104a-44f6-b125-ba458f54f058-run-systemd\") pod \"ovnkube-node-d8f6k\" (UID: \"2f64cab3-104a-44f6-b125-ba458f54f058\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8f6k" Jul 15 23:36:28.039086 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:28.036996 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/abb1e456-40be-492d-9d6b-d10413926119-device-dir\") pod \"aws-ebs-csi-driver-node-6pwvf\" (UID: \"abb1e456-40be-492d-9d6b-d10413926119\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6pwvf" Jul 15 23:36:28.039086 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:28.037036 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/1f66ebce-c0a5-4840-b106-3cedd6448ba5-etc-tuned\") pod \"tuned-tm65j\" (UID: \"1f66ebce-c0a5-4840-b106-3cedd6448ba5\") " pod="openshift-cluster-node-tuning-operator/tuned-tm65j" Jul 15 23:36:28.039086 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:28.037112 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2f64cab3-104a-44f6-b125-ba458f54f058-run-systemd\") pod \"ovnkube-node-d8f6k\" (UID: \"2f64cab3-104a-44f6-b125-ba458f54f058\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8f6k" Jul 15 23:36:28.039086 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:28.037143 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/12202d45-85c8-41de-b95e-f5cf403572ae-cni-binary-copy\") pod \"multus-v62rk\" (UID: \"12202d45-85c8-41de-b95e-f5cf403572ae\") " pod="openshift-multus/multus-v62rk" Jul 15 23:36:28.039086 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:28.037185 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2f64cab3-104a-44f6-b125-ba458f54f058-run-openvswitch\") pod \"ovnkube-node-d8f6k\" (UID: \"2f64cab3-104a-44f6-b125-ba458f54f058\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8f6k" Jul 15 23:36:28.039086 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:28.037291 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2f64cab3-104a-44f6-b125-ba458f54f058-node-log\") pod \"ovnkube-node-d8f6k\" (UID: \"2f64cab3-104a-44f6-b125-ba458f54f058\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8f6k" Jul 15 23:36:28.039086 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:28.037330 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/10a402bd-5fc8-41c2-852a-5df876f74bdb-kubelet-config\") pod \"global-pull-secret-syncer-4zw2v\" (UID: \"10a402bd-5fc8-41c2-852a-5df876f74bdb\") " pod="kube-system/global-pull-secret-syncer-4zw2v" Jul 15 23:36:28.039086 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:28.037334 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2f64cab3-104a-44f6-b125-ba458f54f058-run-openvswitch\") pod \"ovnkube-node-d8f6k\" (UID: \"2f64cab3-104a-44f6-b125-ba458f54f058\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8f6k" Jul 15 23:36:28.039086 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:28.037346 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/387d6414-0c71-495c-acc0-9d64115d31fe-iptables-alerter-script\") pod \"iptables-alerter-v2rfk\" (UID: \"387d6414-0c71-495c-acc0-9d64115d31fe\") " pod="openshift-network-operator/iptables-alerter-v2rfk" Jul 15 23:36:28.039086 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:28.037374 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2f64cab3-104a-44f6-b125-ba458f54f058-node-log\") pod \"ovnkube-node-d8f6k\" (UID: \"2f64cab3-104a-44f6-b125-ba458f54f058\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8f6k" Jul 15 23:36:28.039086 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:28.037407 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/10a402bd-5fc8-41c2-852a-5df876f74bdb-original-pull-secret\") pod \"global-pull-secret-syncer-4zw2v\" (UID: \"10a402bd-5fc8-41c2-852a-5df876f74bdb\") " pod="kube-system/global-pull-secret-syncer-4zw2v" Jul 15 23:36:28.040012 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:28.037441 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a87574b4-736b-4e7d-9af6-3b6f89ac1c31-cni-binary-copy\") pod \"multus-additional-cni-plugins-tg24d\" (UID: \"a87574b4-736b-4e7d-9af6-3b6f89ac1c31\") " pod="openshift-multus/multus-additional-cni-plugins-tg24d" Jul 15 23:36:28.040012 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:28.037484 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/387d6414-0c71-495c-acc0-9d64115d31fe-host-slash\") pod \"iptables-alerter-v2rfk\" (UID: \"387d6414-0c71-495c-acc0-9d64115d31fe\") " pod="openshift-network-operator/iptables-alerter-v2rfk" Jul 15 23:36:28.040012 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:28.037523 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/12202d45-85c8-41de-b95e-f5cf403572ae-os-release\") pod \"multus-v62rk\" (UID: \"12202d45-85c8-41de-b95e-f5cf403572ae\") " pod="openshift-multus/multus-v62rk" Jul 15 23:36:28.040012 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:28.037530 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/387d6414-0c71-495c-acc0-9d64115d31fe-host-slash\") pod \"iptables-alerter-v2rfk\" (UID: \"387d6414-0c71-495c-acc0-9d64115d31fe\") " pod="openshift-network-operator/iptables-alerter-v2rfk" Jul 15 23:36:28.040012 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:28.037556 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/12202d45-85c8-41de-b95e-f5cf403572ae-host-var-lib-cni-bin\") pod \"multus-v62rk\" (UID: \"12202d45-85c8-41de-b95e-f5cf403572ae\") " pod="openshift-multus/multus-v62rk" Jul 15 23:36:28.040012 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:28.037589 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/12202d45-85c8-41de-b95e-f5cf403572ae-hostroot\") pod \"multus-v62rk\" (UID: \"12202d45-85c8-41de-b95e-f5cf403572ae\") " pod="openshift-multus/multus-v62rk" Jul 15 23:36:28.040012 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:28.037613 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2f64cab3-104a-44f6-b125-ba458f54f058-host-kubelet\") pod \"ovnkube-node-d8f6k\" (UID: \"2f64cab3-104a-44f6-b125-ba458f54f058\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8f6k" Jul 15 23:36:28.040012 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:28.037625 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/12202d45-85c8-41de-b95e-f5cf403572ae-os-release\") pod \"multus-v62rk\" (UID: \"12202d45-85c8-41de-b95e-f5cf403572ae\") " pod="openshift-multus/multus-v62rk" Jul 15 23:36:28.040012 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:28.037649 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2f64cab3-104a-44f6-b125-ba458f54f058-etc-openvswitch\") pod \"ovnkube-node-d8f6k\" (UID: \"2f64cab3-104a-44f6-b125-ba458f54f058\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8f6k" Jul 15 23:36:28.040012 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:28.037669 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/12202d45-85c8-41de-b95e-f5cf403572ae-host-var-lib-cni-bin\") pod \"multus-v62rk\" (UID: \"12202d45-85c8-41de-b95e-f5cf403572ae\") " pod="openshift-multus/multus-v62rk" Jul 15 23:36:28.040012 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:28.037676 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2f64cab3-104a-44f6-b125-ba458f54f058-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-d8f6k\" (UID: \"2f64cab3-104a-44f6-b125-ba458f54f058\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8f6k" Jul 15 23:36:28.040012 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:28.037702 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/12202d45-85c8-41de-b95e-f5cf403572ae-hostroot\") pod \"multus-v62rk\" (UID: \"12202d45-85c8-41de-b95e-f5cf403572ae\") " pod="openshift-multus/multus-v62rk" Jul 15 23:36:28.040012 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:28.037707 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/12202d45-85c8-41de-b95e-f5cf403572ae-cni-binary-copy\") pod \"multus-v62rk\" (UID: \"12202d45-85c8-41de-b95e-f5cf403572ae\") " pod="openshift-multus/multus-v62rk" Jul 15 23:36:28.040012 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:28.037764 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2f64cab3-104a-44f6-b125-ba458f54f058-host-kubelet\") pod \"ovnkube-node-d8f6k\" (UID: \"2f64cab3-104a-44f6-b125-ba458f54f058\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8f6k" Jul 15 23:36:28.040012 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:28.037792 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2f64cab3-104a-44f6-b125-ba458f54f058-etc-openvswitch\") pod \"ovnkube-node-d8f6k\" (UID: \"2f64cab3-104a-44f6-b125-ba458f54f058\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8f6k" Jul 15 23:36:28.040012 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:28.037803 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2f64cab3-104a-44f6-b125-ba458f54f058-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-d8f6k\" (UID: \"2f64cab3-104a-44f6-b125-ba458f54f058\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8f6k" Jul 15 23:36:28.040012 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:28.037824 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/12202d45-85c8-41de-b95e-f5cf403572ae-system-cni-dir\") pod \"multus-v62rk\" (UID: \"12202d45-85c8-41de-b95e-f5cf403572ae\") " pod="openshift-multus/multus-v62rk" Jul 15 23:36:28.040012 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:28.037853 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2f64cab3-104a-44f6-b125-ba458f54f058-var-lib-openvswitch\") pod \"ovnkube-node-d8f6k\" (UID: \"2f64cab3-104a-44f6-b125-ba458f54f058\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8f6k" Jul 15 23:36:28.040939 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:28.037869 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/12202d45-85c8-41de-b95e-f5cf403572ae-system-cni-dir\") pod \"multus-v62rk\" (UID: \"12202d45-85c8-41de-b95e-f5cf403572ae\") " pod="openshift-multus/multus-v62rk" Jul 15 23:36:28.040939 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:28.037879 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2f64cab3-104a-44f6-b125-ba458f54f058-ovnkube-config\") pod \"ovnkube-node-d8f6k\" (UID: \"2f64cab3-104a-44f6-b125-ba458f54f058\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8f6k" Jul 15 23:36:28.040939 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:28.037905 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2f64cab3-104a-44f6-b125-ba458f54f058-ovn-node-metrics-cert\") pod \"ovnkube-node-d8f6k\" (UID: \"2f64cab3-104a-44f6-b125-ba458f54f058\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8f6k" Jul 15 23:36:28.040939 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:28.037931 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2f64cab3-104a-44f6-b125-ba458f54f058-var-lib-openvswitch\") pod \"ovnkube-node-d8f6k\" (UID: \"2f64cab3-104a-44f6-b125-ba458f54f058\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8f6k" Jul 15 23:36:28.040939 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:28.037933 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1f66ebce-c0a5-4840-b106-3cedd6448ba5-etc-kubernetes\") pod \"tuned-tm65j\" (UID: \"1f66ebce-c0a5-4840-b106-3cedd6448ba5\") " pod="openshift-cluster-node-tuning-operator/tuned-tm65j" Jul 15 23:36:28.040939 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:28.038090 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1f66ebce-c0a5-4840-b106-3cedd6448ba5-host\") pod \"tuned-tm65j\" (UID: \"1f66ebce-c0a5-4840-b106-3cedd6448ba5\") " pod="openshift-cluster-node-tuning-operator/tuned-tm65j" Jul 15 23:36:28.040939 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:28.038117 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2deddafa-cbdf-407b-98c2-7c73bd107f1b-konnectivity-ca\") pod \"konnectivity-agent-6ff59\" (UID: \"2deddafa-cbdf-407b-98c2-7c73bd107f1b\") " pod="kube-system/konnectivity-agent-6ff59" Jul 15 23:36:28.040939 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:28.038144 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/12202d45-85c8-41de-b95e-f5cf403572ae-multus-daemon-config\") pod \"multus-v62rk\" (UID: \"12202d45-85c8-41de-b95e-f5cf403572ae\") " pod="openshift-multus/multus-v62rk" Jul 15 23:36:28.040939 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:28.038170 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/abb1e456-40be-492d-9d6b-d10413926119-socket-dir\") pod \"aws-ebs-csi-driver-node-6pwvf\" (UID: \"abb1e456-40be-492d-9d6b-d10413926119\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6pwvf" Jul 15 23:36:28.040939 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:28.038195 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-txjg5\" (UniqueName: \"kubernetes.io/projected/1f66ebce-c0a5-4840-b106-3cedd6448ba5-kube-api-access-txjg5\") pod \"tuned-tm65j\" (UID: \"1f66ebce-c0a5-4840-b106-3cedd6448ba5\") " pod="openshift-cluster-node-tuning-operator/tuned-tm65j" Jul 15 23:36:28.040939 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:28.038224 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f9fcf8a8-82dc-4e0b-b211-4611f9436609-tmp-dir\") pod \"node-resolver-z9q4h\" (UID: \"f9fcf8a8-82dc-4e0b-b211-4611f9436609\") " pod="openshift-dns/node-resolver-z9q4h" Jul 15 23:36:28.040939 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:28.038250 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a87574b4-736b-4e7d-9af6-3b6f89ac1c31-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-tg24d\" (UID: \"a87574b4-736b-4e7d-9af6-3b6f89ac1c31\") " pod="openshift-multus/multus-additional-cni-plugins-tg24d" Jul 15 23:36:28.040939 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:28.038290 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/12202d45-85c8-41de-b95e-f5cf403572ae-multus-cni-dir\") pod \"multus-v62rk\" (UID: \"12202d45-85c8-41de-b95e-f5cf403572ae\") " pod="openshift-multus/multus-v62rk" Jul 15 23:36:28.040939 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:28.038295 2563 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 15 23:36:28.040939 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:28.038319 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/12202d45-85c8-41de-b95e-f5cf403572ae-cnibin\") pod \"multus-v62rk\" (UID: \"12202d45-85c8-41de-b95e-f5cf403572ae\") " pod="openshift-multus/multus-v62rk" Jul 15 23:36:28.040939 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:28.038344 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/12202d45-85c8-41de-b95e-f5cf403572ae-multus-conf-dir\") pod \"multus-v62rk\" (UID: \"12202d45-85c8-41de-b95e-f5cf403572ae\") " pod="openshift-multus/multus-v62rk" Jul 15 23:36:28.040939 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:28.038369 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1f66ebce-c0a5-4840-b106-3cedd6448ba5-sys\") pod \"tuned-tm65j\" (UID: \"1f66ebce-c0a5-4840-b106-3cedd6448ba5\") " pod="openshift-cluster-node-tuning-operator/tuned-tm65j" Jul 15 23:36:28.040939 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:28.038384 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2f64cab3-104a-44f6-b125-ba458f54f058-ovnkube-config\") pod \"ovnkube-node-d8f6k\" (UID: \"2f64cab3-104a-44f6-b125-ba458f54f058\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8f6k" Jul 15 23:36:28.041858 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:28.038391 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1f66ebce-c0a5-4840-b106-3cedd6448ba5-tmp\") pod \"tuned-tm65j\" (UID: \"1f66ebce-c0a5-4840-b106-3cedd6448ba5\") " pod="openshift-cluster-node-tuning-operator/tuned-tm65j" Jul 15 23:36:28.041858 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:28.038415 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/10a402bd-5fc8-41c2-852a-5df876f74bdb-dbus\") pod \"global-pull-secret-syncer-4zw2v\" (UID: \"10a402bd-5fc8-41c2-852a-5df876f74bdb\") " pod="kube-system/global-pull-secret-syncer-4zw2v" Jul 15 23:36:28.041858 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:28.038443 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v47j5\" (UniqueName: \"kubernetes.io/projected/b0b0bc56-c9c9-4a96-a97b-15f00f3b2099-kube-api-access-v47j5\") pod \"network-check-target-mp4pr\" (UID: \"b0b0bc56-c9c9-4a96-a97b-15f00f3b2099\") " pod="openshift-network-diagnostics/network-check-target-mp4pr" Jul 15 23:36:28.041858 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:28.038654 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/abb1e456-40be-492d-9d6b-d10413926119-socket-dir\") pod \"aws-ebs-csi-driver-node-6pwvf\" (UID: \"abb1e456-40be-492d-9d6b-d10413926119\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6pwvf" Jul 15 23:36:28.041858 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:28.038715 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/12202d45-85c8-41de-b95e-f5cf403572ae-multus-conf-dir\") pod \"multus-v62rk\" (UID: \"12202d45-85c8-41de-b95e-f5cf403572ae\") " pod="openshift-multus/multus-v62rk" Jul 15 23:36:28.041858 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:28.038775 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/12202d45-85c8-41de-b95e-f5cf403572ae-cnibin\") pod \"multus-v62rk\" (UID: \"12202d45-85c8-41de-b95e-f5cf403572ae\") " pod="openshift-multus/multus-v62rk" Jul 15 23:36:28.041858 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:28.038785 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/12202d45-85c8-41de-b95e-f5cf403572ae-multus-cni-dir\") pod \"multus-v62rk\" (UID: \"12202d45-85c8-41de-b95e-f5cf403572ae\") " pod="openshift-multus/multus-v62rk" Jul 15 23:36:28.041858 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:28.039365 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/12202d45-85c8-41de-b95e-f5cf403572ae-multus-daemon-config\") pod \"multus-v62rk\" (UID: \"12202d45-85c8-41de-b95e-f5cf403572ae\") " pod="openshift-multus/multus-v62rk" Jul 15 23:36:28.042895 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:28.042871 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2f64cab3-104a-44f6-b125-ba458f54f058-ovn-node-metrics-cert\") pod \"ovnkube-node-d8f6k\" (UID: \"2f64cab3-104a-44f6-b125-ba458f54f058\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8f6k" Jul 15 23:36:28.046404 ip-10-0-128-197 kubenswrapper[2563]: E0715 23:36:28.046382 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 15 23:36:28.046519 ip-10-0-128-197 kubenswrapper[2563]: E0715 23:36:28.046410 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 15 23:36:28.046519 ip-10-0-128-197 kubenswrapper[2563]: E0715 23:36:28.046424 2563 projected.go:196] Error preparing data for projected volume kube-api-access-v47j5 for pod openshift-network-diagnostics/network-check-target-mp4pr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 23:36:28.046519 ip-10-0-128-197 kubenswrapper[2563]: E0715 23:36:28.046513 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b0b0bc56-c9c9-4a96-a97b-15f00f3b2099-kube-api-access-v47j5 podName:b0b0bc56-c9c9-4a96-a97b-15f00f3b2099 nodeName:}" failed. No retries permitted until 2026-07-15 23:36:28.54649634 +0000 UTC m=+3.919954283 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-v47j5" (UniqueName: "kubernetes.io/projected/b0b0bc56-c9c9-4a96-a97b-15f00f3b2099-kube-api-access-v47j5") pod "network-check-target-mp4pr" (UID: "b0b0bc56-c9c9-4a96-a97b-15f00f3b2099") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 23:36:28.048680 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:28.048610 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfx42\" (UniqueName: \"kubernetes.io/projected/2f64cab3-104a-44f6-b125-ba458f54f058-kube-api-access-qfx42\") pod \"ovnkube-node-d8f6k\" (UID: \"2f64cab3-104a-44f6-b125-ba458f54f058\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8f6k" Jul 15 23:36:28.051941 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:28.051915 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4tzp\" (UniqueName: \"kubernetes.io/projected/d1c182db-bcdc-40a1-bb82-4f8b64173e43-kube-api-access-b4tzp\") pod \"network-metrics-daemon-7n4t9\" (UID: \"d1c182db-bcdc-40a1-bb82-4f8b64173e43\") " pod="openshift-multus/network-metrics-daemon-7n4t9" Jul 15 23:36:28.052825 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:28.052677 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k668j\" (UniqueName: \"kubernetes.io/projected/12202d45-85c8-41de-b95e-f5cf403572ae-kube-api-access-k668j\") pod \"multus-v62rk\" (UID: \"12202d45-85c8-41de-b95e-f5cf403572ae\") " pod="openshift-multus/multus-v62rk" Jul 15 23:36:28.052917 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:28.052860 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xzlw\" (UniqueName: \"kubernetes.io/projected/498d1e24-acb4-4d75-a5a7-410f00ecc9f9-kube-api-access-7xzlw\") pod \"node-ca-khwkr\" (UID: \"498d1e24-acb4-4d75-a5a7-410f00ecc9f9\") " pod="openshift-image-registry/node-ca-khwkr" Jul 15 23:36:28.053319 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:28.053297 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zt4pq\" (UniqueName: \"kubernetes.io/projected/387d6414-0c71-495c-acc0-9d64115d31fe-kube-api-access-zt4pq\") pod \"iptables-alerter-v2rfk\" (UID: \"387d6414-0c71-495c-acc0-9d64115d31fe\") " pod="openshift-network-operator/iptables-alerter-v2rfk" Jul 15 23:36:28.053738 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:28.053717 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmm79\" (UniqueName: \"kubernetes.io/projected/abb1e456-40be-492d-9d6b-d10413926119-kube-api-access-cmm79\") pod \"aws-ebs-csi-driver-node-6pwvf\" (UID: \"abb1e456-40be-492d-9d6b-d10413926119\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6pwvf" Jul 15 23:36:28.138949 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:28.138860 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/10a402bd-5fc8-41c2-852a-5df876f74bdb-kubelet-config\") pod \"global-pull-secret-syncer-4zw2v\" (UID: \"10a402bd-5fc8-41c2-852a-5df876f74bdb\") " pod="kube-system/global-pull-secret-syncer-4zw2v" Jul 15 23:36:28.138949 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:28.138905 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/10a402bd-5fc8-41c2-852a-5df876f74bdb-original-pull-secret\") pod \"global-pull-secret-syncer-4zw2v\" (UID: \"10a402bd-5fc8-41c2-852a-5df876f74bdb\") " pod="kube-system/global-pull-secret-syncer-4zw2v" Jul 15 23:36:28.138949 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:28.138934 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a87574b4-736b-4e7d-9af6-3b6f89ac1c31-cni-binary-copy\") pod \"multus-additional-cni-plugins-tg24d\" (UID: \"a87574b4-736b-4e7d-9af6-3b6f89ac1c31\") " pod="openshift-multus/multus-additional-cni-plugins-tg24d" Jul 15 23:36:28.139259 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:28.138971 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1f66ebce-c0a5-4840-b106-3cedd6448ba5-etc-kubernetes\") pod \"tuned-tm65j\" (UID: \"1f66ebce-c0a5-4840-b106-3cedd6448ba5\") " pod="openshift-cluster-node-tuning-operator/tuned-tm65j" Jul 15 23:36:28.139259 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:28.139009 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/10a402bd-5fc8-41c2-852a-5df876f74bdb-kubelet-config\") pod \"global-pull-secret-syncer-4zw2v\" (UID: \"10a402bd-5fc8-41c2-852a-5df876f74bdb\") " pod="kube-system/global-pull-secret-syncer-4zw2v" Jul 15 23:36:28.139259 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:28.139016 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1f66ebce-c0a5-4840-b106-3cedd6448ba5-host\") pod \"tuned-tm65j\" (UID: \"1f66ebce-c0a5-4840-b106-3cedd6448ba5\") " pod="openshift-cluster-node-tuning-operator/tuned-tm65j" Jul 15 23:36:28.139259 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:28.139069 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2deddafa-cbdf-407b-98c2-7c73bd107f1b-konnectivity-ca\") pod \"konnectivity-agent-6ff59\" (UID: \"2deddafa-cbdf-407b-98c2-7c73bd107f1b\") " pod="kube-system/konnectivity-agent-6ff59" Jul 15 23:36:28.139259 ip-10-0-128-197 kubenswrapper[2563]: E0715 23:36:28.139092 2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 15 23:36:28.139259 ip-10-0-128-197 kubenswrapper[2563]: E0715 23:36:28.139170 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/10a402bd-5fc8-41c2-852a-5df876f74bdb-original-pull-secret podName:10a402bd-5fc8-41c2-852a-5df876f74bdb nodeName:}" failed. No retries permitted until 2026-07-15 23:36:28.639144626 +0000 UTC m=+4.012602568 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/10a402bd-5fc8-41c2-852a-5df876f74bdb-original-pull-secret") pod "global-pull-secret-syncer-4zw2v" (UID: "10a402bd-5fc8-41c2-852a-5df876f74bdb") : object "kube-system"/"original-pull-secret" not registered Jul 15 23:36:28.139259 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:28.139096 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txjg5\" (UniqueName: \"kubernetes.io/projected/1f66ebce-c0a5-4840-b106-3cedd6448ba5-kube-api-access-txjg5\") pod \"tuned-tm65j\" (UID: \"1f66ebce-c0a5-4840-b106-3cedd6448ba5\") " pod="openshift-cluster-node-tuning-operator/tuned-tm65j" Jul 15 23:36:28.139259 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:28.139215 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f9fcf8a8-82dc-4e0b-b211-4611f9436609-tmp-dir\") pod \"node-resolver-z9q4h\" (UID: \"f9fcf8a8-82dc-4e0b-b211-4611f9436609\") " pod="openshift-dns/node-resolver-z9q4h" Jul 15 23:36:28.139259 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:28.139249 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a87574b4-736b-4e7d-9af6-3b6f89ac1c31-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-tg24d\" (UID: \"a87574b4-736b-4e7d-9af6-3b6f89ac1c31\") " pod="openshift-multus/multus-additional-cni-plugins-tg24d" Jul 15 23:36:28.139725 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:28.139279 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1f66ebce-c0a5-4840-b106-3cedd6448ba5-sys\") pod \"tuned-tm65j\" (UID: \"1f66ebce-c0a5-4840-b106-3cedd6448ba5\") " pod="openshift-cluster-node-tuning-operator/tuned-tm65j" Jul 15 23:36:28.139725 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:28.139306 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1f66ebce-c0a5-4840-b106-3cedd6448ba5-tmp\") pod \"tuned-tm65j\" (UID: \"1f66ebce-c0a5-4840-b106-3cedd6448ba5\") " pod="openshift-cluster-node-tuning-operator/tuned-tm65j" Jul 15 23:36:28.139725 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:28.139310 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1f66ebce-c0a5-4840-b106-3cedd6448ba5-host\") pod \"tuned-tm65j\" (UID: \"1f66ebce-c0a5-4840-b106-3cedd6448ba5\") " pod="openshift-cluster-node-tuning-operator/tuned-tm65j" Jul 15 23:36:28.139725 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:28.139328 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/10a402bd-5fc8-41c2-852a-5df876f74bdb-dbus\") pod \"global-pull-secret-syncer-4zw2v\" (UID: \"10a402bd-5fc8-41c2-852a-5df876f74bdb\") " pod="kube-system/global-pull-secret-syncer-4zw2v" Jul 15 23:36:28.139725 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:28.139369 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1f66ebce-c0a5-4840-b106-3cedd6448ba5-etc-systemd\") pod \"tuned-tm65j\" (UID: \"1f66ebce-c0a5-4840-b106-3cedd6448ba5\") " pod="openshift-cluster-node-tuning-operator/tuned-tm65j" Jul 15 23:36:28.139725 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:28.139397 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a87574b4-736b-4e7d-9af6-3b6f89ac1c31-cnibin\") pod \"multus-additional-cni-plugins-tg24d\" (UID: \"a87574b4-736b-4e7d-9af6-3b6f89ac1c31\") " pod="openshift-multus/multus-additional-cni-plugins-tg24d" Jul 15 23:36:28.139725 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:28.139438 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1f66ebce-c0a5-4840-b106-3cedd6448ba5-etc-sysconfig\") pod \"tuned-tm65j\" (UID: \"1f66ebce-c0a5-4840-b106-3cedd6448ba5\") " pod="openshift-cluster-node-tuning-operator/tuned-tm65j" Jul 15 23:36:28.139725 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:28.139444 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1f66ebce-c0a5-4840-b106-3cedd6448ba5-sys\") pod \"tuned-tm65j\" (UID: \"1f66ebce-c0a5-4840-b106-3cedd6448ba5\") " pod="openshift-cluster-node-tuning-operator/tuned-tm65j" Jul 15 23:36:28.139725 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:28.139464 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1f66ebce-c0a5-4840-b106-3cedd6448ba5-etc-sysctl-d\") pod \"tuned-tm65j\" (UID: \"1f66ebce-c0a5-4840-b106-3cedd6448ba5\") " pod="openshift-cluster-node-tuning-operator/tuned-tm65j" Jul 15 23:36:28.139725 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:28.139489 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1f66ebce-c0a5-4840-b106-3cedd6448ba5-lib-modules\") pod \"tuned-tm65j\" (UID: \"1f66ebce-c0a5-4840-b106-3cedd6448ba5\") " pod="openshift-cluster-node-tuning-operator/tuned-tm65j" Jul 15 23:36:28.139725 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:28.139516 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1f66ebce-c0a5-4840-b106-3cedd6448ba5-etc-modprobe-d\") pod \"tuned-tm65j\" (UID: \"1f66ebce-c0a5-4840-b106-3cedd6448ba5\") " pod="openshift-cluster-node-tuning-operator/tuned-tm65j" Jul 15 23:36:28.139725 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:28.139542 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-686vw\" (UniqueName: \"kubernetes.io/projected/f9fcf8a8-82dc-4e0b-b211-4611f9436609-kube-api-access-686vw\") pod \"node-resolver-z9q4h\" (UID: \"f9fcf8a8-82dc-4e0b-b211-4611f9436609\") " pod="openshift-dns/node-resolver-z9q4h" Jul 15 23:36:28.139725 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:28.139567 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a87574b4-736b-4e7d-9af6-3b6f89ac1c31-tuning-conf-dir\") pod \"multus-additional-cni-plugins-tg24d\" (UID: \"a87574b4-736b-4e7d-9af6-3b6f89ac1c31\") " pod="openshift-multus/multus-additional-cni-plugins-tg24d" Jul 15 23:36:28.139725 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:28.139596 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f9fcf8a8-82dc-4e0b-b211-4611f9436609-hosts-file\") pod \"node-resolver-z9q4h\" (UID: \"f9fcf8a8-82dc-4e0b-b211-4611f9436609\") " pod="openshift-dns/node-resolver-z9q4h" Jul 15 23:36:28.139725 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:28.139624 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a87574b4-736b-4e7d-9af6-3b6f89ac1c31-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-tg24d\" (UID: \"a87574b4-736b-4e7d-9af6-3b6f89ac1c31\") " pod="openshift-multus/multus-additional-cni-plugins-tg24d" Jul 15 23:36:28.139725 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:28.139639 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a87574b4-736b-4e7d-9af6-3b6f89ac1c31-cni-binary-copy\") pod \"multus-additional-cni-plugins-tg24d\" (UID: \"a87574b4-736b-4e7d-9af6-3b6f89ac1c31\") " pod="openshift-multus/multus-additional-cni-plugins-tg24d" Jul 15 23:36:28.139725 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:28.139660 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1f66ebce-c0a5-4840-b106-3cedd6448ba5-etc-sysctl-conf\") pod \"tuned-tm65j\" (UID: \"1f66ebce-c0a5-4840-b106-3cedd6448ba5\") " pod="openshift-cluster-node-tuning-operator/tuned-tm65j" Jul 15 23:36:28.139725 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:28.139684 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1f66ebce-c0a5-4840-b106-3cedd6448ba5-run\") pod \"tuned-tm65j\" (UID: \"1f66ebce-c0a5-4840-b106-3cedd6448ba5\") " pod="openshift-cluster-node-tuning-operator/tuned-tm65j" Jul 15 23:36:28.140629 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:28.139709 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1f66ebce-c0a5-4840-b106-3cedd6448ba5-var-lib-kubelet\") pod \"tuned-tm65j\" (UID: \"1f66ebce-c0a5-4840-b106-3cedd6448ba5\") " pod="openshift-cluster-node-tuning-operator/tuned-tm65j" Jul 15 23:36:28.140629 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:28.139738 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2deddafa-cbdf-407b-98c2-7c73bd107f1b-agent-certs\") pod \"konnectivity-agent-6ff59\" (UID: \"2deddafa-cbdf-407b-98c2-7c73bd107f1b\") " pod="kube-system/konnectivity-agent-6ff59" Jul 15 23:36:28.140629 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:28.139758 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f9fcf8a8-82dc-4e0b-b211-4611f9436609-tmp-dir\") pod \"node-resolver-z9q4h\" (UID: \"f9fcf8a8-82dc-4e0b-b211-4611f9436609\") " pod="openshift-dns/node-resolver-z9q4h" Jul 15 23:36:28.140629 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:28.139765 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a87574b4-736b-4e7d-9af6-3b6f89ac1c31-system-cni-dir\") pod \"multus-additional-cni-plugins-tg24d\" (UID: \"a87574b4-736b-4e7d-9af6-3b6f89ac1c31\") " pod="openshift-multus/multus-additional-cni-plugins-tg24d" Jul 15 23:36:28.140629 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:28.139791 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a87574b4-736b-4e7d-9af6-3b6f89ac1c31-os-release\") pod \"multus-additional-cni-plugins-tg24d\" (UID: \"a87574b4-736b-4e7d-9af6-3b6f89ac1c31\") " pod="openshift-multus/multus-additional-cni-plugins-tg24d" Jul 15 23:36:28.140629 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:28.139801 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1f66ebce-c0a5-4840-b106-3cedd6448ba5-etc-sysctl-d\") pod \"tuned-tm65j\" (UID: \"1f66ebce-c0a5-4840-b106-3cedd6448ba5\") " pod="openshift-cluster-node-tuning-operator/tuned-tm65j" Jul 15 23:36:28.140629 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:28.139817 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhwjf\" (UniqueName: \"kubernetes.io/projected/a87574b4-736b-4e7d-9af6-3b6f89ac1c31-kube-api-access-qhwjf\") pod \"multus-additional-cni-plugins-tg24d\" (UID: \"a87574b4-736b-4e7d-9af6-3b6f89ac1c31\") " pod="openshift-multus/multus-additional-cni-plugins-tg24d" Jul 15 23:36:28.140629 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:28.139848 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/1f66ebce-c0a5-4840-b106-3cedd6448ba5-etc-tuned\") pod \"tuned-tm65j\" (UID: \"1f66ebce-c0a5-4840-b106-3cedd6448ba5\") " pod="openshift-cluster-node-tuning-operator/tuned-tm65j" Jul 15 23:36:28.140629 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:28.139857 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1f66ebce-c0a5-4840-b106-3cedd6448ba5-etc-kubernetes\") pod \"tuned-tm65j\" (UID: \"1f66ebce-c0a5-4840-b106-3cedd6448ba5\") " pod="openshift-cluster-node-tuning-operator/tuned-tm65j" Jul 15 23:36:28.140629 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:28.139943 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f9fcf8a8-82dc-4e0b-b211-4611f9436609-hosts-file\") pod \"node-resolver-z9q4h\" (UID: \"f9fcf8a8-82dc-4e0b-b211-4611f9436609\") " pod="openshift-dns/node-resolver-z9q4h" Jul 15 23:36:28.140629 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:28.139994 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1f66ebce-c0a5-4840-b106-3cedd6448ba5-run\") pod \"tuned-tm65j\" (UID: \"1f66ebce-c0a5-4840-b106-3cedd6448ba5\") " pod="openshift-cluster-node-tuning-operator/tuned-tm65j" Jul 15 23:36:28.140629 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:28.140038 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1f66ebce-c0a5-4840-b106-3cedd6448ba5-var-lib-kubelet\") pod \"tuned-tm65j\" (UID: \"1f66ebce-c0a5-4840-b106-3cedd6448ba5\") " pod="openshift-cluster-node-tuning-operator/tuned-tm65j" Jul 15 23:36:28.140629 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:28.140196 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1f66ebce-c0a5-4840-b106-3cedd6448ba5-etc-systemd\") pod \"tuned-tm65j\" (UID: \"1f66ebce-c0a5-4840-b106-3cedd6448ba5\") " pod="openshift-cluster-node-tuning-operator/tuned-tm65j" Jul 15 23:36:28.140629 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:28.140280 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a87574b4-736b-4e7d-9af6-3b6f89ac1c31-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-tg24d\" (UID: \"a87574b4-736b-4e7d-9af6-3b6f89ac1c31\") " pod="openshift-multus/multus-additional-cni-plugins-tg24d" Jul 15 23:36:28.140629 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:28.140331 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/10a402bd-5fc8-41c2-852a-5df876f74bdb-dbus\") pod \"global-pull-secret-syncer-4zw2v\" (UID: \"10a402bd-5fc8-41c2-852a-5df876f74bdb\") " pod="kube-system/global-pull-secret-syncer-4zw2v" Jul 15 23:36:28.140629 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:28.139817 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2deddafa-cbdf-407b-98c2-7c73bd107f1b-konnectivity-ca\") pod \"konnectivity-agent-6ff59\" (UID: \"2deddafa-cbdf-407b-98c2-7c73bd107f1b\") " pod="kube-system/konnectivity-agent-6ff59" Jul 15 23:36:28.140629 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:28.140357 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a87574b4-736b-4e7d-9af6-3b6f89ac1c31-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-tg24d\" (UID: \"a87574b4-736b-4e7d-9af6-3b6f89ac1c31\") " pod="openshift-multus/multus-additional-cni-plugins-tg24d" Jul 15 23:36:28.141539 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:28.140454 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1f66ebce-c0a5-4840-b106-3cedd6448ba5-lib-modules\") pod \"tuned-tm65j\" (UID: \"1f66ebce-c0a5-4840-b106-3cedd6448ba5\") " pod="openshift-cluster-node-tuning-operator/tuned-tm65j" Jul 15 23:36:28.141539 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:28.140490 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1f66ebce-c0a5-4840-b106-3cedd6448ba5-etc-sysctl-conf\") pod \"tuned-tm65j\" (UID: \"1f66ebce-c0a5-4840-b106-3cedd6448ba5\") " pod="openshift-cluster-node-tuning-operator/tuned-tm65j" Jul 15 23:36:28.141539 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:28.140501 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a87574b4-736b-4e7d-9af6-3b6f89ac1c31-cnibin\") pod \"multus-additional-cni-plugins-tg24d\" (UID: \"a87574b4-736b-4e7d-9af6-3b6f89ac1c31\") " pod="openshift-multus/multus-additional-cni-plugins-tg24d" Jul 15 23:36:28.141539 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:28.140504 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a87574b4-736b-4e7d-9af6-3b6f89ac1c31-os-release\") pod \"multus-additional-cni-plugins-tg24d\" (UID: \"a87574b4-736b-4e7d-9af6-3b6f89ac1c31\") " pod="openshift-multus/multus-additional-cni-plugins-tg24d" Jul 15 23:36:28.141539 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:28.140533 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a87574b4-736b-4e7d-9af6-3b6f89ac1c31-system-cni-dir\") pod \"multus-additional-cni-plugins-tg24d\" (UID: \"a87574b4-736b-4e7d-9af6-3b6f89ac1c31\") " pod="openshift-multus/multus-additional-cni-plugins-tg24d" Jul 15 23:36:28.141539 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:28.140668 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1f66ebce-c0a5-4840-b106-3cedd6448ba5-etc-sysconfig\") pod \"tuned-tm65j\" (UID: \"1f66ebce-c0a5-4840-b106-3cedd6448ba5\") " pod="openshift-cluster-node-tuning-operator/tuned-tm65j" Jul 15 23:36:28.141539 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:28.140674 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a87574b4-736b-4e7d-9af6-3b6f89ac1c31-tuning-conf-dir\") pod \"multus-additional-cni-plugins-tg24d\" (UID: \"a87574b4-736b-4e7d-9af6-3b6f89ac1c31\") " pod="openshift-multus/multus-additional-cni-plugins-tg24d" Jul 15 23:36:28.141539 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:28.140905 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1f66ebce-c0a5-4840-b106-3cedd6448ba5-etc-modprobe-d\") pod \"tuned-tm65j\" (UID: \"1f66ebce-c0a5-4840-b106-3cedd6448ba5\") " pod="openshift-cluster-node-tuning-operator/tuned-tm65j" Jul 15 23:36:28.144312 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:28.144273 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2deddafa-cbdf-407b-98c2-7c73bd107f1b-agent-certs\") pod \"konnectivity-agent-6ff59\" (UID: \"2deddafa-cbdf-407b-98c2-7c73bd107f1b\") " pod="kube-system/konnectivity-agent-6ff59" Jul 15 23:36:28.144596 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:28.144525 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/1f66ebce-c0a5-4840-b106-3cedd6448ba5-etc-tuned\") pod \"tuned-tm65j\" (UID: \"1f66ebce-c0a5-4840-b106-3cedd6448ba5\") " pod="openshift-cluster-node-tuning-operator/tuned-tm65j" Jul 15 23:36:28.144596 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:28.144561 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1f66ebce-c0a5-4840-b106-3cedd6448ba5-tmp\") pod \"tuned-tm65j\" (UID: \"1f66ebce-c0a5-4840-b106-3cedd6448ba5\") " pod="openshift-cluster-node-tuning-operator/tuned-tm65j" Jul 15 23:36:28.148271 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:28.148247 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-txjg5\" (UniqueName: \"kubernetes.io/projected/1f66ebce-c0a5-4840-b106-3cedd6448ba5-kube-api-access-txjg5\") pod \"tuned-tm65j\" (UID: \"1f66ebce-c0a5-4840-b106-3cedd6448ba5\") " pod="openshift-cluster-node-tuning-operator/tuned-tm65j" Jul 15 23:36:28.149457 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:28.149431 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhwjf\" (UniqueName: \"kubernetes.io/projected/a87574b4-736b-4e7d-9af6-3b6f89ac1c31-kube-api-access-qhwjf\") pod \"multus-additional-cni-plugins-tg24d\" (UID: \"a87574b4-736b-4e7d-9af6-3b6f89ac1c31\") " pod="openshift-multus/multus-additional-cni-plugins-tg24d" Jul 15 23:36:28.149885 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:28.149858 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-686vw\" (UniqueName: \"kubernetes.io/projected/f9fcf8a8-82dc-4e0b-b211-4611f9436609-kube-api-access-686vw\") pod \"node-resolver-z9q4h\" (UID: \"f9fcf8a8-82dc-4e0b-b211-4611f9436609\") " pod="openshift-dns/node-resolver-z9q4h" Jul 15 23:36:28.151915 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:28.151893 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-v2rfk" Jul 15 23:36:28.159021 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:28.158952 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-khwkr" Jul 15 23:36:28.168253 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:28.168010 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod387d6414_0c71_495c_acc0_9d64115d31fe.slice/crio-dd8971b217b9b0462a971a99e2bd0468f5e085f08c6c70916f76e6ebebaf4c34 WatchSource:0}: Error finding container dd8971b217b9b0462a971a99e2bd0468f5e085f08c6c70916f76e6ebebaf4c34: Status 404 returned error can't find the container with id dd8971b217b9b0462a971a99e2bd0468f5e085f08c6c70916f76e6ebebaf4c34 Jul 15 23:36:28.168416 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:28.168393 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-d8f6k" Jul 15 23:36:28.171752 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:28.171721 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod498d1e24_acb4_4d75_a5a7_410f00ecc9f9.slice/crio-be2e51bdf6f485ff35ecbd04f81ac8983dcd08b99a0cbb63840c52d97b92bc9f WatchSource:0}: Error finding container be2e51bdf6f485ff35ecbd04f81ac8983dcd08b99a0cbb63840c52d97b92bc9f: Status 404 returned error can't find the container with id be2e51bdf6f485ff35ecbd04f81ac8983dcd08b99a0cbb63840c52d97b92bc9f Jul 15 23:36:28.173872 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:28.173847 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-v62rk" Jul 15 23:36:28.179624 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:28.179581 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6pwvf" Jul 15 23:36:28.181167 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:28.181141 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f64cab3_104a_44f6_b125_ba458f54f058.slice/crio-d04c8fcabf48255b2c723ef14ae5e16a4bc426630067c1c8948c1c4281438fa0 WatchSource:0}: Error finding container d04c8fcabf48255b2c723ef14ae5e16a4bc426630067c1c8948c1c4281438fa0: Status 404 returned error can't find the container with id d04c8fcabf48255b2c723ef14ae5e16a4bc426630067c1c8948c1c4281438fa0 Jul 15 23:36:28.185738 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:28.185674 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-6ff59" Jul 15 23:36:28.190307 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:28.190290 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-tm65j" Jul 15 23:36:28.196058 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:28.196027 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-z9q4h" Jul 15 23:36:28.197157 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:28.197086 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabb1e456_40be_492d_9d6b_d10413926119.slice/crio-e8c09378a9bd2fbcec6891940c296ef0b46c5a0eac0e4264c5d81a4910945027 WatchSource:0}: Error finding container e8c09378a9bd2fbcec6891940c296ef0b46c5a0eac0e4264c5d81a4910945027: Status 404 returned error can't find the container with id e8c09378a9bd2fbcec6891940c296ef0b46c5a0eac0e4264c5d81a4910945027 Jul 15 23:36:28.200099 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:28.200074 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-tg24d" Jul 15 23:36:28.205082 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:28.205054 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2deddafa_cbdf_407b_98c2_7c73bd107f1b.slice/crio-b45eda49ddb5497ff34984e57fafcebe097a6802f74f77627a5746cde268ba6c WatchSource:0}: Error finding container b45eda49ddb5497ff34984e57fafcebe097a6802f74f77627a5746cde268ba6c: Status 404 returned error can't find the container with id b45eda49ddb5497ff34984e57fafcebe097a6802f74f77627a5746cde268ba6c Jul 15 23:36:28.211337 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:28.211302 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f66ebce_c0a5_4840_b106_3cedd6448ba5.slice/crio-684182e1e306b15bdf6798a2ebb6cfd6f9b908235fa5f0170492a1ea5ecb1dc0 WatchSource:0}: Error finding container 684182e1e306b15bdf6798a2ebb6cfd6f9b908235fa5f0170492a1ea5ecb1dc0: Status 404 returned error can't find the container with id 684182e1e306b15bdf6798a2ebb6cfd6f9b908235fa5f0170492a1ea5ecb1dc0 Jul 15 23:36:28.215014 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:28.214990 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9fcf8a8_82dc_4e0b_b211_4611f9436609.slice/crio-d7402dfacb4ea7184bb8c1b6c92cf18f6ef9df40bf7307c46ae7f3167fb1fc9c WatchSource:0}: Error finding container d7402dfacb4ea7184bb8c1b6c92cf18f6ef9df40bf7307c46ae7f3167fb1fc9c: Status 404 returned error can't find the container with id d7402dfacb4ea7184bb8c1b6c92cf18f6ef9df40bf7307c46ae7f3167fb1fc9c Jul 15 23:36:28.218224 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:36:28.218196 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda87574b4_736b_4e7d_9af6_3b6f89ac1c31.slice/crio-b8c25a5f1772a5ce2bf5368586ad225f5f4b3841092543a50500693a69603aaf WatchSource:0}: Error finding container b8c25a5f1772a5ce2bf5368586ad225f5f4b3841092543a50500693a69603aaf: Status 404 returned error can't find the container with id b8c25a5f1772a5ce2bf5368586ad225f5f4b3841092543a50500693a69603aaf Jul 15 23:36:28.543342 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:28.543311 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d1c182db-bcdc-40a1-bb82-4f8b64173e43-metrics-certs\") pod \"network-metrics-daemon-7n4t9\" (UID: \"d1c182db-bcdc-40a1-bb82-4f8b64173e43\") " pod="openshift-multus/network-metrics-daemon-7n4t9" Jul 15 23:36:28.543528 ip-10-0-128-197 kubenswrapper[2563]: E0715 23:36:28.543442 2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 23:36:28.543528 ip-10-0-128-197 kubenswrapper[2563]: E0715 23:36:28.543521 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d1c182db-bcdc-40a1-bb82-4f8b64173e43-metrics-certs podName:d1c182db-bcdc-40a1-bb82-4f8b64173e43 nodeName:}" failed. No retries permitted until 2026-07-15 23:36:29.543502081 +0000 UTC m=+4.916960042 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d1c182db-bcdc-40a1-bb82-4f8b64173e43-metrics-certs") pod "network-metrics-daemon-7n4t9" (UID: "d1c182db-bcdc-40a1-bb82-4f8b64173e43") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 23:36:28.644127 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:28.644094 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/10a402bd-5fc8-41c2-852a-5df876f74bdb-original-pull-secret\") pod \"global-pull-secret-syncer-4zw2v\" (UID: \"10a402bd-5fc8-41c2-852a-5df876f74bdb\") " pod="kube-system/global-pull-secret-syncer-4zw2v" Jul 15 23:36:28.644288 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:28.644149 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v47j5\" (UniqueName: \"kubernetes.io/projected/b0b0bc56-c9c9-4a96-a97b-15f00f3b2099-kube-api-access-v47j5\") pod \"network-check-target-mp4pr\" (UID: \"b0b0bc56-c9c9-4a96-a97b-15f00f3b2099\") " pod="openshift-network-diagnostics/network-check-target-mp4pr" Jul 15 23:36:28.644288 ip-10-0-128-197 kubenswrapper[2563]: E0715 23:36:28.644260 2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 15 23:36:28.644451 ip-10-0-128-197 kubenswrapper[2563]: E0715 23:36:28.644314 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 15 23:36:28.644451 ip-10-0-128-197 kubenswrapper[2563]: E0715 23:36:28.644324 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/10a402bd-5fc8-41c2-852a-5df876f74bdb-original-pull-secret podName:10a402bd-5fc8-41c2-852a-5df876f74bdb nodeName:}" failed. No retries permitted until 2026-07-15 23:36:29.644306355 +0000 UTC m=+5.017764317 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/10a402bd-5fc8-41c2-852a-5df876f74bdb-original-pull-secret") pod "global-pull-secret-syncer-4zw2v" (UID: "10a402bd-5fc8-41c2-852a-5df876f74bdb") : object "kube-system"/"original-pull-secret" not registered Jul 15 23:36:28.644451 ip-10-0-128-197 kubenswrapper[2563]: E0715 23:36:28.644332 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 15 23:36:28.644451 ip-10-0-128-197 kubenswrapper[2563]: E0715 23:36:28.644345 2563 projected.go:196] Error preparing data for projected volume kube-api-access-v47j5 for pod openshift-network-diagnostics/network-check-target-mp4pr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 23:36:28.644451 ip-10-0-128-197 kubenswrapper[2563]: E0715 23:36:28.644380 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b0b0bc56-c9c9-4a96-a97b-15f00f3b2099-kube-api-access-v47j5 podName:b0b0bc56-c9c9-4a96-a97b-15f00f3b2099 nodeName:}" failed. No retries permitted until 2026-07-15 23:36:29.644369616 +0000 UTC m=+5.017827557 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-v47j5" (UniqueName: "kubernetes.io/projected/b0b0bc56-c9c9-4a96-a97b-15f00f3b2099-kube-api-access-v47j5") pod "network-check-target-mp4pr" (UID: "b0b0bc56-c9c9-4a96-a97b-15f00f3b2099") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 23:36:28.831888 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:28.831780 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tg24d" event={"ID":"a87574b4-736b-4e7d-9af6-3b6f89ac1c31","Type":"ContainerStarted","Data":"b8c25a5f1772a5ce2bf5368586ad225f5f4b3841092543a50500693a69603aaf"} Jul 15 23:36:28.833057 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:28.833029 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-z9q4h" event={"ID":"f9fcf8a8-82dc-4e0b-b211-4611f9436609","Type":"ContainerStarted","Data":"d7402dfacb4ea7184bb8c1b6c92cf18f6ef9df40bf7307c46ae7f3167fb1fc9c"} Jul 15 23:36:28.834099 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:28.834064 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-tm65j" event={"ID":"1f66ebce-c0a5-4840-b106-3cedd6448ba5","Type":"ContainerStarted","Data":"684182e1e306b15bdf6798a2ebb6cfd6f9b908235fa5f0170492a1ea5ecb1dc0"} Jul 15 23:36:28.835009 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:28.834975 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-6ff59" event={"ID":"2deddafa-cbdf-407b-98c2-7c73bd107f1b","Type":"ContainerStarted","Data":"b45eda49ddb5497ff34984e57fafcebe097a6802f74f77627a5746cde268ba6c"} Jul 15 23:36:28.836058 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:28.836037 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6pwvf" event={"ID":"abb1e456-40be-492d-9d6b-d10413926119","Type":"ContainerStarted","Data":"e8c09378a9bd2fbcec6891940c296ef0b46c5a0eac0e4264c5d81a4910945027"} Jul 15 23:36:28.837220 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:28.837199 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-d8f6k" event={"ID":"2f64cab3-104a-44f6-b125-ba458f54f058","Type":"ContainerStarted","Data":"d04c8fcabf48255b2c723ef14ae5e16a4bc426630067c1c8948c1c4281438fa0"} Jul 15 23:36:28.838257 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:28.838239 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-v62rk" event={"ID":"12202d45-85c8-41de-b95e-f5cf403572ae","Type":"ContainerStarted","Data":"3725f668100045a3ee4c4e4e5ef8938921b8de535f90ba2e8357ba43ac0a7283"} Jul 15 23:36:28.839275 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:28.839238 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-khwkr" event={"ID":"498d1e24-acb4-4d75-a5a7-410f00ecc9f9","Type":"ContainerStarted","Data":"be2e51bdf6f485ff35ecbd04f81ac8983dcd08b99a0cbb63840c52d97b92bc9f"} Jul 15 23:36:28.840349 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:28.840328 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-v2rfk" event={"ID":"387d6414-0c71-495c-acc0-9d64115d31fe","Type":"ContainerStarted","Data":"dd8971b217b9b0462a971a99e2bd0468f5e085f08c6c70916f76e6ebebaf4c34"} Jul 15 23:36:28.892657 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:28.892629 2563 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-14 23:31:27 +0000 UTC" deadline="2027-12-10 05:23:46.24919598 +0000 UTC" Jul 15 23:36:28.892753 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:28.892654 2563 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12293h47m17.356545264s" Jul 15 23:36:29.163260 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:29.163169 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mp4pr" Jul 15 23:36:29.163647 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:29.163342 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4zw2v" Jul 15 23:36:29.163647 ip-10-0-128-197 kubenswrapper[2563]: E0715 23:36:29.163348 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mp4pr" podUID="b0b0bc56-c9c9-4a96-a97b-15f00f3b2099" Jul 15 23:36:29.163906 ip-10-0-128-197 kubenswrapper[2563]: E0715 23:36:29.163877 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4zw2v" podUID="10a402bd-5fc8-41c2-852a-5df876f74bdb" Jul 15 23:36:29.552749 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:29.552549 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d1c182db-bcdc-40a1-bb82-4f8b64173e43-metrics-certs\") pod \"network-metrics-daemon-7n4t9\" (UID: \"d1c182db-bcdc-40a1-bb82-4f8b64173e43\") " pod="openshift-multus/network-metrics-daemon-7n4t9" Jul 15 23:36:29.552749 ip-10-0-128-197 kubenswrapper[2563]: E0715 23:36:29.552717 2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 23:36:29.552951 ip-10-0-128-197 kubenswrapper[2563]: E0715 23:36:29.552792 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d1c182db-bcdc-40a1-bb82-4f8b64173e43-metrics-certs podName:d1c182db-bcdc-40a1-bb82-4f8b64173e43 nodeName:}" failed. No retries permitted until 2026-07-15 23:36:31.55277069 +0000 UTC m=+6.926228633 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d1c182db-bcdc-40a1-bb82-4f8b64173e43-metrics-certs") pod "network-metrics-daemon-7n4t9" (UID: "d1c182db-bcdc-40a1-bb82-4f8b64173e43") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 23:36:29.654600 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:29.653808 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/10a402bd-5fc8-41c2-852a-5df876f74bdb-original-pull-secret\") pod \"global-pull-secret-syncer-4zw2v\" (UID: \"10a402bd-5fc8-41c2-852a-5df876f74bdb\") " pod="kube-system/global-pull-secret-syncer-4zw2v" Jul 15 23:36:29.654600 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:29.653875 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v47j5\" (UniqueName: \"kubernetes.io/projected/b0b0bc56-c9c9-4a96-a97b-15f00f3b2099-kube-api-access-v47j5\") pod \"network-check-target-mp4pr\" (UID: \"b0b0bc56-c9c9-4a96-a97b-15f00f3b2099\") " pod="openshift-network-diagnostics/network-check-target-mp4pr" Jul 15 23:36:29.654600 ip-10-0-128-197 kubenswrapper[2563]: E0715 23:36:29.654036 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 15 23:36:29.654600 ip-10-0-128-197 kubenswrapper[2563]: E0715 23:36:29.654056 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 15 23:36:29.654600 ip-10-0-128-197 kubenswrapper[2563]: E0715 23:36:29.654069 2563 projected.go:196] Error preparing data for projected volume kube-api-access-v47j5 for pod openshift-network-diagnostics/network-check-target-mp4pr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 23:36:29.654600 ip-10-0-128-197 kubenswrapper[2563]: E0715 23:36:29.654125 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b0b0bc56-c9c9-4a96-a97b-15f00f3b2099-kube-api-access-v47j5 podName:b0b0bc56-c9c9-4a96-a97b-15f00f3b2099 nodeName:}" failed. No retries permitted until 2026-07-15 23:36:31.654105816 +0000 UTC m=+7.027563781 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-v47j5" (UniqueName: "kubernetes.io/projected/b0b0bc56-c9c9-4a96-a97b-15f00f3b2099-kube-api-access-v47j5") pod "network-check-target-mp4pr" (UID: "b0b0bc56-c9c9-4a96-a97b-15f00f3b2099") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 23:36:29.654600 ip-10-0-128-197 kubenswrapper[2563]: E0715 23:36:29.654515 2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 15 23:36:29.654600 ip-10-0-128-197 kubenswrapper[2563]: E0715 23:36:29.654562 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/10a402bd-5fc8-41c2-852a-5df876f74bdb-original-pull-secret podName:10a402bd-5fc8-41c2-852a-5df876f74bdb nodeName:}" failed. No retries permitted until 2026-07-15 23:36:31.654546722 +0000 UTC m=+7.028004662 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/10a402bd-5fc8-41c2-852a-5df876f74bdb-original-pull-secret") pod "global-pull-secret-syncer-4zw2v" (UID: "10a402bd-5fc8-41c2-852a-5df876f74bdb") : object "kube-system"/"original-pull-secret" not registered Jul 15 23:36:29.893647 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:29.893540 2563 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-14 23:31:27 +0000 UTC" deadline="2028-03-06 20:48:39.169535681 +0000 UTC" Jul 15 23:36:29.893647 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:29.893598 2563 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14397h12m9.275942099s" Jul 15 23:36:30.166059 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:30.163233 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7n4t9" Jul 15 23:36:30.166059 ip-10-0-128-197 kubenswrapper[2563]: E0715 23:36:30.163380 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7n4t9" podUID="d1c182db-bcdc-40a1-bb82-4f8b64173e43" Jul 15 23:36:30.864438 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:30.863729 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-197.ec2.internal" event={"ID":"1c95fb996cc0f9a497fcf20794fa4db8","Type":"ContainerStarted","Data":"1e6bab9f75c65d0334863121211a5813b800fabf1b409eeeca7a4c15cce21867"} Jul 15 23:36:30.881381 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:30.881304 2563 generic.go:356] "Generic (PLEG): container finished" podID="2e2be974bacf2b9813c8842cd4ec626c" containerID="7bf5fda9e788a5b10693c9071fc42e6e31b90417984028e05361e29182e77751" exitCode=0 Jul 15 23:36:30.881381 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:30.881359 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-197.ec2.internal" event={"ID":"2e2be974bacf2b9813c8842cd4ec626c","Type":"ContainerDied","Data":"7bf5fda9e788a5b10693c9071fc42e6e31b90417984028e05361e29182e77751"} Jul 15 23:36:30.898289 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:30.897536 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-197.ec2.internal" podStartSLOduration=3.897517912 podStartE2EDuration="3.897517912s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 23:36:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 23:36:30.881093267 +0000 UTC m=+6.254551231" watchObservedRunningTime="2026-07-15 23:36:30.897517912 +0000 UTC m=+6.270975877" Jul 15 23:36:31.164041 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:31.163608 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mp4pr" Jul 15 23:36:31.164041 ip-10-0-128-197 kubenswrapper[2563]: E0715 23:36:31.163732 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mp4pr" podUID="b0b0bc56-c9c9-4a96-a97b-15f00f3b2099" Jul 15 23:36:31.164300 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:31.164076 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4zw2v" Jul 15 23:36:31.164300 ip-10-0-128-197 kubenswrapper[2563]: E0715 23:36:31.164174 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4zw2v" podUID="10a402bd-5fc8-41c2-852a-5df876f74bdb" Jul 15 23:36:31.569020 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:31.568990 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d1c182db-bcdc-40a1-bb82-4f8b64173e43-metrics-certs\") pod \"network-metrics-daemon-7n4t9\" (UID: \"d1c182db-bcdc-40a1-bb82-4f8b64173e43\") " pod="openshift-multus/network-metrics-daemon-7n4t9" Jul 15 23:36:31.569475 ip-10-0-128-197 kubenswrapper[2563]: E0715 23:36:31.569143 2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 23:36:31.569475 ip-10-0-128-197 kubenswrapper[2563]: E0715 23:36:31.569215 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d1c182db-bcdc-40a1-bb82-4f8b64173e43-metrics-certs podName:d1c182db-bcdc-40a1-bb82-4f8b64173e43 nodeName:}" failed. No retries permitted until 2026-07-15 23:36:35.569193557 +0000 UTC m=+10.942651500 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d1c182db-bcdc-40a1-bb82-4f8b64173e43-metrics-certs") pod "network-metrics-daemon-7n4t9" (UID: "d1c182db-bcdc-40a1-bb82-4f8b64173e43") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 23:36:31.669703 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:31.669645 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/10a402bd-5fc8-41c2-852a-5df876f74bdb-original-pull-secret\") pod \"global-pull-secret-syncer-4zw2v\" (UID: \"10a402bd-5fc8-41c2-852a-5df876f74bdb\") " pod="kube-system/global-pull-secret-syncer-4zw2v" Jul 15 23:36:31.669902 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:31.669741 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v47j5\" (UniqueName: \"kubernetes.io/projected/b0b0bc56-c9c9-4a96-a97b-15f00f3b2099-kube-api-access-v47j5\") pod \"network-check-target-mp4pr\" (UID: \"b0b0bc56-c9c9-4a96-a97b-15f00f3b2099\") " pod="openshift-network-diagnostics/network-check-target-mp4pr" Jul 15 23:36:31.670021 ip-10-0-128-197 kubenswrapper[2563]: E0715 23:36:31.669956 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 15 23:36:31.670021 ip-10-0-128-197 kubenswrapper[2563]: E0715 23:36:31.669989 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 15 23:36:31.670021 ip-10-0-128-197 kubenswrapper[2563]: E0715 23:36:31.670003 2563 projected.go:196] Error preparing data for projected volume kube-api-access-v47j5 for pod openshift-network-diagnostics/network-check-target-mp4pr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 23:36:31.670179 ip-10-0-128-197 kubenswrapper[2563]: E0715 23:36:31.670061 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b0b0bc56-c9c9-4a96-a97b-15f00f3b2099-kube-api-access-v47j5 podName:b0b0bc56-c9c9-4a96-a97b-15f00f3b2099 nodeName:}" failed. No retries permitted until 2026-07-15 23:36:35.670040967 +0000 UTC m=+11.043498911 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-v47j5" (UniqueName: "kubernetes.io/projected/b0b0bc56-c9c9-4a96-a97b-15f00f3b2099-kube-api-access-v47j5") pod "network-check-target-mp4pr" (UID: "b0b0bc56-c9c9-4a96-a97b-15f00f3b2099") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 23:36:31.670376 ip-10-0-128-197 kubenswrapper[2563]: E0715 23:36:31.670319 2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 15 23:36:31.670376 ip-10-0-128-197 kubenswrapper[2563]: E0715 23:36:31.670371 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/10a402bd-5fc8-41c2-852a-5df876f74bdb-original-pull-secret podName:10a402bd-5fc8-41c2-852a-5df876f74bdb nodeName:}" failed. No retries permitted until 2026-07-15 23:36:35.670356621 +0000 UTC m=+11.043814573 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/10a402bd-5fc8-41c2-852a-5df876f74bdb-original-pull-secret") pod "global-pull-secret-syncer-4zw2v" (UID: "10a402bd-5fc8-41c2-852a-5df876f74bdb") : object "kube-system"/"original-pull-secret" not registered Jul 15 23:36:31.891754 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:31.891657 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-197.ec2.internal" event={"ID":"2e2be974bacf2b9813c8842cd4ec626c","Type":"ContainerStarted","Data":"121472a7c586dde64ed66e4bce14f80b298077dc6cf245c43da37373ac89a0b4"} Jul 15 23:36:32.163863 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:32.163783 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7n4t9" Jul 15 23:36:32.164053 ip-10-0-128-197 kubenswrapper[2563]: E0715 23:36:32.163926 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7n4t9" podUID="d1c182db-bcdc-40a1-bb82-4f8b64173e43" Jul 15 23:36:33.164582 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:33.164079 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4zw2v" Jul 15 23:36:33.164582 ip-10-0-128-197 kubenswrapper[2563]: E0715 23:36:33.164206 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4zw2v" podUID="10a402bd-5fc8-41c2-852a-5df876f74bdb" Jul 15 23:36:33.165325 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:33.165160 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mp4pr" Jul 15 23:36:33.165325 ip-10-0-128-197 kubenswrapper[2563]: E0715 23:36:33.165264 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mp4pr" podUID="b0b0bc56-c9c9-4a96-a97b-15f00f3b2099" Jul 15 23:36:34.163425 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:34.163393 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7n4t9" Jul 15 23:36:34.163615 ip-10-0-128-197 kubenswrapper[2563]: E0715 23:36:34.163555 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7n4t9" podUID="d1c182db-bcdc-40a1-bb82-4f8b64173e43" Jul 15 23:36:35.164361 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:35.163853 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4zw2v" Jul 15 23:36:35.164361 ip-10-0-128-197 kubenswrapper[2563]: E0715 23:36:35.163982 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4zw2v" podUID="10a402bd-5fc8-41c2-852a-5df876f74bdb" Jul 15 23:36:35.164361 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:35.164327 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mp4pr" Jul 15 23:36:35.164931 ip-10-0-128-197 kubenswrapper[2563]: E0715 23:36:35.164403 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mp4pr" podUID="b0b0bc56-c9c9-4a96-a97b-15f00f3b2099" Jul 15 23:36:35.603051 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:35.602948 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d1c182db-bcdc-40a1-bb82-4f8b64173e43-metrics-certs\") pod \"network-metrics-daemon-7n4t9\" (UID: \"d1c182db-bcdc-40a1-bb82-4f8b64173e43\") " pod="openshift-multus/network-metrics-daemon-7n4t9" Jul 15 23:36:35.603240 ip-10-0-128-197 kubenswrapper[2563]: E0715 23:36:35.603104 2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 23:36:35.603240 ip-10-0-128-197 kubenswrapper[2563]: E0715 23:36:35.603177 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d1c182db-bcdc-40a1-bb82-4f8b64173e43-metrics-certs podName:d1c182db-bcdc-40a1-bb82-4f8b64173e43 nodeName:}" failed. No retries permitted until 2026-07-15 23:36:43.603148226 +0000 UTC m=+18.976606184 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d1c182db-bcdc-40a1-bb82-4f8b64173e43-metrics-certs") pod "network-metrics-daemon-7n4t9" (UID: "d1c182db-bcdc-40a1-bb82-4f8b64173e43") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 23:36:35.704458 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:35.704121 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/10a402bd-5fc8-41c2-852a-5df876f74bdb-original-pull-secret\") pod \"global-pull-secret-syncer-4zw2v\" (UID: \"10a402bd-5fc8-41c2-852a-5df876f74bdb\") " pod="kube-system/global-pull-secret-syncer-4zw2v" Jul 15 23:36:35.704458 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:35.704188 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v47j5\" (UniqueName: \"kubernetes.io/projected/b0b0bc56-c9c9-4a96-a97b-15f00f3b2099-kube-api-access-v47j5\") pod \"network-check-target-mp4pr\" (UID: \"b0b0bc56-c9c9-4a96-a97b-15f00f3b2099\") " pod="openshift-network-diagnostics/network-check-target-mp4pr" Jul 15 23:36:35.704458 ip-10-0-128-197 kubenswrapper[2563]: E0715 23:36:35.704281 2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 15 23:36:35.704458 ip-10-0-128-197 kubenswrapper[2563]: E0715 23:36:35.704349 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 15 23:36:35.704458 ip-10-0-128-197 kubenswrapper[2563]: E0715 23:36:35.704360 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/10a402bd-5fc8-41c2-852a-5df876f74bdb-original-pull-secret podName:10a402bd-5fc8-41c2-852a-5df876f74bdb nodeName:}" failed. No retries permitted until 2026-07-15 23:36:43.704339263 +0000 UTC m=+19.077797207 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/10a402bd-5fc8-41c2-852a-5df876f74bdb-original-pull-secret") pod "global-pull-secret-syncer-4zw2v" (UID: "10a402bd-5fc8-41c2-852a-5df876f74bdb") : object "kube-system"/"original-pull-secret" not registered Jul 15 23:36:35.704458 ip-10-0-128-197 kubenswrapper[2563]: E0715 23:36:35.704368 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 15 23:36:35.704458 ip-10-0-128-197 kubenswrapper[2563]: E0715 23:36:35.704382 2563 projected.go:196] Error preparing data for projected volume kube-api-access-v47j5 for pod openshift-network-diagnostics/network-check-target-mp4pr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 23:36:35.704458 ip-10-0-128-197 kubenswrapper[2563]: E0715 23:36:35.704434 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b0b0bc56-c9c9-4a96-a97b-15f00f3b2099-kube-api-access-v47j5 podName:b0b0bc56-c9c9-4a96-a97b-15f00f3b2099 nodeName:}" failed. No retries permitted until 2026-07-15 23:36:43.704418879 +0000 UTC m=+19.077876823 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-v47j5" (UniqueName: "kubernetes.io/projected/b0b0bc56-c9c9-4a96-a97b-15f00f3b2099-kube-api-access-v47j5") pod "network-check-target-mp4pr" (UID: "b0b0bc56-c9c9-4a96-a97b-15f00f3b2099") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 23:36:36.164325 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:36.163694 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7n4t9" Jul 15 23:36:36.164325 ip-10-0-128-197 kubenswrapper[2563]: E0715 23:36:36.163851 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7n4t9" podUID="d1c182db-bcdc-40a1-bb82-4f8b64173e43" Jul 15 23:36:37.163496 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:37.163457 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4zw2v" Jul 15 23:36:37.163680 ip-10-0-128-197 kubenswrapper[2563]: E0715 23:36:37.163612 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4zw2v" podUID="10a402bd-5fc8-41c2-852a-5df876f74bdb" Jul 15 23:36:37.163680 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:37.163457 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mp4pr" Jul 15 23:36:37.163795 ip-10-0-128-197 kubenswrapper[2563]: E0715 23:36:37.163716 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mp4pr" podUID="b0b0bc56-c9c9-4a96-a97b-15f00f3b2099" Jul 15 23:36:38.163007 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:38.162959 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7n4t9" Jul 15 23:36:38.163476 ip-10-0-128-197 kubenswrapper[2563]: E0715 23:36:38.163116 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7n4t9" podUID="d1c182db-bcdc-40a1-bb82-4f8b64173e43" Jul 15 23:36:39.162949 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:39.162912 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mp4pr" Jul 15 23:36:39.163127 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:39.162919 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4zw2v" Jul 15 23:36:39.163127 ip-10-0-128-197 kubenswrapper[2563]: E0715 23:36:39.163062 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mp4pr" podUID="b0b0bc56-c9c9-4a96-a97b-15f00f3b2099" Jul 15 23:36:39.163444 ip-10-0-128-197 kubenswrapper[2563]: E0715 23:36:39.163124 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4zw2v" podUID="10a402bd-5fc8-41c2-852a-5df876f74bdb" Jul 15 23:36:40.163172 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:40.163144 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7n4t9" Jul 15 23:36:40.163570 ip-10-0-128-197 kubenswrapper[2563]: E0715 23:36:40.163282 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7n4t9" podUID="d1c182db-bcdc-40a1-bb82-4f8b64173e43" Jul 15 23:36:41.163659 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:41.163624 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4zw2v" Jul 15 23:36:41.164126 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:41.163624 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mp4pr" Jul 15 23:36:41.164126 ip-10-0-128-197 kubenswrapper[2563]: E0715 23:36:41.163743 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4zw2v" podUID="10a402bd-5fc8-41c2-852a-5df876f74bdb" Jul 15 23:36:41.164126 ip-10-0-128-197 kubenswrapper[2563]: E0715 23:36:41.163853 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mp4pr" podUID="b0b0bc56-c9c9-4a96-a97b-15f00f3b2099" Jul 15 23:36:42.163608 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:42.163574 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7n4t9" Jul 15 23:36:42.163778 ip-10-0-128-197 kubenswrapper[2563]: E0715 23:36:42.163692 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7n4t9" podUID="d1c182db-bcdc-40a1-bb82-4f8b64173e43" Jul 15 23:36:43.163454 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:43.163415 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4zw2v" Jul 15 23:36:43.163661 ip-10-0-128-197 kubenswrapper[2563]: E0715 23:36:43.163545 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4zw2v" podUID="10a402bd-5fc8-41c2-852a-5df876f74bdb" Jul 15 23:36:43.163661 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:43.163596 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mp4pr" Jul 15 23:36:43.163767 ip-10-0-128-197 kubenswrapper[2563]: E0715 23:36:43.163677 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mp4pr" podUID="b0b0bc56-c9c9-4a96-a97b-15f00f3b2099" Jul 15 23:36:43.665418 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:43.665387 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d1c182db-bcdc-40a1-bb82-4f8b64173e43-metrics-certs\") pod \"network-metrics-daemon-7n4t9\" (UID: \"d1c182db-bcdc-40a1-bb82-4f8b64173e43\") " pod="openshift-multus/network-metrics-daemon-7n4t9" Jul 15 23:36:43.665772 ip-10-0-128-197 kubenswrapper[2563]: E0715 23:36:43.665508 2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 23:36:43.665772 ip-10-0-128-197 kubenswrapper[2563]: E0715 23:36:43.665569 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d1c182db-bcdc-40a1-bb82-4f8b64173e43-metrics-certs podName:d1c182db-bcdc-40a1-bb82-4f8b64173e43 nodeName:}" failed. No retries permitted until 2026-07-15 23:36:59.66554932 +0000 UTC m=+35.039007273 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d1c182db-bcdc-40a1-bb82-4f8b64173e43-metrics-certs") pod "network-metrics-daemon-7n4t9" (UID: "d1c182db-bcdc-40a1-bb82-4f8b64173e43") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 23:36:43.766009 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:43.765921 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/10a402bd-5fc8-41c2-852a-5df876f74bdb-original-pull-secret\") pod \"global-pull-secret-syncer-4zw2v\" (UID: \"10a402bd-5fc8-41c2-852a-5df876f74bdb\") " pod="kube-system/global-pull-secret-syncer-4zw2v" Jul 15 23:36:43.766009 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:43.765998 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v47j5\" (UniqueName: \"kubernetes.io/projected/b0b0bc56-c9c9-4a96-a97b-15f00f3b2099-kube-api-access-v47j5\") pod \"network-check-target-mp4pr\" (UID: \"b0b0bc56-c9c9-4a96-a97b-15f00f3b2099\") " pod="openshift-network-diagnostics/network-check-target-mp4pr" Jul 15 23:36:43.766258 ip-10-0-128-197 kubenswrapper[2563]: E0715 23:36:43.766089 2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 15 23:36:43.766258 ip-10-0-128-197 kubenswrapper[2563]: E0715 23:36:43.766141 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 15 23:36:43.766258 ip-10-0-128-197 kubenswrapper[2563]: E0715 23:36:43.766160 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 15 23:36:43.766258 ip-10-0-128-197 kubenswrapper[2563]: E0715 23:36:43.766166 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/10a402bd-5fc8-41c2-852a-5df876f74bdb-original-pull-secret podName:10a402bd-5fc8-41c2-852a-5df876f74bdb nodeName:}" failed. No retries permitted until 2026-07-15 23:36:59.766148838 +0000 UTC m=+35.139606799 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/10a402bd-5fc8-41c2-852a-5df876f74bdb-original-pull-secret") pod "global-pull-secret-syncer-4zw2v" (UID: "10a402bd-5fc8-41c2-852a-5df876f74bdb") : object "kube-system"/"original-pull-secret" not registered Jul 15 23:36:43.766258 ip-10-0-128-197 kubenswrapper[2563]: E0715 23:36:43.766172 2563 projected.go:196] Error preparing data for projected volume kube-api-access-v47j5 for pod openshift-network-diagnostics/network-check-target-mp4pr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 23:36:43.766258 ip-10-0-128-197 kubenswrapper[2563]: E0715 23:36:43.766220 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b0b0bc56-c9c9-4a96-a97b-15f00f3b2099-kube-api-access-v47j5 podName:b0b0bc56-c9c9-4a96-a97b-15f00f3b2099 nodeName:}" failed. No retries permitted until 2026-07-15 23:36:59.766205588 +0000 UTC m=+35.139663530 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-v47j5" (UniqueName: "kubernetes.io/projected/b0b0bc56-c9c9-4a96-a97b-15f00f3b2099-kube-api-access-v47j5") pod "network-check-target-mp4pr" (UID: "b0b0bc56-c9c9-4a96-a97b-15f00f3b2099") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 23:36:44.163681 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:44.163605 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7n4t9" Jul 15 23:36:44.163835 ip-10-0-128-197 kubenswrapper[2563]: E0715 23:36:44.163750 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7n4t9" podUID="d1c182db-bcdc-40a1-bb82-4f8b64173e43" Jul 15 23:36:45.164130 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:45.164091 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4zw2v" Jul 15 23:36:45.164602 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:45.164156 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mp4pr" Jul 15 23:36:45.164602 ip-10-0-128-197 kubenswrapper[2563]: E0715 23:36:45.164236 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mp4pr" podUID="b0b0bc56-c9c9-4a96-a97b-15f00f3b2099" Jul 15 23:36:45.164602 ip-10-0-128-197 kubenswrapper[2563]: E0715 23:36:45.164300 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4zw2v" podUID="10a402bd-5fc8-41c2-852a-5df876f74bdb" Jul 15 23:36:46.163366 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:46.163331 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7n4t9" Jul 15 23:36:46.163535 ip-10-0-128-197 kubenswrapper[2563]: E0715 23:36:46.163459 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7n4t9" podUID="d1c182db-bcdc-40a1-bb82-4f8b64173e43" Jul 15 23:36:47.163814 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:47.163754 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4zw2v" Jul 15 23:36:47.163814 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:47.163798 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mp4pr" Jul 15 23:36:47.164347 ip-10-0-128-197 kubenswrapper[2563]: E0715 23:36:47.163883 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4zw2v" podUID="10a402bd-5fc8-41c2-852a-5df876f74bdb" Jul 15 23:36:47.164347 ip-10-0-128-197 kubenswrapper[2563]: E0715 23:36:47.164036 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mp4pr" podUID="b0b0bc56-c9c9-4a96-a97b-15f00f3b2099" Jul 15 23:36:47.931601 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:47.930468 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-6ff59" event={"ID":"2deddafa-cbdf-407b-98c2-7c73bd107f1b","Type":"ContainerStarted","Data":"b7bcc493f5f4c57ae76d1dba902b34f317d12a9feac25497caea763bed938c7a"} Jul 15 23:36:47.943998 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:47.943610 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-197.ec2.internal" podStartSLOduration=20.943596357 podStartE2EDuration="20.943596357s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 23:36:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 23:36:31.906138884 +0000 UTC m=+7.279596848" watchObservedRunningTime="2026-07-15 23:36:47.943596357 +0000 UTC m=+23.317054319" Jul 15 23:36:47.943998 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:47.943864 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-6ff59" podStartSLOduration=7.643599963 podStartE2EDuration="22.943860215s" totalImagesPullingTime="15.300260252s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 23:36:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 23:36:47.94320784 +0000 UTC m=+23.316665804" watchObservedRunningTime="2026-07-15 23:36:47.943860215 +0000 UTC m=+23.317318198" Jul 15 23:36:48.163083 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:48.163062 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7n4t9" Jul 15 23:36:48.163180 ip-10-0-128-197 kubenswrapper[2563]: E0715 23:36:48.163162 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7n4t9" podUID="d1c182db-bcdc-40a1-bb82-4f8b64173e43" Jul 15 23:36:48.937934 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:48.937758 2563 generic.go:356] "Generic (PLEG): container finished" podID="a87574b4-736b-4e7d-9af6-3b6f89ac1c31" containerID="124ea7a573114866f0b646daac4237879ba22dc084e1488f1ede5e939651c414" exitCode=0 Jul 15 23:36:48.938797 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:48.937847 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tg24d" event={"ID":"a87574b4-736b-4e7d-9af6-3b6f89ac1c31","Type":"ContainerDied","Data":"124ea7a573114866f0b646daac4237879ba22dc084e1488f1ede5e939651c414"} Jul 15 23:36:48.939609 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:48.939429 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-z9q4h" event={"ID":"f9fcf8a8-82dc-4e0b-b211-4611f9436609","Type":"ContainerStarted","Data":"3c3ab7e0b25daf55055d59a5b25f45b2ccadfe47aecc32e351d76f8b9e1a520d"} Jul 15 23:36:48.940957 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:48.940935 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-tm65j" event={"ID":"1f66ebce-c0a5-4840-b106-3cedd6448ba5","Type":"ContainerStarted","Data":"e96068511a77c7c7d654c6f2727140a808dcbc17e4b6e658769b17d14b14a79a"} Jul 15 23:36:48.942404 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:48.942385 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6pwvf" event={"ID":"abb1e456-40be-492d-9d6b-d10413926119","Type":"ContainerStarted","Data":"c9d58bcd1261dd9fd11744d73122c3cc44af6434cfaa08addaf2303709d8cebf"} Jul 15 23:36:48.945451 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:48.945436 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d8f6k_2f64cab3-104a-44f6-b125-ba458f54f058/ovn-acl-logging/0.log" Jul 15 23:36:48.945730 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:48.945711 2563 generic.go:356] "Generic (PLEG): container finished" podID="2f64cab3-104a-44f6-b125-ba458f54f058" containerID="c6fdf76b941db6040a63b4e69f6b26ee8d5d86176772c41ee7d66eb2f63dc078" exitCode=1 Jul 15 23:36:48.945803 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:48.945775 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-d8f6k" event={"ID":"2f64cab3-104a-44f6-b125-ba458f54f058","Type":"ContainerStarted","Data":"b09abadbe01396a0629dcededf4028b4e69ac86b4cceac63388b445e540a99d5"} Jul 15 23:36:48.945847 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:48.945826 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-d8f6k" event={"ID":"2f64cab3-104a-44f6-b125-ba458f54f058","Type":"ContainerStarted","Data":"db4bb2aafb2fc401249a705c2f6f65ec8f009281bfe800dd6aeff6317b417cce"} Jul 15 23:36:48.945886 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:48.945857 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-d8f6k" event={"ID":"2f64cab3-104a-44f6-b125-ba458f54f058","Type":"ContainerStarted","Data":"43678cd7404d3e412e9302ddd74fee25688324ad1fd8e54c625b01fa073be3a0"} Jul 15 23:36:48.945936 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:48.945891 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-d8f6k" event={"ID":"2f64cab3-104a-44f6-b125-ba458f54f058","Type":"ContainerStarted","Data":"c651dc49b03b2362157c381960c88d6df4b2867a27d1a346c8661efba5d93f79"} Jul 15 23:36:48.945936 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:48.945921 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-d8f6k" event={"ID":"2f64cab3-104a-44f6-b125-ba458f54f058","Type":"ContainerDied","Data":"c6fdf76b941db6040a63b4e69f6b26ee8d5d86176772c41ee7d66eb2f63dc078"} Jul 15 23:36:48.946059 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:48.945949 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-d8f6k" event={"ID":"2f64cab3-104a-44f6-b125-ba458f54f058","Type":"ContainerStarted","Data":"e9141afbcede9d8d59cce9bf7afb9f3b2e6b374030a43e9d0d89ba402dd8a4a4"} Jul 15 23:36:48.947001 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:48.946982 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-v62rk" event={"ID":"12202d45-85c8-41de-b95e-f5cf403572ae","Type":"ContainerStarted","Data":"08348572ca29c91bba0c0da66cd50ae8746d1df63a8a368ef5da4255756c17eb"} Jul 15 23:36:48.948284 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:48.948266 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-khwkr" event={"ID":"498d1e24-acb4-4d75-a5a7-410f00ecc9f9","Type":"ContainerStarted","Data":"35b0878cadb73ef260123cd6a15bef68d4e1396933d5a65022afc73f54e36c8d"} Jul 15 23:36:48.969502 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:48.969462 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-z9q4h" podStartSLOduration=4.44037663 podStartE2EDuration="23.969451513s" totalImagesPullingTime="19.529074883s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 23:36:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 23:36:48.969083374 +0000 UTC m=+24.342541337" watchObservedRunningTime="2026-07-15 23:36:48.969451513 +0000 UTC m=+24.342909476" Jul 15 23:36:48.983312 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:48.983279 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-v62rk" podStartSLOduration=4.391500138 podStartE2EDuration="23.983270654s" totalImagesPullingTime="19.591770516s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 23:36:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 23:36:48.983142658 +0000 UTC m=+24.356600620" watchObservedRunningTime="2026-07-15 23:36:48.983270654 +0000 UTC m=+24.356728616" Jul 15 23:36:48.998471 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:48.998437 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-tm65j" podStartSLOduration=4.465382711 podStartE2EDuration="23.99842754s" totalImagesPullingTime="19.533044829s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 23:36:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 23:36:48.998331462 +0000 UTC m=+24.371789417" watchObservedRunningTime="2026-07-15 23:36:48.99842754 +0000 UTC m=+24.371885503" Jul 15 23:36:49.010746 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:49.010718 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-khwkr" podStartSLOduration=4.441825888 podStartE2EDuration="24.010709337s" totalImagesPullingTime="19.568883449s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 23:36:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 23:36:49.010222829 +0000 UTC m=+24.383680812" watchObservedRunningTime="2026-07-15 23:36:49.010709337 +0000 UTC m=+24.384167300" Jul 15 23:36:49.163874 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:49.163847 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4zw2v" Jul 15 23:36:49.164147 ip-10-0-128-197 kubenswrapper[2563]: E0715 23:36:49.163942 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4zw2v" podUID="10a402bd-5fc8-41c2-852a-5df876f74bdb" Jul 15 23:36:49.164147 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:49.164001 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mp4pr" Jul 15 23:36:49.164147 ip-10-0-128-197 kubenswrapper[2563]: E0715 23:36:49.164080 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mp4pr" podUID="b0b0bc56-c9c9-4a96-a97b-15f00f3b2099" Jul 15 23:36:49.288389 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:49.288236 2563 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 15 23:36:49.838012 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:49.837868 2563 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-15T23:36:49.288382962Z","UUID":"b86207b3-1594-40cd-8f57-27954122b09b","Handler":null,"Name":"","Endpoint":""} Jul 15 23:36:49.839565 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:49.839543 2563 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 15 23:36:49.839703 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:49.839583 2563 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 15 23:36:49.952756 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:49.952715 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6pwvf" event={"ID":"abb1e456-40be-492d-9d6b-d10413926119","Type":"ContainerStarted","Data":"18ed8ba619b1eee3704bed8e68d8192b648a815a2500277ad4d50513bd865751"} Jul 15 23:36:49.954472 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:49.954447 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-v2rfk" event={"ID":"387d6414-0c71-495c-acc0-9d64115d31fe","Type":"ContainerStarted","Data":"91178a69600384238c1d7dc7ed82fcdaac2f63d632eaab6342ebb1e9ee40652c"} Jul 15 23:36:50.163117 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:50.163093 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7n4t9" Jul 15 23:36:50.163288 ip-10-0-128-197 kubenswrapper[2563]: E0715 23:36:50.163263 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7n4t9" podUID="d1c182db-bcdc-40a1-bb82-4f8b64173e43" Jul 15 23:36:50.413290 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:50.413216 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-6ff59" Jul 15 23:36:50.413780 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:50.413751 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-6ff59" Jul 15 23:36:50.428151 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:50.428111 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-v2rfk" podStartSLOduration=5.856044744 podStartE2EDuration="25.428098376s" totalImagesPullingTime="19.572053632s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 23:36:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 23:36:49.96763465 +0000 UTC m=+25.341092612" watchObservedRunningTime="2026-07-15 23:36:50.428098376 +0000 UTC m=+25.801556338" Jul 15 23:36:50.959246 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:50.959160 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6pwvf" event={"ID":"abb1e456-40be-492d-9d6b-d10413926119","Type":"ContainerStarted","Data":"ed36291a7db6dc3581ec2700dddcfcc82f0870fcd2225db1cd4f44619a7add83"} Jul 15 23:36:50.962624 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:50.962596 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d8f6k_2f64cab3-104a-44f6-b125-ba458f54f058/ovn-acl-logging/0.log" Jul 15 23:36:50.963030 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:50.963001 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-d8f6k" event={"ID":"2f64cab3-104a-44f6-b125-ba458f54f058","Type":"ContainerStarted","Data":"240f2df647b9be1537cadec5e1c2a8c7187d1108d4bf301ec539251485460eef"} Jul 15 23:36:50.963338 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:50.963302 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-6ff59" Jul 15 23:36:50.963723 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:50.963695 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-6ff59" Jul 15 23:36:50.987162 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:50.987114 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6pwvf" podStartSLOduration=4.387396718 podStartE2EDuration="25.987094514s" totalImagesPullingTime="21.599697796s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 23:36:25 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 23:36:50.974670089 +0000 UTC m=+26.348128068" watchObservedRunningTime="2026-07-15 23:36:50.987094514 +0000 UTC m=+26.360552478" Jul 15 23:36:51.163528 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:51.163489 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mp4pr" Jul 15 23:36:51.163711 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:51.163499 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4zw2v" Jul 15 23:36:51.163711 ip-10-0-128-197 kubenswrapper[2563]: E0715 23:36:51.163623 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mp4pr" podUID="b0b0bc56-c9c9-4a96-a97b-15f00f3b2099" Jul 15 23:36:51.163711 ip-10-0-128-197 kubenswrapper[2563]: E0715 23:36:51.163693 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4zw2v" podUID="10a402bd-5fc8-41c2-852a-5df876f74bdb" Jul 15 23:36:52.163514 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:52.163483 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7n4t9" Jul 15 23:36:52.164126 ip-10-0-128-197 kubenswrapper[2563]: E0715 23:36:52.163621 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7n4t9" podUID="d1c182db-bcdc-40a1-bb82-4f8b64173e43" Jul 15 23:36:52.323451 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:52.323398 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:36:52.407859 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:52.407820 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:36:52.505310 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:52.505276 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:36:52.677221 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:52.677185 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:36:52.733532 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:52.733500 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:36:52.793395 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:52.793324 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:36:53.163658 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:53.163536 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4zw2v" Jul 15 23:36:53.164026 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:53.163598 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mp4pr" Jul 15 23:36:53.164026 ip-10-0-128-197 kubenswrapper[2563]: E0715 23:36:53.163847 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mp4pr" podUID="b0b0bc56-c9c9-4a96-a97b-15f00f3b2099" Jul 15 23:36:53.164026 ip-10-0-128-197 kubenswrapper[2563]: E0715 23:36:53.163734 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4zw2v" podUID="10a402bd-5fc8-41c2-852a-5df876f74bdb" Jul 15 23:36:53.971125 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:53.971087 2563 generic.go:356] "Generic (PLEG): container finished" podID="a87574b4-736b-4e7d-9af6-3b6f89ac1c31" containerID="7aa3836db336e0026d1c14506fbe4613c86bd0ec4d6fb0b26dd2ef74086e4cde" exitCode=0 Jul 15 23:36:53.971305 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:53.971169 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tg24d" event={"ID":"a87574b4-736b-4e7d-9af6-3b6f89ac1c31","Type":"ContainerDied","Data":"7aa3836db336e0026d1c14506fbe4613c86bd0ec4d6fb0b26dd2ef74086e4cde"} Jul 15 23:36:53.974927 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:53.974903 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d8f6k_2f64cab3-104a-44f6-b125-ba458f54f058/ovn-acl-logging/0.log" Jul 15 23:36:53.975288 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:53.975267 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-d8f6k" event={"ID":"2f64cab3-104a-44f6-b125-ba458f54f058","Type":"ContainerStarted","Data":"0c374aaa45d40b82df08650d94f14a4f4a64f2b040b008858d7901ce5c30211f"} Jul 15 23:36:53.975502 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:53.975489 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-d8f6k" Jul 15 23:36:53.975567 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:53.975534 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-d8f6k" Jul 15 23:36:53.975644 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:53.975627 2563 scope.go:117] "RemoveContainer" containerID="c6fdf76b941db6040a63b4e69f6b26ee8d5d86176772c41ee7d66eb2f63dc078" Jul 15 23:36:53.991098 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:53.991081 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-d8f6k" Jul 15 23:36:54.163873 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:54.163844 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7n4t9" Jul 15 23:36:54.164267 ip-10-0-128-197 kubenswrapper[2563]: E0715 23:36:54.163945 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7n4t9" podUID="d1c182db-bcdc-40a1-bb82-4f8b64173e43" Jul 15 23:36:54.980504 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:54.980368 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d8f6k_2f64cab3-104a-44f6-b125-ba458f54f058/ovn-acl-logging/0.log" Jul 15 23:36:54.980799 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:54.980778 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-d8f6k" event={"ID":"2f64cab3-104a-44f6-b125-ba458f54f058","Type":"ContainerStarted","Data":"59c6081420aa6328bd57a62fca52bf8fb8fb34946b70a3da85d35bca4b2e273b"} Jul 15 23:36:54.981092 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:54.981073 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-d8f6k" Jul 15 23:36:54.995652 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:54.995633 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-d8f6k" Jul 15 23:36:55.005562 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:55.005498 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-d8f6k" podStartSLOduration=10.310850482 podStartE2EDuration="30.005483733s" totalImagesPullingTime="19.694633251s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 23:36:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 23:36:55.003762436 +0000 UTC m=+30.377220398" watchObservedRunningTime="2026-07-15 23:36:55.005483733 +0000 UTC m=+30.378941698" Jul 15 23:36:55.021645 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:55.021620 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-4zw2v"] Jul 15 23:36:55.021751 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:55.021738 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4zw2v" Jul 15 23:36:55.021853 ip-10-0-128-197 kubenswrapper[2563]: E0715 23:36:55.021835 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4zw2v" podUID="10a402bd-5fc8-41c2-852a-5df876f74bdb" Jul 15 23:36:55.023925 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:55.023904 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-mp4pr"] Jul 15 23:36:55.024050 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:55.024012 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mp4pr" Jul 15 23:36:55.024106 ip-10-0-128-197 kubenswrapper[2563]: E0715 23:36:55.024088 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mp4pr" podUID="b0b0bc56-c9c9-4a96-a97b-15f00f3b2099" Jul 15 23:36:55.024573 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:55.024554 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-7n4t9"] Jul 15 23:36:55.024654 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:55.024641 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7n4t9" Jul 15 23:36:55.024776 ip-10-0-128-197 kubenswrapper[2563]: E0715 23:36:55.024753 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7n4t9" podUID="d1c182db-bcdc-40a1-bb82-4f8b64173e43" Jul 15 23:36:55.079918 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:55.079848 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:36:55.132062 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:55.132026 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:36:55.441707 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:55.441616 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:36:55.500008 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:55.499982 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:36:55.985118 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:55.985086 2563 generic.go:356] "Generic (PLEG): container finished" podID="a87574b4-736b-4e7d-9af6-3b6f89ac1c31" containerID="ce9a34d3890775020fa117a118e1cd386eb8b125c8e3159322b2d236f249b04b" exitCode=0 Jul 15 23:36:55.985282 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:55.985147 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tg24d" event={"ID":"a87574b4-736b-4e7d-9af6-3b6f89ac1c31","Type":"ContainerDied","Data":"ce9a34d3890775020fa117a118e1cd386eb8b125c8e3159322b2d236f249b04b"} Jul 15 23:36:56.395351 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:56.395279 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:36:56.495086 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:56.495059 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:36:57.163610 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:57.163582 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4zw2v" Jul 15 23:36:57.163733 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:57.163582 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7n4t9" Jul 15 23:36:57.163733 ip-10-0-128-197 kubenswrapper[2563]: E0715 23:36:57.163701 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4zw2v" podUID="10a402bd-5fc8-41c2-852a-5df876f74bdb" Jul 15 23:36:57.163816 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:57.163588 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mp4pr" Jul 15 23:36:57.163816 ip-10-0-128-197 kubenswrapper[2563]: E0715 23:36:57.163797 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7n4t9" podUID="d1c182db-bcdc-40a1-bb82-4f8b64173e43" Jul 15 23:36:57.163886 ip-10-0-128-197 kubenswrapper[2563]: E0715 23:36:57.163834 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mp4pr" podUID="b0b0bc56-c9c9-4a96-a97b-15f00f3b2099" Jul 15 23:36:57.421420 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:57.421181 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:36:57.991996 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:57.991948 2563 generic.go:356] "Generic (PLEG): container finished" podID="a87574b4-736b-4e7d-9af6-3b6f89ac1c31" containerID="0ce2e64c1f5681457b6410841ac5179ab7946c9ffde9ea3a4d34bb81d8836dad" exitCode=0 Jul 15 23:36:57.992563 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:57.992019 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tg24d" event={"ID":"a87574b4-736b-4e7d-9af6-3b6f89ac1c31","Type":"ContainerDied","Data":"0ce2e64c1f5681457b6410841ac5179ab7946c9ffde9ea3a4d34bb81d8836dad"} Jul 15 23:36:59.163179 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:59.163145 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7n4t9" Jul 15 23:36:59.163600 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:59.163145 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4zw2v" Jul 15 23:36:59.163600 ip-10-0-128-197 kubenswrapper[2563]: E0715 23:36:59.163311 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7n4t9" podUID="d1c182db-bcdc-40a1-bb82-4f8b64173e43" Jul 15 23:36:59.163600 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:59.163145 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mp4pr" Jul 15 23:36:59.163600 ip-10-0-128-197 kubenswrapper[2563]: E0715 23:36:59.163368 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4zw2v" podUID="10a402bd-5fc8-41c2-852a-5df876f74bdb" Jul 15 23:36:59.163600 ip-10-0-128-197 kubenswrapper[2563]: E0715 23:36:59.163432 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mp4pr" podUID="b0b0bc56-c9c9-4a96-a97b-15f00f3b2099" Jul 15 23:36:59.682675 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:59.682643 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d1c182db-bcdc-40a1-bb82-4f8b64173e43-metrics-certs\") pod \"network-metrics-daemon-7n4t9\" (UID: \"d1c182db-bcdc-40a1-bb82-4f8b64173e43\") " pod="openshift-multus/network-metrics-daemon-7n4t9" Jul 15 23:36:59.682854 ip-10-0-128-197 kubenswrapper[2563]: E0715 23:36:59.682793 2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 23:36:59.682854 ip-10-0-128-197 kubenswrapper[2563]: E0715 23:36:59.682846 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d1c182db-bcdc-40a1-bb82-4f8b64173e43-metrics-certs podName:d1c182db-bcdc-40a1-bb82-4f8b64173e43 nodeName:}" failed. No retries permitted until 2026-07-15 23:37:31.682831632 +0000 UTC m=+67.056289576 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d1c182db-bcdc-40a1-bb82-4f8b64173e43-metrics-certs") pod "network-metrics-daemon-7n4t9" (UID: "d1c182db-bcdc-40a1-bb82-4f8b64173e43") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 23:36:59.784009 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:59.783959 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v47j5\" (UniqueName: \"kubernetes.io/projected/b0b0bc56-c9c9-4a96-a97b-15f00f3b2099-kube-api-access-v47j5\") pod \"network-check-target-mp4pr\" (UID: \"b0b0bc56-c9c9-4a96-a97b-15f00f3b2099\") " pod="openshift-network-diagnostics/network-check-target-mp4pr" Jul 15 23:36:59.784180 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:36:59.784070 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/10a402bd-5fc8-41c2-852a-5df876f74bdb-original-pull-secret\") pod \"global-pull-secret-syncer-4zw2v\" (UID: \"10a402bd-5fc8-41c2-852a-5df876f74bdb\") " pod="kube-system/global-pull-secret-syncer-4zw2v" Jul 15 23:36:59.784180 ip-10-0-128-197 kubenswrapper[2563]: E0715 23:36:59.784116 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 15 23:36:59.784180 ip-10-0-128-197 kubenswrapper[2563]: E0715 23:36:59.784141 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 15 23:36:59.784180 ip-10-0-128-197 kubenswrapper[2563]: E0715 23:36:59.784153 2563 projected.go:196] Error preparing data for projected volume kube-api-access-v47j5 for pod openshift-network-diagnostics/network-check-target-mp4pr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 23:36:59.784358 ip-10-0-128-197 kubenswrapper[2563]: E0715 23:36:59.784183 2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 15 23:36:59.784358 ip-10-0-128-197 kubenswrapper[2563]: E0715 23:36:59.784211 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b0b0bc56-c9c9-4a96-a97b-15f00f3b2099-kube-api-access-v47j5 podName:b0b0bc56-c9c9-4a96-a97b-15f00f3b2099 nodeName:}" failed. No retries permitted until 2026-07-15 23:37:31.784196715 +0000 UTC m=+67.157654655 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-v47j5" (UniqueName: "kubernetes.io/projected/b0b0bc56-c9c9-4a96-a97b-15f00f3b2099-kube-api-access-v47j5") pod "network-check-target-mp4pr" (UID: "b0b0bc56-c9c9-4a96-a97b-15f00f3b2099") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 23:36:59.784358 ip-10-0-128-197 kubenswrapper[2563]: E0715 23:36:59.784236 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/10a402bd-5fc8-41c2-852a-5df876f74bdb-original-pull-secret podName:10a402bd-5fc8-41c2-852a-5df876f74bdb nodeName:}" failed. No retries permitted until 2026-07-15 23:37:31.784219423 +0000 UTC m=+67.157677368 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/10a402bd-5fc8-41c2-852a-5df876f74bdb-original-pull-secret") pod "global-pull-secret-syncer-4zw2v" (UID: "10a402bd-5fc8-41c2-852a-5df876f74bdb") : object "kube-system"/"original-pull-secret" not registered Jul 15 23:37:00.106362 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:00.106329 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:37:00.205310 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:00.205278 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:37:00.280099 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:00.280070 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:37:00.527337 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:00.527283 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:37:00.726775 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:00.726748 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:37:00.779438 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:00.779365 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:37:00.983309 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:00.983274 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:37:01.071128 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:01.071046 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:37:01.130006 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:01.129943 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:37:01.163192 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:01.163162 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4zw2v" Jul 15 23:37:01.163378 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:01.163218 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mp4pr" Jul 15 23:37:01.163378 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:01.163162 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7n4t9" Jul 15 23:37:01.163378 ip-10-0-128-197 kubenswrapper[2563]: E0715 23:37:01.163331 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4zw2v" podUID="10a402bd-5fc8-41c2-852a-5df876f74bdb" Jul 15 23:37:01.163543 ip-10-0-128-197 kubenswrapper[2563]: E0715 23:37:01.163402 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mp4pr" podUID="b0b0bc56-c9c9-4a96-a97b-15f00f3b2099" Jul 15 23:37:01.163543 ip-10-0-128-197 kubenswrapper[2563]: E0715 23:37:01.163509 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7n4t9" podUID="d1c182db-bcdc-40a1-bb82-4f8b64173e43" Jul 15 23:37:01.470243 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:01.470174 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-197.ec2.internal" event="NodeReady" Jul 15 23:37:01.473358 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:01.470707 2563 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 15 23:37:01.512833 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:01.512800 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-wkhxm"] Jul 15 23:37:01.515288 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:01.515254 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-wkhxm" Jul 15 23:37:01.515732 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:01.515700 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-vlt7c"] Jul 15 23:37:01.517459 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:01.517439 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-vlt7c" Jul 15 23:37:01.517578 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:01.517485 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-qqf2z\"" Jul 15 23:37:01.517578 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:01.517552 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 15 23:37:01.517692 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:01.517663 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 15 23:37:01.519623 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:01.519604 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-cl7fb\"" Jul 15 23:37:01.519905 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:01.519879 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 15 23:37:01.520013 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:01.519942 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 15 23:37:01.520173 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:01.520157 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 15 23:37:01.524943 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:01.524922 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-wkhxm"] Jul 15 23:37:01.534184 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:01.534162 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-vlt7c"] Jul 15 23:37:01.543295 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:01.543273 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-5q8md"] Jul 15 23:37:01.545565 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:01.545544 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-5q8md" Jul 15 23:37:01.548039 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:01.548010 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 15 23:37:01.548039 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:01.548035 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 15 23:37:01.548199 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:01.548048 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 15 23:37:01.548199 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:01.548062 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-58kg9\"" Jul 15 23:37:01.548379 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:01.548364 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 15 23:37:01.554869 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:01.554847 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-5q8md"] Jul 15 23:37:01.596415 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:01.596382 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/def13391-3ff3-48cf-83d3-03df84d280da-cert\") pod \"ingress-canary-vlt7c\" (UID: \"def13391-3ff3-48cf-83d3-03df84d280da\") " pod="openshift-ingress-canary/ingress-canary-vlt7c" Jul 15 23:37:01.596603 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:01.596459 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/409f6854-b231-41aa-8715-a300ee9aa09e-crio-socket\") pod \"insights-runtime-extractor-5q8md\" (UID: \"409f6854-b231-41aa-8715-a300ee9aa09e\") " pod="openshift-insights/insights-runtime-extractor-5q8md" Jul 15 23:37:01.596603 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:01.596494 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/409f6854-b231-41aa-8715-a300ee9aa09e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5q8md\" (UID: \"409f6854-b231-41aa-8715-a300ee9aa09e\") " pod="openshift-insights/insights-runtime-extractor-5q8md" Jul 15 23:37:01.596603 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:01.596520 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4nt7\" (UniqueName: \"kubernetes.io/projected/e53610cd-0777-4a19-afd0-6133052fc189-kube-api-access-l4nt7\") pod \"dns-default-wkhxm\" (UID: \"e53610cd-0777-4a19-afd0-6133052fc189\") " pod="openshift-dns/dns-default-wkhxm" Jul 15 23:37:01.596603 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:01.596602 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/409f6854-b231-41aa-8715-a300ee9aa09e-data-volume\") pod \"insights-runtime-extractor-5q8md\" (UID: \"409f6854-b231-41aa-8715-a300ee9aa09e\") " pod="openshift-insights/insights-runtime-extractor-5q8md" Jul 15 23:37:01.596805 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:01.596627 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sb44j\" (UniqueName: \"kubernetes.io/projected/409f6854-b231-41aa-8715-a300ee9aa09e-kube-api-access-sb44j\") pod \"insights-runtime-extractor-5q8md\" (UID: \"409f6854-b231-41aa-8715-a300ee9aa09e\") " pod="openshift-insights/insights-runtime-extractor-5q8md" Jul 15 23:37:01.596805 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:01.596664 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hz2h6\" (UniqueName: \"kubernetes.io/projected/def13391-3ff3-48cf-83d3-03df84d280da-kube-api-access-hz2h6\") pod \"ingress-canary-vlt7c\" (UID: \"def13391-3ff3-48cf-83d3-03df84d280da\") " pod="openshift-ingress-canary/ingress-canary-vlt7c" Jul 15 23:37:01.596805 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:01.596706 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e53610cd-0777-4a19-afd0-6133052fc189-tmp-dir\") pod \"dns-default-wkhxm\" (UID: \"e53610cd-0777-4a19-afd0-6133052fc189\") " pod="openshift-dns/dns-default-wkhxm" Jul 15 23:37:01.596805 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:01.596743 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/409f6854-b231-41aa-8715-a300ee9aa09e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5q8md\" (UID: \"409f6854-b231-41aa-8715-a300ee9aa09e\") " pod="openshift-insights/insights-runtime-extractor-5q8md" Jul 15 23:37:01.596805 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:01.596771 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e53610cd-0777-4a19-afd0-6133052fc189-metrics-tls\") pod \"dns-default-wkhxm\" (UID: \"e53610cd-0777-4a19-afd0-6133052fc189\") " pod="openshift-dns/dns-default-wkhxm" Jul 15 23:37:01.596805 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:01.596797 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e53610cd-0777-4a19-afd0-6133052fc189-config-volume\") pod \"dns-default-wkhxm\" (UID: \"e53610cd-0777-4a19-afd0-6133052fc189\") " pod="openshift-dns/dns-default-wkhxm" Jul 15 23:37:01.643884 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:01.643838 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:37:01.698089 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:01.698047 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/409f6854-b231-41aa-8715-a300ee9aa09e-data-volume\") pod \"insights-runtime-extractor-5q8md\" (UID: \"409f6854-b231-41aa-8715-a300ee9aa09e\") " pod="openshift-insights/insights-runtime-extractor-5q8md" Jul 15 23:37:01.698089 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:01.698097 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sb44j\" (UniqueName: \"kubernetes.io/projected/409f6854-b231-41aa-8715-a300ee9aa09e-kube-api-access-sb44j\") pod \"insights-runtime-extractor-5q8md\" (UID: \"409f6854-b231-41aa-8715-a300ee9aa09e\") " pod="openshift-insights/insights-runtime-extractor-5q8md" Jul 15 23:37:01.698330 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:01.698245 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hz2h6\" (UniqueName: \"kubernetes.io/projected/def13391-3ff3-48cf-83d3-03df84d280da-kube-api-access-hz2h6\") pod \"ingress-canary-vlt7c\" (UID: \"def13391-3ff3-48cf-83d3-03df84d280da\") " pod="openshift-ingress-canary/ingress-canary-vlt7c" Jul 15 23:37:01.698330 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:01.698292 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e53610cd-0777-4a19-afd0-6133052fc189-tmp-dir\") pod \"dns-default-wkhxm\" (UID: \"e53610cd-0777-4a19-afd0-6133052fc189\") " pod="openshift-dns/dns-default-wkhxm" Jul 15 23:37:01.698330 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:01.698318 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/409f6854-b231-41aa-8715-a300ee9aa09e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5q8md\" (UID: \"409f6854-b231-41aa-8715-a300ee9aa09e\") " pod="openshift-insights/insights-runtime-extractor-5q8md" Jul 15 23:37:01.698474 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:01.698342 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e53610cd-0777-4a19-afd0-6133052fc189-metrics-tls\") pod \"dns-default-wkhxm\" (UID: \"e53610cd-0777-4a19-afd0-6133052fc189\") " pod="openshift-dns/dns-default-wkhxm" Jul 15 23:37:01.698474 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:01.698370 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e53610cd-0777-4a19-afd0-6133052fc189-config-volume\") pod \"dns-default-wkhxm\" (UID: \"e53610cd-0777-4a19-afd0-6133052fc189\") " pod="openshift-dns/dns-default-wkhxm" Jul 15 23:37:01.698474 ip-10-0-128-197 kubenswrapper[2563]: E0715 23:37:01.698434 2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 15 23:37:01.698613 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:01.698493 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/def13391-3ff3-48cf-83d3-03df84d280da-cert\") pod \"ingress-canary-vlt7c\" (UID: \"def13391-3ff3-48cf-83d3-03df84d280da\") " pod="openshift-ingress-canary/ingress-canary-vlt7c" Jul 15 23:37:01.698613 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:01.698432 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/409f6854-b231-41aa-8715-a300ee9aa09e-data-volume\") pod \"insights-runtime-extractor-5q8md\" (UID: \"409f6854-b231-41aa-8715-a300ee9aa09e\") " pod="openshift-insights/insights-runtime-extractor-5q8md" Jul 15 23:37:01.698613 ip-10-0-128-197 kubenswrapper[2563]: E0715 23:37:01.698524 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/409f6854-b231-41aa-8715-a300ee9aa09e-kube-rbac-proxy-cm podName:409f6854-b231-41aa-8715-a300ee9aa09e nodeName:}" failed. No retries permitted until 2026-07-15 23:37:02.19850338 +0000 UTC m=+37.571961336 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/409f6854-b231-41aa-8715-a300ee9aa09e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5q8md" (UID: "409f6854-b231-41aa-8715-a300ee9aa09e") : configmap "kube-rbac-proxy" not found Jul 15 23:37:01.698613 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:01.698563 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/409f6854-b231-41aa-8715-a300ee9aa09e-crio-socket\") pod \"insights-runtime-extractor-5q8md\" (UID: \"409f6854-b231-41aa-8715-a300ee9aa09e\") " pod="openshift-insights/insights-runtime-extractor-5q8md" Jul 15 23:37:01.698613 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:01.698591 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/409f6854-b231-41aa-8715-a300ee9aa09e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5q8md\" (UID: \"409f6854-b231-41aa-8715-a300ee9aa09e\") " pod="openshift-insights/insights-runtime-extractor-5q8md" Jul 15 23:37:01.698827 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:01.698619 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4nt7\" (UniqueName: \"kubernetes.io/projected/e53610cd-0777-4a19-afd0-6133052fc189-kube-api-access-l4nt7\") pod \"dns-default-wkhxm\" (UID: \"e53610cd-0777-4a19-afd0-6133052fc189\") " pod="openshift-dns/dns-default-wkhxm" Jul 15 23:37:01.698827 ip-10-0-128-197 kubenswrapper[2563]: E0715 23:37:01.698744 2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 15 23:37:01.698827 ip-10-0-128-197 kubenswrapper[2563]: E0715 23:37:01.698782 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/409f6854-b231-41aa-8715-a300ee9aa09e-insights-runtime-extractor-tls podName:409f6854-b231-41aa-8715-a300ee9aa09e nodeName:}" failed. No retries permitted until 2026-07-15 23:37:02.198770791 +0000 UTC m=+37.572228733 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/409f6854-b231-41aa-8715-a300ee9aa09e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5q8md" (UID: "409f6854-b231-41aa-8715-a300ee9aa09e") : secret "insights-runtime-extractor-tls" not found Jul 15 23:37:01.698827 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:01.698744 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e53610cd-0777-4a19-afd0-6133052fc189-tmp-dir\") pod \"dns-default-wkhxm\" (UID: \"e53610cd-0777-4a19-afd0-6133052fc189\") " pod="openshift-dns/dns-default-wkhxm" Jul 15 23:37:01.698827 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:01.698793 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/409f6854-b231-41aa-8715-a300ee9aa09e-crio-socket\") pod \"insights-runtime-extractor-5q8md\" (UID: \"409f6854-b231-41aa-8715-a300ee9aa09e\") " pod="openshift-insights/insights-runtime-extractor-5q8md" Jul 15 23:37:01.699107 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:01.698939 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e53610cd-0777-4a19-afd0-6133052fc189-config-volume\") pod \"dns-default-wkhxm\" (UID: \"e53610cd-0777-4a19-afd0-6133052fc189\") " pod="openshift-dns/dns-default-wkhxm" Jul 15 23:37:01.702883 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:01.702861 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e53610cd-0777-4a19-afd0-6133052fc189-metrics-tls\") pod \"dns-default-wkhxm\" (UID: \"e53610cd-0777-4a19-afd0-6133052fc189\") " pod="openshift-dns/dns-default-wkhxm" Jul 15 23:37:01.703005 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:01.702926 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/def13391-3ff3-48cf-83d3-03df84d280da-cert\") pod \"ingress-canary-vlt7c\" (UID: \"def13391-3ff3-48cf-83d3-03df84d280da\") " pod="openshift-ingress-canary/ingress-canary-vlt7c" Jul 15 23:37:01.712751 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:01.712726 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sb44j\" (UniqueName: \"kubernetes.io/projected/409f6854-b231-41aa-8715-a300ee9aa09e-kube-api-access-sb44j\") pod \"insights-runtime-extractor-5q8md\" (UID: \"409f6854-b231-41aa-8715-a300ee9aa09e\") " pod="openshift-insights/insights-runtime-extractor-5q8md" Jul 15 23:37:01.712937 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:01.712918 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-68c9cbb497-4xmmb"] Jul 15 23:37:01.715303 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:01.715280 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4nt7\" (UniqueName: \"kubernetes.io/projected/e53610cd-0777-4a19-afd0-6133052fc189-kube-api-access-l4nt7\") pod \"dns-default-wkhxm\" (UID: \"e53610cd-0777-4a19-afd0-6133052fc189\") " pod="openshift-dns/dns-default-wkhxm" Jul 15 23:37:01.715576 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:01.715558 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-68c9cbb497-4xmmb" Jul 15 23:37:01.715698 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:01.715678 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hz2h6\" (UniqueName: \"kubernetes.io/projected/def13391-3ff3-48cf-83d3-03df84d280da-kube-api-access-hz2h6\") pod \"ingress-canary-vlt7c\" (UID: \"def13391-3ff3-48cf-83d3-03df84d280da\") " pod="openshift-ingress-canary/ingress-canary-vlt7c" Jul 15 23:37:01.716637 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:01.716611 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:37:01.717848 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:01.717828 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 15 23:37:01.718052 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:01.718023 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 15 23:37:01.718704 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:01.718317 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-74xmc\"" Jul 15 23:37:01.749199 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:01.748936 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-68c9cbb497-4xmmb"] Jul 15 23:37:01.772799 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:01.772755 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:37:01.799980 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:01.799925 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zc74c\" (UniqueName: \"kubernetes.io/projected/59e5d97d-b146-4ca9-96c3-a2702d1d36ae-kube-api-access-zc74c\") pod \"downloads-68c9cbb497-4xmmb\" (UID: \"59e5d97d-b146-4ca9-96c3-a2702d1d36ae\") " pod="openshift-console/downloads-68c9cbb497-4xmmb" Jul 15 23:37:01.828928 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:01.828899 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-wkhxm" Jul 15 23:37:01.834748 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:01.834728 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-vlt7c" Jul 15 23:37:01.875255 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:01.875186 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:37:01.902101 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:01.901908 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zc74c\" (UniqueName: \"kubernetes.io/projected/59e5d97d-b146-4ca9-96c3-a2702d1d36ae-kube-api-access-zc74c\") pod \"downloads-68c9cbb497-4xmmb\" (UID: \"59e5d97d-b146-4ca9-96c3-a2702d1d36ae\") " pod="openshift-console/downloads-68c9cbb497-4xmmb" Jul 15 23:37:01.911801 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:01.911427 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zc74c\" (UniqueName: \"kubernetes.io/projected/59e5d97d-b146-4ca9-96c3-a2702d1d36ae-kube-api-access-zc74c\") pod \"downloads-68c9cbb497-4xmmb\" (UID: \"59e5d97d-b146-4ca9-96c3-a2702d1d36ae\") " pod="openshift-console/downloads-68c9cbb497-4xmmb" Jul 15 23:37:01.979566 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:01.979537 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-vlt7c"] Jul 15 23:37:01.987036 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:01.986958 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-wkhxm"] Jul 15 23:37:01.990641 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:37:01.990609 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddef13391_3ff3_48cf_83d3_03df84d280da.slice/crio-04e0a6a68b1d204a1414bb3d6cf4c1b56c59aa5f89a952d61ab273233902adb9 WatchSource:0}: Error finding container 04e0a6a68b1d204a1414bb3d6cf4c1b56c59aa5f89a952d61ab273233902adb9: Status 404 returned error can't find the container with id 04e0a6a68b1d204a1414bb3d6cf4c1b56c59aa5f89a952d61ab273233902adb9 Jul 15 23:37:01.991310 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:37:01.991271 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode53610cd_0777_4a19_afd0_6133052fc189.slice/crio-1703acf51e21abddefd9ed7d46fb486717a31971972f517173cd5a8d8f765ff9 WatchSource:0}: Error finding container 1703acf51e21abddefd9ed7d46fb486717a31971972f517173cd5a8d8f765ff9: Status 404 returned error can't find the container with id 1703acf51e21abddefd9ed7d46fb486717a31971972f517173cd5a8d8f765ff9 Jul 15 23:37:02.004262 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:02.004202 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:37:02.004950 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:02.004927 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-vlt7c" event={"ID":"def13391-3ff3-48cf-83d3-03df84d280da","Type":"ContainerStarted","Data":"04e0a6a68b1d204a1414bb3d6cf4c1b56c59aa5f89a952d61ab273233902adb9"} Jul 15 23:37:02.006118 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:02.006096 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-wkhxm" event={"ID":"e53610cd-0777-4a19-afd0-6133052fc189","Type":"ContainerStarted","Data":"1703acf51e21abddefd9ed7d46fb486717a31971972f517173cd5a8d8f765ff9"} Jul 15 23:37:02.008807 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:02.008787 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tg24d" event={"ID":"a87574b4-736b-4e7d-9af6-3b6f89ac1c31","Type":"ContainerStarted","Data":"6c3813576f708f7d1a6a503ab1daeab33dd2620df4ac74ecb44af6858a657b1c"} Jul 15 23:37:02.055308 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:02.055278 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-68c9cbb497-4xmmb" Jul 15 23:37:02.180420 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:02.180378 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-68c9cbb497-4xmmb"] Jul 15 23:37:02.183680 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:02.183651 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:37:02.184017 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:37:02.183994 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59e5d97d_b146_4ca9_96c3_a2702d1d36ae.slice/crio-2c270573a1646456a8af537b541809dc14b149da515d0045f5e5581730283567 WatchSource:0}: Error finding container 2c270573a1646456a8af537b541809dc14b149da515d0045f5e5581730283567: Status 404 returned error can't find the container with id 2c270573a1646456a8af537b541809dc14b149da515d0045f5e5581730283567 Jul 15 23:37:02.204421 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:02.204396 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/409f6854-b231-41aa-8715-a300ee9aa09e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5q8md\" (UID: \"409f6854-b231-41aa-8715-a300ee9aa09e\") " pod="openshift-insights/insights-runtime-extractor-5q8md" Jul 15 23:37:02.204538 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:02.204437 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/409f6854-b231-41aa-8715-a300ee9aa09e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5q8md\" (UID: \"409f6854-b231-41aa-8715-a300ee9aa09e\") " pod="openshift-insights/insights-runtime-extractor-5q8md" Jul 15 23:37:02.204538 ip-10-0-128-197 kubenswrapper[2563]: E0715 23:37:02.204517 2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 15 23:37:02.204616 ip-10-0-128-197 kubenswrapper[2563]: E0715 23:37:02.204603 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/409f6854-b231-41aa-8715-a300ee9aa09e-kube-rbac-proxy-cm podName:409f6854-b231-41aa-8715-a300ee9aa09e nodeName:}" failed. No retries permitted until 2026-07-15 23:37:03.204582713 +0000 UTC m=+38.578040670 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/409f6854-b231-41aa-8715-a300ee9aa09e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5q8md" (UID: "409f6854-b231-41aa-8715-a300ee9aa09e") : configmap "kube-rbac-proxy" not found Jul 15 23:37:02.204616 ip-10-0-128-197 kubenswrapper[2563]: E0715 23:37:02.204610 2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 15 23:37:02.204683 ip-10-0-128-197 kubenswrapper[2563]: E0715 23:37:02.204652 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/409f6854-b231-41aa-8715-a300ee9aa09e-insights-runtime-extractor-tls podName:409f6854-b231-41aa-8715-a300ee9aa09e nodeName:}" failed. No retries permitted until 2026-07-15 23:37:03.204640387 +0000 UTC m=+38.578098330 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/409f6854-b231-41aa-8715-a300ee9aa09e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5q8md" (UID: "409f6854-b231-41aa-8715-a300ee9aa09e") : secret "insights-runtime-extractor-tls" not found Jul 15 23:37:02.460426 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:02.460340 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-z9q4h_f9fcf8a8-82dc-4e0b-b211-4611f9436609/dns-node-resolver/0.log" Jul 15 23:37:02.501004 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:02.500955 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:37:02.600869 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:02.600793 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:37:02.657065 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:02.657033 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:37:02.792003 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:02.791954 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:37:03.013665 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:03.013628 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-68c9cbb497-4xmmb" event={"ID":"59e5d97d-b146-4ca9-96c3-a2702d1d36ae","Type":"ContainerStarted","Data":"2c270573a1646456a8af537b541809dc14b149da515d0045f5e5581730283567"} Jul 15 23:37:03.017605 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:03.017566 2563 generic.go:356] "Generic (PLEG): container finished" podID="a87574b4-736b-4e7d-9af6-3b6f89ac1c31" containerID="6c3813576f708f7d1a6a503ab1daeab33dd2620df4ac74ecb44af6858a657b1c" exitCode=0 Jul 15 23:37:03.017605 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:03.017606 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tg24d" event={"ID":"a87574b4-736b-4e7d-9af6-3b6f89ac1c31","Type":"ContainerDied","Data":"6c3813576f708f7d1a6a503ab1daeab33dd2620df4ac74ecb44af6858a657b1c"} Jul 15 23:37:03.086058 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:03.085950 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:37:03.163622 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:03.163522 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mp4pr" Jul 15 23:37:03.163622 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:03.163597 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7n4t9" Jul 15 23:37:03.163622 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:03.163596 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4zw2v" Jul 15 23:37:03.166657 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:03.166094 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 15 23:37:03.166657 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:03.166149 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 15 23:37:03.166657 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:03.166367 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 15 23:37:03.166657 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:03.166413 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 15 23:37:03.166657 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:03.166367 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-qkb4z\"" Jul 15 23:37:03.166657 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:03.166608 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-s2fsj\"" Jul 15 23:37:03.187287 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:03.187203 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:37:03.212335 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:03.211700 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/409f6854-b231-41aa-8715-a300ee9aa09e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5q8md\" (UID: \"409f6854-b231-41aa-8715-a300ee9aa09e\") " pod="openshift-insights/insights-runtime-extractor-5q8md" Jul 15 23:37:03.212335 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:03.211764 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/409f6854-b231-41aa-8715-a300ee9aa09e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5q8md\" (UID: \"409f6854-b231-41aa-8715-a300ee9aa09e\") " pod="openshift-insights/insights-runtime-extractor-5q8md" Jul 15 23:37:03.212335 ip-10-0-128-197 kubenswrapper[2563]: E0715 23:37:03.211920 2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 15 23:37:03.212335 ip-10-0-128-197 kubenswrapper[2563]: E0715 23:37:03.212008 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/409f6854-b231-41aa-8715-a300ee9aa09e-insights-runtime-extractor-tls podName:409f6854-b231-41aa-8715-a300ee9aa09e nodeName:}" failed. No retries permitted until 2026-07-15 23:37:05.211958736 +0000 UTC m=+40.585416697 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/409f6854-b231-41aa-8715-a300ee9aa09e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5q8md" (UID: "409f6854-b231-41aa-8715-a300ee9aa09e") : secret "insights-runtime-extractor-tls" not found Jul 15 23:37:03.212335 ip-10-0-128-197 kubenswrapper[2563]: E0715 23:37:03.212062 2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 15 23:37:03.212335 ip-10-0-128-197 kubenswrapper[2563]: E0715 23:37:03.212130 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/409f6854-b231-41aa-8715-a300ee9aa09e-kube-rbac-proxy-cm podName:409f6854-b231-41aa-8715-a300ee9aa09e nodeName:}" failed. No retries permitted until 2026-07-15 23:37:05.212102217 +0000 UTC m=+40.585560159 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/409f6854-b231-41aa-8715-a300ee9aa09e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5q8md" (UID: "409f6854-b231-41aa-8715-a300ee9aa09e") : configmap "kube-rbac-proxy" not found Jul 15 23:37:03.260813 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:03.260721 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-khwkr_498d1e24-acb4-4d75-a5a7-410f00ecc9f9/node-ca/0.log" Jul 15 23:37:03.413881 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:03.413770 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:37:03.465934 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:03.465897 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:37:03.518905 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:03.518420 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:37:03.599233 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:03.599188 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:37:03.695743 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:03.695668 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:37:04.025310 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:04.025264 2563 generic.go:356] "Generic (PLEG): container finished" podID="a87574b4-736b-4e7d-9af6-3b6f89ac1c31" containerID="5dcd50343e0a2e676a1642f1e7ede631815fef1a82063d92b579eab62c7d79a2" exitCode=0 Jul 15 23:37:04.025505 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:04.025346 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tg24d" event={"ID":"a87574b4-736b-4e7d-9af6-3b6f89ac1c31","Type":"ContainerDied","Data":"5dcd50343e0a2e676a1642f1e7ede631815fef1a82063d92b579eab62c7d79a2"} Jul 15 23:37:04.293990 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:04.293855 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:37:04.319569 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:04.319492 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-64785b6675-gr6rk"] Jul 15 23:37:04.346570 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:04.346510 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-64785b6675-gr6rk"] Jul 15 23:37:04.346724 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:04.346624 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-64785b6675-gr6rk" Jul 15 23:37:04.349240 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:04.349178 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 15 23:37:04.350088 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:04.349843 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 15 23:37:04.350088 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:04.349892 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Jul 15 23:37:04.350088 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:04.349903 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 15 23:37:04.350088 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:04.349903 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Jul 15 23:37:04.350088 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:04.349960 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-88pfk\"" Jul 15 23:37:04.368345 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:04.368322 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:37:04.423069 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:04.423015 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1a03a952-c7c3-4029-b3dc-f5c233499798-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-64785b6675-gr6rk\" (UID: \"1a03a952-c7c3-4029-b3dc-f5c233499798\") " pod="openshift-monitoring/prometheus-operator-64785b6675-gr6rk" Jul 15 23:37:04.423221 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:04.423085 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1a03a952-c7c3-4029-b3dc-f5c233499798-metrics-client-ca\") pod \"prometheus-operator-64785b6675-gr6rk\" (UID: \"1a03a952-c7c3-4029-b3dc-f5c233499798\") " pod="openshift-monitoring/prometheus-operator-64785b6675-gr6rk" Jul 15 23:37:04.423221 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:04.423119 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6v92\" (UniqueName: \"kubernetes.io/projected/1a03a952-c7c3-4029-b3dc-f5c233499798-kube-api-access-q6v92\") pod \"prometheus-operator-64785b6675-gr6rk\" (UID: \"1a03a952-c7c3-4029-b3dc-f5c233499798\") " pod="openshift-monitoring/prometheus-operator-64785b6675-gr6rk" Jul 15 23:37:04.423329 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:04.423225 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/1a03a952-c7c3-4029-b3dc-f5c233499798-prometheus-operator-tls\") pod \"prometheus-operator-64785b6675-gr6rk\" (UID: \"1a03a952-c7c3-4029-b3dc-f5c233499798\") " pod="openshift-monitoring/prometheus-operator-64785b6675-gr6rk" Jul 15 23:37:04.524067 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:04.523716 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1a03a952-c7c3-4029-b3dc-f5c233499798-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-64785b6675-gr6rk\" (UID: \"1a03a952-c7c3-4029-b3dc-f5c233499798\") " pod="openshift-monitoring/prometheus-operator-64785b6675-gr6rk" Jul 15 23:37:04.524067 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:04.523778 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1a03a952-c7c3-4029-b3dc-f5c233499798-metrics-client-ca\") pod \"prometheus-operator-64785b6675-gr6rk\" (UID: \"1a03a952-c7c3-4029-b3dc-f5c233499798\") " pod="openshift-monitoring/prometheus-operator-64785b6675-gr6rk" Jul 15 23:37:04.524067 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:04.523798 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6v92\" (UniqueName: \"kubernetes.io/projected/1a03a952-c7c3-4029-b3dc-f5c233499798-kube-api-access-q6v92\") pod \"prometheus-operator-64785b6675-gr6rk\" (UID: \"1a03a952-c7c3-4029-b3dc-f5c233499798\") " pod="openshift-monitoring/prometheus-operator-64785b6675-gr6rk" Jul 15 23:37:04.524067 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:04.523848 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/1a03a952-c7c3-4029-b3dc-f5c233499798-prometheus-operator-tls\") pod \"prometheus-operator-64785b6675-gr6rk\" (UID: \"1a03a952-c7c3-4029-b3dc-f5c233499798\") " pod="openshift-monitoring/prometheus-operator-64785b6675-gr6rk" Jul 15 23:37:04.524067 ip-10-0-128-197 kubenswrapper[2563]: E0715 23:37:04.523990 2563 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found Jul 15 23:37:04.524067 ip-10-0-128-197 kubenswrapper[2563]: E0715 23:37:04.524051 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a03a952-c7c3-4029-b3dc-f5c233499798-prometheus-operator-tls podName:1a03a952-c7c3-4029-b3dc-f5c233499798 nodeName:}" failed. No retries permitted until 2026-07-15 23:37:05.024032286 +0000 UTC m=+40.397490244 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/1a03a952-c7c3-4029-b3dc-f5c233499798-prometheus-operator-tls") pod "prometheus-operator-64785b6675-gr6rk" (UID: "1a03a952-c7c3-4029-b3dc-f5c233499798") : secret "prometheus-operator-tls" not found Jul 15 23:37:04.524865 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:04.524580 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1a03a952-c7c3-4029-b3dc-f5c233499798-metrics-client-ca\") pod \"prometheus-operator-64785b6675-gr6rk\" (UID: \"1a03a952-c7c3-4029-b3dc-f5c233499798\") " pod="openshift-monitoring/prometheus-operator-64785b6675-gr6rk" Jul 15 23:37:04.528041 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:04.528006 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1a03a952-c7c3-4029-b3dc-f5c233499798-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-64785b6675-gr6rk\" (UID: \"1a03a952-c7c3-4029-b3dc-f5c233499798\") " pod="openshift-monitoring/prometheus-operator-64785b6675-gr6rk" Jul 15 23:37:04.534979 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:04.534937 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6v92\" (UniqueName: \"kubernetes.io/projected/1a03a952-c7c3-4029-b3dc-f5c233499798-kube-api-access-q6v92\") pod \"prometheus-operator-64785b6675-gr6rk\" (UID: \"1a03a952-c7c3-4029-b3dc-f5c233499798\") " pod="openshift-monitoring/prometheus-operator-64785b6675-gr6rk" Jul 15 23:37:04.908082 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:04.908018 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:37:05.029456 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:05.029416 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/1a03a952-c7c3-4029-b3dc-f5c233499798-prometheus-operator-tls\") pod \"prometheus-operator-64785b6675-gr6rk\" (UID: \"1a03a952-c7c3-4029-b3dc-f5c233499798\") " pod="openshift-monitoring/prometheus-operator-64785b6675-gr6rk" Jul 15 23:37:05.032180 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:05.032153 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/1a03a952-c7c3-4029-b3dc-f5c233499798-prometheus-operator-tls\") pod \"prometheus-operator-64785b6675-gr6rk\" (UID: \"1a03a952-c7c3-4029-b3dc-f5c233499798\") " pod="openshift-monitoring/prometheus-operator-64785b6675-gr6rk" Jul 15 23:37:05.130858 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:05.130790 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:37:05.230801 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:05.230732 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/409f6854-b231-41aa-8715-a300ee9aa09e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5q8md\" (UID: \"409f6854-b231-41aa-8715-a300ee9aa09e\") " pod="openshift-insights/insights-runtime-extractor-5q8md" Jul 15 23:37:05.230801 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:05.230800 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/409f6854-b231-41aa-8715-a300ee9aa09e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5q8md\" (UID: \"409f6854-b231-41aa-8715-a300ee9aa09e\") " pod="openshift-insights/insights-runtime-extractor-5q8md" Jul 15 23:37:05.231037 ip-10-0-128-197 kubenswrapper[2563]: E0715 23:37:05.231020 2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 15 23:37:05.231103 ip-10-0-128-197 kubenswrapper[2563]: E0715 23:37:05.231084 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/409f6854-b231-41aa-8715-a300ee9aa09e-insights-runtime-extractor-tls podName:409f6854-b231-41aa-8715-a300ee9aa09e nodeName:}" failed. No retries permitted until 2026-07-15 23:37:09.231065244 +0000 UTC m=+44.604523185 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/409f6854-b231-41aa-8715-a300ee9aa09e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5q8md" (UID: "409f6854-b231-41aa-8715-a300ee9aa09e") : secret "insights-runtime-extractor-tls" not found Jul 15 23:37:05.231168 ip-10-0-128-197 kubenswrapper[2563]: E0715 23:37:05.231127 2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 15 23:37:05.231231 ip-10-0-128-197 kubenswrapper[2563]: E0715 23:37:05.231184 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/409f6854-b231-41aa-8715-a300ee9aa09e-kube-rbac-proxy-cm podName:409f6854-b231-41aa-8715-a300ee9aa09e nodeName:}" failed. No retries permitted until 2026-07-15 23:37:09.231166172 +0000 UTC m=+44.604624119 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/409f6854-b231-41aa-8715-a300ee9aa09e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5q8md" (UID: "409f6854-b231-41aa-8715-a300ee9aa09e") : configmap "kube-rbac-proxy" not found Jul 15 23:37:05.231961 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:05.231929 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:37:05.258710 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:05.258686 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-64785b6675-gr6rk" Jul 15 23:37:05.332468 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:05.332376 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:37:05.423161 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:05.423042 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-64785b6675-gr6rk"] Jul 15 23:37:05.426647 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:37:05.426618 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a03a952_c7c3_4029_b3dc_f5c233499798.slice/crio-2e58929b68df21dbcda9306e468ba29d526c2a730006a02f7809d858bcd35e0c WatchSource:0}: Error finding container 2e58929b68df21dbcda9306e468ba29d526c2a730006a02f7809d858bcd35e0c: Status 404 returned error can't find the container with id 2e58929b68df21dbcda9306e468ba29d526c2a730006a02f7809d858bcd35e0c Jul 15 23:37:05.447386 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:05.447359 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:37:05.616594 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:05.616562 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:37:06.035310 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:06.035272 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-vlt7c" event={"ID":"def13391-3ff3-48cf-83d3-03df84d280da","Type":"ContainerStarted","Data":"0b42950997b44a3dc8f11271efb99ef2813d9a6c2f315eaa669cd3bf5f0b2ecb"} Jul 15 23:37:06.039520 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:06.039490 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tg24d" event={"ID":"a87574b4-736b-4e7d-9af6-3b6f89ac1c31","Type":"ContainerStarted","Data":"8f3282517e4422402c24ba4a0ae287b8b84f0493b90b556f2d533ebe1e270232"} Jul 15 23:37:06.041344 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:06.041319 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-wkhxm" event={"ID":"e53610cd-0777-4a19-afd0-6133052fc189","Type":"ContainerStarted","Data":"73bd7889154d0b660a0a80aca5e0888d2eed779375fe39c3f56024340e091ee0"} Jul 15 23:37:06.041482 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:06.041372 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-wkhxm" event={"ID":"e53610cd-0777-4a19-afd0-6133052fc189","Type":"ContainerStarted","Data":"4d1f9c7fb3d539f41f4301dcf14e43102c6215acf4598e71d982474274c1c52d"} Jul 15 23:37:06.041482 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:06.041403 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-wkhxm" Jul 15 23:37:06.042628 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:06.042604 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-64785b6675-gr6rk" event={"ID":"1a03a952-c7c3-4029-b3dc-f5c233499798","Type":"ContainerStarted","Data":"2e58929b68df21dbcda9306e468ba29d526c2a730006a02f7809d858bcd35e0c"} Jul 15 23:37:06.048823 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:06.048773 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-vlt7c" podStartSLOduration=1.986660233 podStartE2EDuration="5.048755235s" totalImagesPullingTime="3.062095002s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 23:37:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 23:37:06.048292238 +0000 UTC m=+41.421750202" watchObservedRunningTime="2026-07-15 23:37:06.048755235 +0000 UTC m=+41.422213182" Jul 15 23:37:06.068853 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:06.068805 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-tg24d" podStartSLOduration=11.572112353 podStartE2EDuration="41.0687854s" totalImagesPullingTime="29.496673047s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 23:36:25 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 23:37:06.067563242 +0000 UTC m=+41.441021206" watchObservedRunningTime="2026-07-15 23:37:06.0687854 +0000 UTC m=+41.442243364" Jul 15 23:37:06.084822 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:06.084778 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-wkhxm" podStartSLOduration=2.028395422 podStartE2EDuration="5.084763245s" totalImagesPullingTime="3.056367823s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 23:37:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 23:37:06.083492016 +0000 UTC m=+41.456949980" watchObservedRunningTime="2026-07-15 23:37:06.084763245 +0000 UTC m=+41.458221211" Jul 15 23:37:06.151523 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:06.151492 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:37:06.253206 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:06.253152 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:37:06.298722 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:06.298641 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:37:07.323599 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:07.323569 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:37:07.420658 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:07.420619 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:37:07.423664 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:07.423635 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-75d9954bb7-rlshl"] Jul 15 23:37:07.439945 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:07.439922 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-75d9954bb7-rlshl"] Jul 15 23:37:07.440072 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:07.440057 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-75d9954bb7-rlshl" Jul 15 23:37:07.442619 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:07.442360 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 15 23:37:07.442619 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:07.442403 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 15 23:37:07.442619 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:07.442474 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 15 23:37:07.442619 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:07.442609 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 15 23:37:07.443226 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:07.443199 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-89vtl\"" Jul 15 23:37:07.443394 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:07.443275 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 15 23:37:07.477187 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:07.477154 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:37:07.548959 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:07.548935 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5efd853d-ff2f-43d9-bbe8-fd8ee02465ec-console-config\") pod \"console-75d9954bb7-rlshl\" (UID: \"5efd853d-ff2f-43d9-bbe8-fd8ee02465ec\") " pod="openshift-console/console-75d9954bb7-rlshl" Jul 15 23:37:07.549092 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:07.549000 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qs7xp\" (UniqueName: \"kubernetes.io/projected/5efd853d-ff2f-43d9-bbe8-fd8ee02465ec-kube-api-access-qs7xp\") pod \"console-75d9954bb7-rlshl\" (UID: \"5efd853d-ff2f-43d9-bbe8-fd8ee02465ec\") " pod="openshift-console/console-75d9954bb7-rlshl" Jul 15 23:37:07.549092 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:07.549060 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5efd853d-ff2f-43d9-bbe8-fd8ee02465ec-console-serving-cert\") pod \"console-75d9954bb7-rlshl\" (UID: \"5efd853d-ff2f-43d9-bbe8-fd8ee02465ec\") " pod="openshift-console/console-75d9954bb7-rlshl" Jul 15 23:37:07.549092 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:07.549076 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5efd853d-ff2f-43d9-bbe8-fd8ee02465ec-service-ca\") pod \"console-75d9954bb7-rlshl\" (UID: \"5efd853d-ff2f-43d9-bbe8-fd8ee02465ec\") " pod="openshift-console/console-75d9954bb7-rlshl" Jul 15 23:37:07.549255 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:07.549122 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5efd853d-ff2f-43d9-bbe8-fd8ee02465ec-oauth-serving-cert\") pod \"console-75d9954bb7-rlshl\" (UID: \"5efd853d-ff2f-43d9-bbe8-fd8ee02465ec\") " pod="openshift-console/console-75d9954bb7-rlshl" Jul 15 23:37:07.549255 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:07.549151 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5efd853d-ff2f-43d9-bbe8-fd8ee02465ec-console-oauth-config\") pod \"console-75d9954bb7-rlshl\" (UID: \"5efd853d-ff2f-43d9-bbe8-fd8ee02465ec\") " pod="openshift-console/console-75d9954bb7-rlshl" Jul 15 23:37:07.650058 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:07.650014 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5efd853d-ff2f-43d9-bbe8-fd8ee02465ec-console-config\") pod \"console-75d9954bb7-rlshl\" (UID: \"5efd853d-ff2f-43d9-bbe8-fd8ee02465ec\") " pod="openshift-console/console-75d9954bb7-rlshl" Jul 15 23:37:07.650241 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:07.650072 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qs7xp\" (UniqueName: \"kubernetes.io/projected/5efd853d-ff2f-43d9-bbe8-fd8ee02465ec-kube-api-access-qs7xp\") pod \"console-75d9954bb7-rlshl\" (UID: \"5efd853d-ff2f-43d9-bbe8-fd8ee02465ec\") " pod="openshift-console/console-75d9954bb7-rlshl" Jul 15 23:37:07.650241 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:07.650143 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5efd853d-ff2f-43d9-bbe8-fd8ee02465ec-console-serving-cert\") pod \"console-75d9954bb7-rlshl\" (UID: \"5efd853d-ff2f-43d9-bbe8-fd8ee02465ec\") " pod="openshift-console/console-75d9954bb7-rlshl" Jul 15 23:37:07.650241 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:07.650162 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5efd853d-ff2f-43d9-bbe8-fd8ee02465ec-service-ca\") pod \"console-75d9954bb7-rlshl\" (UID: \"5efd853d-ff2f-43d9-bbe8-fd8ee02465ec\") " pod="openshift-console/console-75d9954bb7-rlshl" Jul 15 23:37:07.650241 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:07.650205 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5efd853d-ff2f-43d9-bbe8-fd8ee02465ec-oauth-serving-cert\") pod \"console-75d9954bb7-rlshl\" (UID: \"5efd853d-ff2f-43d9-bbe8-fd8ee02465ec\") " pod="openshift-console/console-75d9954bb7-rlshl" Jul 15 23:37:07.650241 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:07.650233 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5efd853d-ff2f-43d9-bbe8-fd8ee02465ec-console-oauth-config\") pod \"console-75d9954bb7-rlshl\" (UID: \"5efd853d-ff2f-43d9-bbe8-fd8ee02465ec\") " pod="openshift-console/console-75d9954bb7-rlshl" Jul 15 23:37:07.650958 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:07.650903 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5efd853d-ff2f-43d9-bbe8-fd8ee02465ec-console-config\") pod \"console-75d9954bb7-rlshl\" (UID: \"5efd853d-ff2f-43d9-bbe8-fd8ee02465ec\") " pod="openshift-console/console-75d9954bb7-rlshl" Jul 15 23:37:07.651124 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:07.651103 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5efd853d-ff2f-43d9-bbe8-fd8ee02465ec-service-ca\") pod \"console-75d9954bb7-rlshl\" (UID: \"5efd853d-ff2f-43d9-bbe8-fd8ee02465ec\") " pod="openshift-console/console-75d9954bb7-rlshl" Jul 15 23:37:07.651232 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:07.651214 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5efd853d-ff2f-43d9-bbe8-fd8ee02465ec-oauth-serving-cert\") pod \"console-75d9954bb7-rlshl\" (UID: \"5efd853d-ff2f-43d9-bbe8-fd8ee02465ec\") " pod="openshift-console/console-75d9954bb7-rlshl" Jul 15 23:37:07.653141 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:07.653120 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5efd853d-ff2f-43d9-bbe8-fd8ee02465ec-console-oauth-config\") pod \"console-75d9954bb7-rlshl\" (UID: \"5efd853d-ff2f-43d9-bbe8-fd8ee02465ec\") " pod="openshift-console/console-75d9954bb7-rlshl" Jul 15 23:37:07.653281 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:07.653262 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5efd853d-ff2f-43d9-bbe8-fd8ee02465ec-console-serving-cert\") pod \"console-75d9954bb7-rlshl\" (UID: \"5efd853d-ff2f-43d9-bbe8-fd8ee02465ec\") " pod="openshift-console/console-75d9954bb7-rlshl" Jul 15 23:37:07.657689 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:07.657668 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qs7xp\" (UniqueName: \"kubernetes.io/projected/5efd853d-ff2f-43d9-bbe8-fd8ee02465ec-kube-api-access-qs7xp\") pod \"console-75d9954bb7-rlshl\" (UID: \"5efd853d-ff2f-43d9-bbe8-fd8ee02465ec\") " pod="openshift-console/console-75d9954bb7-rlshl" Jul 15 23:37:07.660794 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:07.660739 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:37:07.751130 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:07.751096 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-75d9954bb7-rlshl" Jul 15 23:37:07.912528 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:07.912483 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-75d9954bb7-rlshl"] Jul 15 23:37:07.915218 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:37:07.915182 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5efd853d_ff2f_43d9_bbe8_fd8ee02465ec.slice/crio-3c01c2651f37f6ee2e194cd291c4d352173f6c4d363c57db1b9932bbaf21a3d3 WatchSource:0}: Error finding container 3c01c2651f37f6ee2e194cd291c4d352173f6c4d363c57db1b9932bbaf21a3d3: Status 404 returned error can't find the container with id 3c01c2651f37f6ee2e194cd291c4d352173f6c4d363c57db1b9932bbaf21a3d3 Jul 15 23:37:08.054625 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:08.054583 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-64785b6675-gr6rk" event={"ID":"1a03a952-c7c3-4029-b3dc-f5c233499798","Type":"ContainerStarted","Data":"db00526d208d0cc4bc19df0b616f7b051bb05212322896fbf24668d0c2113202"} Jul 15 23:37:08.054783 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:08.054696 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-64785b6675-gr6rk" event={"ID":"1a03a952-c7c3-4029-b3dc-f5c233499798","Type":"ContainerStarted","Data":"ecad1e6afced7759af6153f8eec82a100ae31bd5f4c14655612d7cfe90e0cb25"} Jul 15 23:37:08.055959 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:08.055934 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-75d9954bb7-rlshl" event={"ID":"5efd853d-ff2f-43d9-bbe8-fd8ee02465ec","Type":"ContainerStarted","Data":"3c01c2651f37f6ee2e194cd291c4d352173f6c4d363c57db1b9932bbaf21a3d3"} Jul 15 23:37:08.069688 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:08.069644 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-64785b6675-gr6rk" podStartSLOduration=2.174790948 podStartE2EDuration="4.069630726s" totalImagesPullingTime="1.894839778s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 23:37:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 23:37:08.06879615 +0000 UTC m=+43.442254142" watchObservedRunningTime="2026-07-15 23:37:08.069630726 +0000 UTC m=+43.443088689" Jul 15 23:37:09.263074 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:09.263035 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/409f6854-b231-41aa-8715-a300ee9aa09e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5q8md\" (UID: \"409f6854-b231-41aa-8715-a300ee9aa09e\") " pod="openshift-insights/insights-runtime-extractor-5q8md" Jul 15 23:37:09.263531 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:09.263107 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/409f6854-b231-41aa-8715-a300ee9aa09e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5q8md\" (UID: \"409f6854-b231-41aa-8715-a300ee9aa09e\") " pod="openshift-insights/insights-runtime-extractor-5q8md" Jul 15 23:37:09.263690 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:09.263666 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/409f6854-b231-41aa-8715-a300ee9aa09e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5q8md\" (UID: \"409f6854-b231-41aa-8715-a300ee9aa09e\") " pod="openshift-insights/insights-runtime-extractor-5q8md" Jul 15 23:37:09.266256 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:09.266228 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/409f6854-b231-41aa-8715-a300ee9aa09e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5q8md\" (UID: \"409f6854-b231-41aa-8715-a300ee9aa09e\") " pod="openshift-insights/insights-runtime-extractor-5q8md" Jul 15 23:37:09.357323 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:09.357287 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-5q8md" Jul 15 23:37:09.430386 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:09.430331 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:37:09.536363 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:09.536218 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-5q8md"] Jul 15 23:37:09.539812 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:37:09.539762 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod409f6854_b231_41aa_8715_a300ee9aa09e.slice/crio-44d9bebbc18ede096ded6887a0f681bf220cc3f3299a27354960ad205a35cf92 WatchSource:0}: Error finding container 44d9bebbc18ede096ded6887a0f681bf220cc3f3299a27354960ad205a35cf92: Status 404 returned error can't find the container with id 44d9bebbc18ede096ded6887a0f681bf220cc3f3299a27354960ad205a35cf92 Jul 15 23:37:09.664300 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:09.664268 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:37:09.699338 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:09.699277 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-84bd68b896-8sn4g"] Jul 15 23:37:09.733243 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:09.733215 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-785778ff74-2dsvn"] Jul 15 23:37:09.733381 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:09.733359 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-8sn4g" Jul 15 23:37:09.736004 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:09.735752 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Jul 15 23:37:09.736004 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:09.735756 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-hbg8l\"" Jul 15 23:37:09.736004 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:09.735763 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Jul 15 23:37:09.745226 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:09.745194 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:37:09.747249 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:09.747229 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-84bd68b896-8sn4g"] Jul 15 23:37:09.747344 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:09.747282 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-w5dxc"] Jul 15 23:37:09.747455 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:09.747430 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-785778ff74-2dsvn" Jul 15 23:37:09.749929 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:09.749908 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Jul 15 23:37:09.750138 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:09.749911 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Jul 15 23:37:09.750719 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:09.750551 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-w7dkt\"" Jul 15 23:37:09.750719 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:09.750655 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Jul 15 23:37:09.767318 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:09.767267 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-785778ff74-2dsvn"] Jul 15 23:37:09.767412 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:09.767376 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-w5dxc" Jul 15 23:37:09.769893 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:09.769716 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 15 23:37:09.769893 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:09.769780 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 15 23:37:09.769893 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:09.769807 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-8ccc6\"" Jul 15 23:37:09.769893 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:09.769844 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 15 23:37:09.816248 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:09.816170 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:37:09.864802 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:09.864767 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:37:09.868717 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:09.868688 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/96ec5aef-3317-46d0-be5b-952af4818e44-node-exporter-accelerators-collector-config\") pod \"node-exporter-w5dxc\" (UID: \"96ec5aef-3317-46d0-be5b-952af4818e44\") " pod="openshift-monitoring/node-exporter-w5dxc" Jul 15 23:37:09.868836 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:09.868741 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/dccc9670-e502-402d-b0cd-63d1c54f136f-openshift-state-metrics-tls\") pod \"openshift-state-metrics-84bd68b896-8sn4g\" (UID: \"dccc9670-e502-402d-b0cd-63d1c54f136f\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-8sn4g" Jul 15 23:37:09.868836 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:09.868792 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2d2f\" (UniqueName: \"kubernetes.io/projected/96ec5aef-3317-46d0-be5b-952af4818e44-kube-api-access-f2d2f\") pod \"node-exporter-w5dxc\" (UID: \"96ec5aef-3317-46d0-be5b-952af4818e44\") " pod="openshift-monitoring/node-exporter-w5dxc" Jul 15 23:37:09.868941 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:09.868848 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/96ec5aef-3317-46d0-be5b-952af4818e44-sys\") pod \"node-exporter-w5dxc\" (UID: \"96ec5aef-3317-46d0-be5b-952af4818e44\") " pod="openshift-monitoring/node-exporter-w5dxc" Jul 15 23:37:09.868941 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:09.868878 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f6b39980-f5f7-4709-b768-6cc2b2016cd1-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-785778ff74-2dsvn\" (UID: \"f6b39980-f5f7-4709-b768-6cc2b2016cd1\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-2dsvn" Jul 15 23:37:09.869033 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:09.869013 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f6b39980-f5f7-4709-b768-6cc2b2016cd1-metrics-client-ca\") pod \"kube-state-metrics-785778ff74-2dsvn\" (UID: \"f6b39980-f5f7-4709-b768-6cc2b2016cd1\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-2dsvn" Jul 15 23:37:09.869089 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:09.869043 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/96ec5aef-3317-46d0-be5b-952af4818e44-root\") pod \"node-exporter-w5dxc\" (UID: \"96ec5aef-3317-46d0-be5b-952af4818e44\") " pod="openshift-monitoring/node-exporter-w5dxc" Jul 15 23:37:09.869128 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:09.869093 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/f6b39980-f5f7-4709-b768-6cc2b2016cd1-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-785778ff74-2dsvn\" (UID: \"f6b39980-f5f7-4709-b768-6cc2b2016cd1\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-2dsvn" Jul 15 23:37:09.869175 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:09.869131 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/f6b39980-f5f7-4709-b768-6cc2b2016cd1-volume-directive-shadow\") pod \"kube-state-metrics-785778ff74-2dsvn\" (UID: \"f6b39980-f5f7-4709-b768-6cc2b2016cd1\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-2dsvn" Jul 15 23:37:09.869175 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:09.869159 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/96ec5aef-3317-46d0-be5b-952af4818e44-node-exporter-wtmp\") pod \"node-exporter-w5dxc\" (UID: \"96ec5aef-3317-46d0-be5b-952af4818e44\") " pod="openshift-monitoring/node-exporter-w5dxc" Jul 15 23:37:09.869262 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:09.869182 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79g52\" (UniqueName: \"kubernetes.io/projected/f6b39980-f5f7-4709-b768-6cc2b2016cd1-kube-api-access-79g52\") pod \"kube-state-metrics-785778ff74-2dsvn\" (UID: \"f6b39980-f5f7-4709-b768-6cc2b2016cd1\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-2dsvn" Jul 15 23:37:09.869262 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:09.869205 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/96ec5aef-3317-46d0-be5b-952af4818e44-node-exporter-textfile\") pod \"node-exporter-w5dxc\" (UID: \"96ec5aef-3317-46d0-be5b-952af4818e44\") " pod="openshift-monitoring/node-exporter-w5dxc" Jul 15 23:37:09.869262 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:09.869231 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/96ec5aef-3317-46d0-be5b-952af4818e44-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-w5dxc\" (UID: \"96ec5aef-3317-46d0-be5b-952af4818e44\") " pod="openshift-monitoring/node-exporter-w5dxc" Jul 15 23:37:09.869262 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:09.869257 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/96ec5aef-3317-46d0-be5b-952af4818e44-metrics-client-ca\") pod \"node-exporter-w5dxc\" (UID: \"96ec5aef-3317-46d0-be5b-952af4818e44\") " pod="openshift-monitoring/node-exporter-w5dxc" Jul 15 23:37:09.869457 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:09.869282 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dccc9670-e502-402d-b0cd-63d1c54f136f-metrics-client-ca\") pod \"openshift-state-metrics-84bd68b896-8sn4g\" (UID: \"dccc9670-e502-402d-b0cd-63d1c54f136f\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-8sn4g" Jul 15 23:37:09.869457 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:09.869304 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4h92d\" (UniqueName: \"kubernetes.io/projected/dccc9670-e502-402d-b0cd-63d1c54f136f-kube-api-access-4h92d\") pod \"openshift-state-metrics-84bd68b896-8sn4g\" (UID: \"dccc9670-e502-402d-b0cd-63d1c54f136f\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-8sn4g" Jul 15 23:37:09.869457 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:09.869356 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/96ec5aef-3317-46d0-be5b-952af4818e44-node-exporter-tls\") pod \"node-exporter-w5dxc\" (UID: \"96ec5aef-3317-46d0-be5b-952af4818e44\") " pod="openshift-monitoring/node-exporter-w5dxc" Jul 15 23:37:09.869457 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:09.869383 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/f6b39980-f5f7-4709-b768-6cc2b2016cd1-kube-state-metrics-tls\") pod \"kube-state-metrics-785778ff74-2dsvn\" (UID: \"f6b39980-f5f7-4709-b768-6cc2b2016cd1\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-2dsvn" Jul 15 23:37:09.869457 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:09.869433 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/dccc9670-e502-402d-b0cd-63d1c54f136f-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-84bd68b896-8sn4g\" (UID: \"dccc9670-e502-402d-b0cd-63d1c54f136f\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-8sn4g" Jul 15 23:37:09.928496 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:09.928456 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:37:09.970757 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:09.970723 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dccc9670-e502-402d-b0cd-63d1c54f136f-metrics-client-ca\") pod \"openshift-state-metrics-84bd68b896-8sn4g\" (UID: \"dccc9670-e502-402d-b0cd-63d1c54f136f\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-8sn4g" Jul 15 23:37:09.970924 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:09.970771 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4h92d\" (UniqueName: \"kubernetes.io/projected/dccc9670-e502-402d-b0cd-63d1c54f136f-kube-api-access-4h92d\") pod \"openshift-state-metrics-84bd68b896-8sn4g\" (UID: \"dccc9670-e502-402d-b0cd-63d1c54f136f\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-8sn4g" Jul 15 23:37:09.970924 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:09.970808 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/96ec5aef-3317-46d0-be5b-952af4818e44-node-exporter-tls\") pod \"node-exporter-w5dxc\" (UID: \"96ec5aef-3317-46d0-be5b-952af4818e44\") " pod="openshift-monitoring/node-exporter-w5dxc" Jul 15 23:37:09.970924 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:09.970837 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/f6b39980-f5f7-4709-b768-6cc2b2016cd1-kube-state-metrics-tls\") pod \"kube-state-metrics-785778ff74-2dsvn\" (UID: \"f6b39980-f5f7-4709-b768-6cc2b2016cd1\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-2dsvn" Jul 15 23:37:09.970924 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:09.970870 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/dccc9670-e502-402d-b0cd-63d1c54f136f-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-84bd68b896-8sn4g\" (UID: \"dccc9670-e502-402d-b0cd-63d1c54f136f\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-8sn4g" Jul 15 23:37:09.970924 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:09.970905 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/96ec5aef-3317-46d0-be5b-952af4818e44-node-exporter-accelerators-collector-config\") pod \"node-exporter-w5dxc\" (UID: \"96ec5aef-3317-46d0-be5b-952af4818e44\") " pod="openshift-monitoring/node-exporter-w5dxc" Jul 15 23:37:09.971141 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:09.970951 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/dccc9670-e502-402d-b0cd-63d1c54f136f-openshift-state-metrics-tls\") pod \"openshift-state-metrics-84bd68b896-8sn4g\" (UID: \"dccc9670-e502-402d-b0cd-63d1c54f136f\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-8sn4g" Jul 15 23:37:09.971141 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:09.970994 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2d2f\" (UniqueName: \"kubernetes.io/projected/96ec5aef-3317-46d0-be5b-952af4818e44-kube-api-access-f2d2f\") pod \"node-exporter-w5dxc\" (UID: \"96ec5aef-3317-46d0-be5b-952af4818e44\") " pod="openshift-monitoring/node-exporter-w5dxc" Jul 15 23:37:09.971141 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:09.971032 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/96ec5aef-3317-46d0-be5b-952af4818e44-sys\") pod \"node-exporter-w5dxc\" (UID: \"96ec5aef-3317-46d0-be5b-952af4818e44\") " pod="openshift-monitoring/node-exporter-w5dxc" Jul 15 23:37:09.971141 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:09.971059 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f6b39980-f5f7-4709-b768-6cc2b2016cd1-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-785778ff74-2dsvn\" (UID: \"f6b39980-f5f7-4709-b768-6cc2b2016cd1\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-2dsvn" Jul 15 23:37:09.971141 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:09.971097 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f6b39980-f5f7-4709-b768-6cc2b2016cd1-metrics-client-ca\") pod \"kube-state-metrics-785778ff74-2dsvn\" (UID: \"f6b39980-f5f7-4709-b768-6cc2b2016cd1\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-2dsvn" Jul 15 23:37:09.971141 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:09.971118 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/96ec5aef-3317-46d0-be5b-952af4818e44-root\") pod \"node-exporter-w5dxc\" (UID: \"96ec5aef-3317-46d0-be5b-952af4818e44\") " pod="openshift-monitoring/node-exporter-w5dxc" Jul 15 23:37:09.971340 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:09.971165 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/f6b39980-f5f7-4709-b768-6cc2b2016cd1-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-785778ff74-2dsvn\" (UID: \"f6b39980-f5f7-4709-b768-6cc2b2016cd1\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-2dsvn" Jul 15 23:37:09.971340 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:09.971195 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/f6b39980-f5f7-4709-b768-6cc2b2016cd1-volume-directive-shadow\") pod \"kube-state-metrics-785778ff74-2dsvn\" (UID: \"f6b39980-f5f7-4709-b768-6cc2b2016cd1\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-2dsvn" Jul 15 23:37:09.971340 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:09.971240 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/96ec5aef-3317-46d0-be5b-952af4818e44-node-exporter-wtmp\") pod \"node-exporter-w5dxc\" (UID: \"96ec5aef-3317-46d0-be5b-952af4818e44\") " pod="openshift-monitoring/node-exporter-w5dxc" Jul 15 23:37:09.971340 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:09.971265 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79g52\" (UniqueName: \"kubernetes.io/projected/f6b39980-f5f7-4709-b768-6cc2b2016cd1-kube-api-access-79g52\") pod \"kube-state-metrics-785778ff74-2dsvn\" (UID: \"f6b39980-f5f7-4709-b768-6cc2b2016cd1\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-2dsvn" Jul 15 23:37:09.971340 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:09.971289 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/96ec5aef-3317-46d0-be5b-952af4818e44-node-exporter-textfile\") pod \"node-exporter-w5dxc\" (UID: \"96ec5aef-3317-46d0-be5b-952af4818e44\") " pod="openshift-monitoring/node-exporter-w5dxc" Jul 15 23:37:09.971340 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:09.971317 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/96ec5aef-3317-46d0-be5b-952af4818e44-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-w5dxc\" (UID: \"96ec5aef-3317-46d0-be5b-952af4818e44\") " pod="openshift-monitoring/node-exporter-w5dxc" Jul 15 23:37:09.971604 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:09.971348 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/96ec5aef-3317-46d0-be5b-952af4818e44-metrics-client-ca\") pod \"node-exporter-w5dxc\" (UID: \"96ec5aef-3317-46d0-be5b-952af4818e44\") " pod="openshift-monitoring/node-exporter-w5dxc" Jul 15 23:37:09.971604 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:09.971491 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dccc9670-e502-402d-b0cd-63d1c54f136f-metrics-client-ca\") pod \"openshift-state-metrics-84bd68b896-8sn4g\" (UID: \"dccc9670-e502-402d-b0cd-63d1c54f136f\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-8sn4g" Jul 15 23:37:09.971985 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:09.971940 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/96ec5aef-3317-46d0-be5b-952af4818e44-metrics-client-ca\") pod \"node-exporter-w5dxc\" (UID: \"96ec5aef-3317-46d0-be5b-952af4818e44\") " pod="openshift-monitoring/node-exporter-w5dxc" Jul 15 23:37:09.972060 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:09.972041 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/f6b39980-f5f7-4709-b768-6cc2b2016cd1-volume-directive-shadow\") pod \"kube-state-metrics-785778ff74-2dsvn\" (UID: \"f6b39980-f5f7-4709-b768-6cc2b2016cd1\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-2dsvn" Jul 15 23:37:09.972195 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:09.972175 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/96ec5aef-3317-46d0-be5b-952af4818e44-node-exporter-wtmp\") pod \"node-exporter-w5dxc\" (UID: \"96ec5aef-3317-46d0-be5b-952af4818e44\") " pod="openshift-monitoring/node-exporter-w5dxc" Jul 15 23:37:09.972256 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:09.972190 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/f6b39980-f5f7-4709-b768-6cc2b2016cd1-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-785778ff74-2dsvn\" (UID: \"f6b39980-f5f7-4709-b768-6cc2b2016cd1\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-2dsvn" Jul 15 23:37:09.972466 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:09.972448 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/96ec5aef-3317-46d0-be5b-952af4818e44-sys\") pod \"node-exporter-w5dxc\" (UID: \"96ec5aef-3317-46d0-be5b-952af4818e44\") " pod="openshift-monitoring/node-exporter-w5dxc" Jul 15 23:37:09.974381 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:09.973104 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f6b39980-f5f7-4709-b768-6cc2b2016cd1-metrics-client-ca\") pod \"kube-state-metrics-785778ff74-2dsvn\" (UID: \"f6b39980-f5f7-4709-b768-6cc2b2016cd1\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-2dsvn" Jul 15 23:37:09.974381 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:09.972182 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/96ec5aef-3317-46d0-be5b-952af4818e44-node-exporter-textfile\") pod \"node-exporter-w5dxc\" (UID: \"96ec5aef-3317-46d0-be5b-952af4818e44\") " pod="openshift-monitoring/node-exporter-w5dxc" Jul 15 23:37:09.974381 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:09.973614 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/96ec5aef-3317-46d0-be5b-952af4818e44-root\") pod \"node-exporter-w5dxc\" (UID: \"96ec5aef-3317-46d0-be5b-952af4818e44\") " pod="openshift-monitoring/node-exporter-w5dxc" Jul 15 23:37:09.974381 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:09.974338 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/96ec5aef-3317-46d0-be5b-952af4818e44-node-exporter-accelerators-collector-config\") pod \"node-exporter-w5dxc\" (UID: \"96ec5aef-3317-46d0-be5b-952af4818e44\") " pod="openshift-monitoring/node-exporter-w5dxc" Jul 15 23:37:09.974743 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:09.974717 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/96ec5aef-3317-46d0-be5b-952af4818e44-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-w5dxc\" (UID: \"96ec5aef-3317-46d0-be5b-952af4818e44\") " pod="openshift-monitoring/node-exporter-w5dxc" Jul 15 23:37:09.974819 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:09.974742 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/dccc9670-e502-402d-b0cd-63d1c54f136f-openshift-state-metrics-tls\") pod \"openshift-state-metrics-84bd68b896-8sn4g\" (UID: \"dccc9670-e502-402d-b0cd-63d1c54f136f\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-8sn4g" Jul 15 23:37:09.975297 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:09.975271 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/f6b39980-f5f7-4709-b768-6cc2b2016cd1-kube-state-metrics-tls\") pod \"kube-state-metrics-785778ff74-2dsvn\" (UID: \"f6b39980-f5f7-4709-b768-6cc2b2016cd1\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-2dsvn" Jul 15 23:37:09.975858 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:09.975814 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f6b39980-f5f7-4709-b768-6cc2b2016cd1-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-785778ff74-2dsvn\" (UID: \"f6b39980-f5f7-4709-b768-6cc2b2016cd1\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-2dsvn" Jul 15 23:37:09.976569 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:09.976517 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/96ec5aef-3317-46d0-be5b-952af4818e44-node-exporter-tls\") pod \"node-exporter-w5dxc\" (UID: \"96ec5aef-3317-46d0-be5b-952af4818e44\") " pod="openshift-monitoring/node-exporter-w5dxc" Jul 15 23:37:09.978835 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:09.978778 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-79g52\" (UniqueName: \"kubernetes.io/projected/f6b39980-f5f7-4709-b768-6cc2b2016cd1-kube-api-access-79g52\") pod \"kube-state-metrics-785778ff74-2dsvn\" (UID: \"f6b39980-f5f7-4709-b768-6cc2b2016cd1\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-2dsvn" Jul 15 23:37:09.979619 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:09.979028 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/dccc9670-e502-402d-b0cd-63d1c54f136f-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-84bd68b896-8sn4g\" (UID: \"dccc9670-e502-402d-b0cd-63d1c54f136f\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-8sn4g" Jul 15 23:37:09.981149 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:09.980554 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4h92d\" (UniqueName: \"kubernetes.io/projected/dccc9670-e502-402d-b0cd-63d1c54f136f-kube-api-access-4h92d\") pod \"openshift-state-metrics-84bd68b896-8sn4g\" (UID: \"dccc9670-e502-402d-b0cd-63d1c54f136f\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-8sn4g" Jul 15 23:37:09.981397 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:09.981339 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2d2f\" (UniqueName: \"kubernetes.io/projected/96ec5aef-3317-46d0-be5b-952af4818e44-kube-api-access-f2d2f\") pod \"node-exporter-w5dxc\" (UID: \"96ec5aef-3317-46d0-be5b-952af4818e44\") " pod="openshift-monitoring/node-exporter-w5dxc" Jul 15 23:37:10.045327 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:10.045296 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-8sn4g" Jul 15 23:37:10.058745 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:10.058556 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-785778ff74-2dsvn" Jul 15 23:37:10.059595 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:10.059532 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:37:10.065454 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:10.065425 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5q8md" event={"ID":"409f6854-b231-41aa-8715-a300ee9aa09e","Type":"ContainerStarted","Data":"44d9bebbc18ede096ded6887a0f681bf220cc3f3299a27354960ad205a35cf92"} Jul 15 23:37:10.079299 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:10.079275 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-w5dxc" Jul 15 23:37:10.165351 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:10.164671 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:37:10.261131 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:10.261065 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:37:10.653883 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:10.653805 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:37:10.744269 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:10.744206 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:37:10.745532 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:10.745505 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 15 23:37:10.771068 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:10.771037 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 15 23:37:10.771764 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:10.771277 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 15 23:37:10.774794 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:10.774146 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 15 23:37:10.774794 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:10.774208 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 15 23:37:10.774794 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:10.774433 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 15 23:37:10.774794 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:10.774577 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 15 23:37:10.774794 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:10.774617 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-6xzdv\"" Jul 15 23:37:10.775737 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:10.775422 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 15 23:37:10.775737 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:10.775480 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 15 23:37:10.775737 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:10.775659 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 15 23:37:10.775737 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:10.775712 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 15 23:37:10.776019 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:10.775842 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 15 23:37:10.879933 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:10.879275 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/5e3e5aa3-8bc5-49ff-8706-6107d3e3d2bb-config-volume\") pod \"alertmanager-main-0\" (UID: \"5e3e5aa3-8bc5-49ff-8706-6107d3e3d2bb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 23:37:10.879933 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:10.879320 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/5e3e5aa3-8bc5-49ff-8706-6107d3e3d2bb-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"5e3e5aa3-8bc5-49ff-8706-6107d3e3d2bb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 23:37:10.879933 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:10.879354 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/5e3e5aa3-8bc5-49ff-8706-6107d3e3d2bb-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"5e3e5aa3-8bc5-49ff-8706-6107d3e3d2bb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 23:37:10.879933 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:10.879393 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5e3e5aa3-8bc5-49ff-8706-6107d3e3d2bb-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"5e3e5aa3-8bc5-49ff-8706-6107d3e3d2bb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 23:37:10.879933 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:10.879417 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/5e3e5aa3-8bc5-49ff-8706-6107d3e3d2bb-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"5e3e5aa3-8bc5-49ff-8706-6107d3e3d2bb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 23:37:10.879933 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:10.879462 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5e3e5aa3-8bc5-49ff-8706-6107d3e3d2bb-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"5e3e5aa3-8bc5-49ff-8706-6107d3e3d2bb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 23:37:10.879933 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:10.879500 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwzrf\" (UniqueName: \"kubernetes.io/projected/5e3e5aa3-8bc5-49ff-8706-6107d3e3d2bb-kube-api-access-cwzrf\") pod \"alertmanager-main-0\" (UID: \"5e3e5aa3-8bc5-49ff-8706-6107d3e3d2bb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 23:37:10.879933 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:10.879531 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5e3e5aa3-8bc5-49ff-8706-6107d3e3d2bb-config-out\") pod \"alertmanager-main-0\" (UID: \"5e3e5aa3-8bc5-49ff-8706-6107d3e3d2bb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 23:37:10.879933 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:10.879552 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/5e3e5aa3-8bc5-49ff-8706-6107d3e3d2bb-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"5e3e5aa3-8bc5-49ff-8706-6107d3e3d2bb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 23:37:10.879933 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:10.879575 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5e3e5aa3-8bc5-49ff-8706-6107d3e3d2bb-web-config\") pod \"alertmanager-main-0\" (UID: \"5e3e5aa3-8bc5-49ff-8706-6107d3e3d2bb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 23:37:10.879933 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:10.879638 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5e3e5aa3-8bc5-49ff-8706-6107d3e3d2bb-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"5e3e5aa3-8bc5-49ff-8706-6107d3e3d2bb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 23:37:10.879933 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:10.879696 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5e3e5aa3-8bc5-49ff-8706-6107d3e3d2bb-tls-assets\") pod \"alertmanager-main-0\" (UID: \"5e3e5aa3-8bc5-49ff-8706-6107d3e3d2bb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 23:37:10.879933 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:10.879720 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5e3e5aa3-8bc5-49ff-8706-6107d3e3d2bb-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"5e3e5aa3-8bc5-49ff-8706-6107d3e3d2bb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 23:37:10.981374 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:10.981056 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5e3e5aa3-8bc5-49ff-8706-6107d3e3d2bb-tls-assets\") pod \"alertmanager-main-0\" (UID: \"5e3e5aa3-8bc5-49ff-8706-6107d3e3d2bb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 23:37:10.981374 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:10.981104 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5e3e5aa3-8bc5-49ff-8706-6107d3e3d2bb-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"5e3e5aa3-8bc5-49ff-8706-6107d3e3d2bb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 23:37:10.981374 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:10.981137 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/5e3e5aa3-8bc5-49ff-8706-6107d3e3d2bb-config-volume\") pod \"alertmanager-main-0\" (UID: \"5e3e5aa3-8bc5-49ff-8706-6107d3e3d2bb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 23:37:10.981374 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:10.981162 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/5e3e5aa3-8bc5-49ff-8706-6107d3e3d2bb-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"5e3e5aa3-8bc5-49ff-8706-6107d3e3d2bb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 23:37:10.981374 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:10.981192 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/5e3e5aa3-8bc5-49ff-8706-6107d3e3d2bb-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"5e3e5aa3-8bc5-49ff-8706-6107d3e3d2bb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 23:37:10.981374 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:10.981228 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5e3e5aa3-8bc5-49ff-8706-6107d3e3d2bb-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"5e3e5aa3-8bc5-49ff-8706-6107d3e3d2bb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 23:37:10.981374 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:10.981255 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/5e3e5aa3-8bc5-49ff-8706-6107d3e3d2bb-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"5e3e5aa3-8bc5-49ff-8706-6107d3e3d2bb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 23:37:10.981374 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:10.981296 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5e3e5aa3-8bc5-49ff-8706-6107d3e3d2bb-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"5e3e5aa3-8bc5-49ff-8706-6107d3e3d2bb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 23:37:10.983189 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:10.981339 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwzrf\" (UniqueName: \"kubernetes.io/projected/5e3e5aa3-8bc5-49ff-8706-6107d3e3d2bb-kube-api-access-cwzrf\") pod \"alertmanager-main-0\" (UID: \"5e3e5aa3-8bc5-49ff-8706-6107d3e3d2bb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 23:37:10.983189 ip-10-0-128-197 kubenswrapper[2563]: E0715 23:37:10.981437 2563 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found Jul 15 23:37:10.983189 ip-10-0-128-197 kubenswrapper[2563]: E0715 23:37:10.981896 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5e3e5aa3-8bc5-49ff-8706-6107d3e3d2bb-alertmanager-trusted-ca-bundle podName:5e3e5aa3-8bc5-49ff-8706-6107d3e3d2bb nodeName:}" failed. No retries permitted until 2026-07-15 23:37:11.481872514 +0000 UTC m=+46.855330476 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "alertmanager-trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/5e3e5aa3-8bc5-49ff-8706-6107d3e3d2bb-alertmanager-trusted-ca-bundle") pod "alertmanager-main-0" (UID: "5e3e5aa3-8bc5-49ff-8706-6107d3e3d2bb") : configmap references non-existent config key: ca-bundle.crt Jul 15 23:37:10.983189 ip-10-0-128-197 kubenswrapper[2563]: E0715 23:37:10.982041 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e3e5aa3-8bc5-49ff-8706-6107d3e3d2bb-secret-alertmanager-main-tls podName:5e3e5aa3-8bc5-49ff-8706-6107d3e3d2bb nodeName:}" failed. No retries permitted until 2026-07-15 23:37:11.482020701 +0000 UTC m=+46.855478647 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/5e3e5aa3-8bc5-49ff-8706-6107d3e3d2bb-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "5e3e5aa3-8bc5-49ff-8706-6107d3e3d2bb") : secret "alertmanager-main-tls" not found Jul 15 23:37:10.983189 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:10.982084 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5e3e5aa3-8bc5-49ff-8706-6107d3e3d2bb-config-out\") pod \"alertmanager-main-0\" (UID: \"5e3e5aa3-8bc5-49ff-8706-6107d3e3d2bb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 23:37:10.983189 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:10.982110 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/5e3e5aa3-8bc5-49ff-8706-6107d3e3d2bb-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"5e3e5aa3-8bc5-49ff-8706-6107d3e3d2bb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 23:37:10.983189 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:10.982137 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5e3e5aa3-8bc5-49ff-8706-6107d3e3d2bb-web-config\") pod \"alertmanager-main-0\" (UID: \"5e3e5aa3-8bc5-49ff-8706-6107d3e3d2bb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 23:37:10.983189 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:10.982187 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/5e3e5aa3-8bc5-49ff-8706-6107d3e3d2bb-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"5e3e5aa3-8bc5-49ff-8706-6107d3e3d2bb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 23:37:10.983189 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:10.982205 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5e3e5aa3-8bc5-49ff-8706-6107d3e3d2bb-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"5e3e5aa3-8bc5-49ff-8706-6107d3e3d2bb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 23:37:10.983189 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:10.982871 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5e3e5aa3-8bc5-49ff-8706-6107d3e3d2bb-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"5e3e5aa3-8bc5-49ff-8706-6107d3e3d2bb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 23:37:10.985922 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:10.985323 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5e3e5aa3-8bc5-49ff-8706-6107d3e3d2bb-tls-assets\") pod \"alertmanager-main-0\" (UID: \"5e3e5aa3-8bc5-49ff-8706-6107d3e3d2bb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 23:37:10.986289 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:10.986242 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/5e3e5aa3-8bc5-49ff-8706-6107d3e3d2bb-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"5e3e5aa3-8bc5-49ff-8706-6107d3e3d2bb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 23:37:10.986713 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:10.986663 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/5e3e5aa3-8bc5-49ff-8706-6107d3e3d2bb-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"5e3e5aa3-8bc5-49ff-8706-6107d3e3d2bb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 23:37:10.987375 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:10.987330 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/5e3e5aa3-8bc5-49ff-8706-6107d3e3d2bb-config-volume\") pod \"alertmanager-main-0\" (UID: \"5e3e5aa3-8bc5-49ff-8706-6107d3e3d2bb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 23:37:10.988302 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:10.987946 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5e3e5aa3-8bc5-49ff-8706-6107d3e3d2bb-web-config\") pod \"alertmanager-main-0\" (UID: \"5e3e5aa3-8bc5-49ff-8706-6107d3e3d2bb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 23:37:10.988302 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:10.988177 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5e3e5aa3-8bc5-49ff-8706-6107d3e3d2bb-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"5e3e5aa3-8bc5-49ff-8706-6107d3e3d2bb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 23:37:10.991653 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:10.991204 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5e3e5aa3-8bc5-49ff-8706-6107d3e3d2bb-config-out\") pod \"alertmanager-main-0\" (UID: \"5e3e5aa3-8bc5-49ff-8706-6107d3e3d2bb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 23:37:10.992260 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:10.992238 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5e3e5aa3-8bc5-49ff-8706-6107d3e3d2bb-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"5e3e5aa3-8bc5-49ff-8706-6107d3e3d2bb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 23:37:10.994475 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:10.994428 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwzrf\" (UniqueName: \"kubernetes.io/projected/5e3e5aa3-8bc5-49ff-8706-6107d3e3d2bb-kube-api-access-cwzrf\") pod \"alertmanager-main-0\" (UID: \"5e3e5aa3-8bc5-49ff-8706-6107d3e3d2bb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 23:37:11.027519 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:11.027493 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:37:11.488819 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:11.488291 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/5e3e5aa3-8bc5-49ff-8706-6107d3e3d2bb-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"5e3e5aa3-8bc5-49ff-8706-6107d3e3d2bb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 23:37:11.488819 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:11.488427 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5e3e5aa3-8bc5-49ff-8706-6107d3e3d2bb-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"5e3e5aa3-8bc5-49ff-8706-6107d3e3d2bb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 23:37:11.490553 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:11.490527 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5e3e5aa3-8bc5-49ff-8706-6107d3e3d2bb-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"5e3e5aa3-8bc5-49ff-8706-6107d3e3d2bb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 23:37:11.494694 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:11.494645 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/5e3e5aa3-8bc5-49ff-8706-6107d3e3d2bb-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"5e3e5aa3-8bc5-49ff-8706-6107d3e3d2bb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 23:37:11.506682 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:11.506575 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-785778ff74-2dsvn"] Jul 15 23:37:11.509728 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:37:11.509698 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6b39980_f5f7_4709_b768_6cc2b2016cd1.slice/crio-5d10f4d3da12e432d1727afc3b5fdfff7ddf5237903ebefc5f822217575a5137 WatchSource:0}: Error finding container 5d10f4d3da12e432d1727afc3b5fdfff7ddf5237903ebefc5f822217575a5137: Status 404 returned error can't find the container with id 5d10f4d3da12e432d1727afc3b5fdfff7ddf5237903ebefc5f822217575a5137 Jul 15 23:37:11.521605 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:11.521580 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-84bd68b896-8sn4g"] Jul 15 23:37:11.655660 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:11.655587 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-b7fc86bbb-vrm2p"] Jul 15 23:37:11.679049 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:11.679020 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-b7fc86bbb-vrm2p"] Jul 15 23:37:11.679178 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:11.679093 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-b7fc86bbb-vrm2p" Jul 15 23:37:11.682763 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:11.682738 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-alevlgkrdg1o6\"" Jul 15 23:37:11.682893 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:11.682874 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Jul 15 23:37:11.682989 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:11.682881 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Jul 15 23:37:11.683188 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:11.683168 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Jul 15 23:37:11.683330 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:11.683287 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-6dgfz\"" Jul 15 23:37:11.683330 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:11.683331 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Jul 15 23:37:11.683581 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:11.683561 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Jul 15 23:37:11.684835 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:11.684816 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 15 23:37:11.690836 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:37:11.690811 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddccc9670_e502_402d_b0cd_63d1c54f136f.slice/crio-c80fb6647fb9e635655530797fe5bb3e6289522ea23e4a61d61e64d8090560e2 WatchSource:0}: Error finding container c80fb6647fb9e635655530797fe5bb3e6289522ea23e4a61d61e64d8090560e2: Status 404 returned error can't find the container with id c80fb6647fb9e635655530797fe5bb3e6289522ea23e4a61d61e64d8090560e2 Jul 15 23:37:11.791317 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:11.791295 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/cc051ada-7c71-4701-94cf-914ad0e6c65c-secret-thanos-querier-tls\") pod \"thanos-querier-b7fc86bbb-vrm2p\" (UID: \"cc051ada-7c71-4701-94cf-914ad0e6c65c\") " pod="openshift-monitoring/thanos-querier-b7fc86bbb-vrm2p" Jul 15 23:37:11.791437 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:11.791386 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/cc051ada-7c71-4701-94cf-914ad0e6c65c-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-b7fc86bbb-vrm2p\" (UID: \"cc051ada-7c71-4701-94cf-914ad0e6c65c\") " pod="openshift-monitoring/thanos-querier-b7fc86bbb-vrm2p" Jul 15 23:37:11.791495 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:11.791434 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/cc051ada-7c71-4701-94cf-914ad0e6c65c-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-b7fc86bbb-vrm2p\" (UID: \"cc051ada-7c71-4701-94cf-914ad0e6c65c\") " pod="openshift-monitoring/thanos-querier-b7fc86bbb-vrm2p" Jul 15 23:37:11.791495 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:11.791472 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cc051ada-7c71-4701-94cf-914ad0e6c65c-metrics-client-ca\") pod \"thanos-querier-b7fc86bbb-vrm2p\" (UID: \"cc051ada-7c71-4701-94cf-914ad0e6c65c\") " pod="openshift-monitoring/thanos-querier-b7fc86bbb-vrm2p" Jul 15 23:37:11.791600 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:11.791515 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lgsps\" (UniqueName: \"kubernetes.io/projected/cc051ada-7c71-4701-94cf-914ad0e6c65c-kube-api-access-lgsps\") pod \"thanos-querier-b7fc86bbb-vrm2p\" (UID: \"cc051ada-7c71-4701-94cf-914ad0e6c65c\") " pod="openshift-monitoring/thanos-querier-b7fc86bbb-vrm2p" Jul 15 23:37:11.791600 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:11.791551 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/cc051ada-7c71-4701-94cf-914ad0e6c65c-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-b7fc86bbb-vrm2p\" (UID: \"cc051ada-7c71-4701-94cf-914ad0e6c65c\") " pod="openshift-monitoring/thanos-querier-b7fc86bbb-vrm2p" Jul 15 23:37:11.791600 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:11.791578 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/cc051ada-7c71-4701-94cf-914ad0e6c65c-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-b7fc86bbb-vrm2p\" (UID: \"cc051ada-7c71-4701-94cf-914ad0e6c65c\") " pod="openshift-monitoring/thanos-querier-b7fc86bbb-vrm2p" Jul 15 23:37:11.791738 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:11.791603 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/cc051ada-7c71-4701-94cf-914ad0e6c65c-secret-grpc-tls\") pod \"thanos-querier-b7fc86bbb-vrm2p\" (UID: \"cc051ada-7c71-4701-94cf-914ad0e6c65c\") " pod="openshift-monitoring/thanos-querier-b7fc86bbb-vrm2p" Jul 15 23:37:11.876182 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:11.876052 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 15 23:37:11.879151 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:37:11.879123 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e3e5aa3_8bc5_49ff_8706_6107d3e3d2bb.slice/crio-5cf03839cfe4212e881a3fe9ca9f7b3f27ac1cc32d685dbc3db8649f98d0e9f4 WatchSource:0}: Error finding container 5cf03839cfe4212e881a3fe9ca9f7b3f27ac1cc32d685dbc3db8649f98d0e9f4: Status 404 returned error can't find the container with id 5cf03839cfe4212e881a3fe9ca9f7b3f27ac1cc32d685dbc3db8649f98d0e9f4 Jul 15 23:37:11.892341 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:11.892255 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/cc051ada-7c71-4701-94cf-914ad0e6c65c-secret-thanos-querier-tls\") pod \"thanos-querier-b7fc86bbb-vrm2p\" (UID: \"cc051ada-7c71-4701-94cf-914ad0e6c65c\") " pod="openshift-monitoring/thanos-querier-b7fc86bbb-vrm2p" Jul 15 23:37:11.892434 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:11.892338 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/cc051ada-7c71-4701-94cf-914ad0e6c65c-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-b7fc86bbb-vrm2p\" (UID: \"cc051ada-7c71-4701-94cf-914ad0e6c65c\") " pod="openshift-monitoring/thanos-querier-b7fc86bbb-vrm2p" Jul 15 23:37:11.892434 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:11.892384 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/cc051ada-7c71-4701-94cf-914ad0e6c65c-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-b7fc86bbb-vrm2p\" (UID: \"cc051ada-7c71-4701-94cf-914ad0e6c65c\") " pod="openshift-monitoring/thanos-querier-b7fc86bbb-vrm2p" Jul 15 23:37:11.892434 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:11.892430 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cc051ada-7c71-4701-94cf-914ad0e6c65c-metrics-client-ca\") pod \"thanos-querier-b7fc86bbb-vrm2p\" (UID: \"cc051ada-7c71-4701-94cf-914ad0e6c65c\") " pod="openshift-monitoring/thanos-querier-b7fc86bbb-vrm2p" Jul 15 23:37:11.892635 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:11.892474 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lgsps\" (UniqueName: \"kubernetes.io/projected/cc051ada-7c71-4701-94cf-914ad0e6c65c-kube-api-access-lgsps\") pod \"thanos-querier-b7fc86bbb-vrm2p\" (UID: \"cc051ada-7c71-4701-94cf-914ad0e6c65c\") " pod="openshift-monitoring/thanos-querier-b7fc86bbb-vrm2p" Jul 15 23:37:11.892635 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:11.892509 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/cc051ada-7c71-4701-94cf-914ad0e6c65c-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-b7fc86bbb-vrm2p\" (UID: \"cc051ada-7c71-4701-94cf-914ad0e6c65c\") " pod="openshift-monitoring/thanos-querier-b7fc86bbb-vrm2p" Jul 15 23:37:11.892635 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:11.892539 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/cc051ada-7c71-4701-94cf-914ad0e6c65c-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-b7fc86bbb-vrm2p\" (UID: \"cc051ada-7c71-4701-94cf-914ad0e6c65c\") " pod="openshift-monitoring/thanos-querier-b7fc86bbb-vrm2p" Jul 15 23:37:11.892635 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:11.892565 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/cc051ada-7c71-4701-94cf-914ad0e6c65c-secret-grpc-tls\") pod \"thanos-querier-b7fc86bbb-vrm2p\" (UID: \"cc051ada-7c71-4701-94cf-914ad0e6c65c\") " pod="openshift-monitoring/thanos-querier-b7fc86bbb-vrm2p" Jul 15 23:37:11.893312 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:11.893284 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cc051ada-7c71-4701-94cf-914ad0e6c65c-metrics-client-ca\") pod \"thanos-querier-b7fc86bbb-vrm2p\" (UID: \"cc051ada-7c71-4701-94cf-914ad0e6c65c\") " pod="openshift-monitoring/thanos-querier-b7fc86bbb-vrm2p" Jul 15 23:37:11.895776 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:11.895729 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/cc051ada-7c71-4701-94cf-914ad0e6c65c-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-b7fc86bbb-vrm2p\" (UID: \"cc051ada-7c71-4701-94cf-914ad0e6c65c\") " pod="openshift-monitoring/thanos-querier-b7fc86bbb-vrm2p" Jul 15 23:37:11.895870 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:11.895815 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/cc051ada-7c71-4701-94cf-914ad0e6c65c-secret-grpc-tls\") pod \"thanos-querier-b7fc86bbb-vrm2p\" (UID: \"cc051ada-7c71-4701-94cf-914ad0e6c65c\") " pod="openshift-monitoring/thanos-querier-b7fc86bbb-vrm2p" Jul 15 23:37:11.895948 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:11.895923 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/cc051ada-7c71-4701-94cf-914ad0e6c65c-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-b7fc86bbb-vrm2p\" (UID: \"cc051ada-7c71-4701-94cf-914ad0e6c65c\") " pod="openshift-monitoring/thanos-querier-b7fc86bbb-vrm2p" Jul 15 23:37:11.896349 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:11.896307 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/cc051ada-7c71-4701-94cf-914ad0e6c65c-secret-thanos-querier-tls\") pod \"thanos-querier-b7fc86bbb-vrm2p\" (UID: \"cc051ada-7c71-4701-94cf-914ad0e6c65c\") " pod="openshift-monitoring/thanos-querier-b7fc86bbb-vrm2p" Jul 15 23:37:11.897163 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:11.897101 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/cc051ada-7c71-4701-94cf-914ad0e6c65c-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-b7fc86bbb-vrm2p\" (UID: \"cc051ada-7c71-4701-94cf-914ad0e6c65c\") " pod="openshift-monitoring/thanos-querier-b7fc86bbb-vrm2p" Jul 15 23:37:11.899151 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:11.899103 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/cc051ada-7c71-4701-94cf-914ad0e6c65c-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-b7fc86bbb-vrm2p\" (UID: \"cc051ada-7c71-4701-94cf-914ad0e6c65c\") " pod="openshift-monitoring/thanos-querier-b7fc86bbb-vrm2p" Jul 15 23:37:11.900751 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:11.900730 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lgsps\" (UniqueName: \"kubernetes.io/projected/cc051ada-7c71-4701-94cf-914ad0e6c65c-kube-api-access-lgsps\") pod \"thanos-querier-b7fc86bbb-vrm2p\" (UID: \"cc051ada-7c71-4701-94cf-914ad0e6c65c\") " pod="openshift-monitoring/thanos-querier-b7fc86bbb-vrm2p" Jul 15 23:37:11.993029 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:11.992864 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-b7fc86bbb-vrm2p" Jul 15 23:37:12.080159 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:12.080026 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5e3e5aa3-8bc5-49ff-8706-6107d3e3d2bb","Type":"ContainerStarted","Data":"5cf03839cfe4212e881a3fe9ca9f7b3f27ac1cc32d685dbc3db8649f98d0e9f4"} Jul 15 23:37:12.082190 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:12.082106 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-w5dxc" event={"ID":"96ec5aef-3317-46d0-be5b-952af4818e44","Type":"ContainerStarted","Data":"937a03f53e7fe26c5bcad47eafdf78c1a002911ee12473f59e476e484e63b900"} Jul 15 23:37:12.088867 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:12.088758 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-8sn4g" event={"ID":"dccc9670-e502-402d-b0cd-63d1c54f136f","Type":"ContainerStarted","Data":"54fa6e7aee1134b2dfa6246e23dc9da8c03b31aa5498a0510d12bf08c16967bc"} Jul 15 23:37:12.089199 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:12.089085 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-8sn4g" event={"ID":"dccc9670-e502-402d-b0cd-63d1c54f136f","Type":"ContainerStarted","Data":"8ec10d823e150f09c3c0783c316fb3a3f2eff5e281758c985019af2b9f126323"} Jul 15 23:37:12.089199 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:12.089132 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-8sn4g" event={"ID":"dccc9670-e502-402d-b0cd-63d1c54f136f","Type":"ContainerStarted","Data":"c80fb6647fb9e635655530797fe5bb3e6289522ea23e4a61d61e64d8090560e2"} Jul 15 23:37:12.092458 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:12.092398 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-75d9954bb7-rlshl" event={"ID":"5efd853d-ff2f-43d9-bbe8-fd8ee02465ec","Type":"ContainerStarted","Data":"90ed71e26eb77a56336d1451a0c80808798e2e4a46fc387193582c8282b05c04"} Jul 15 23:37:12.095583 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:12.095557 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-2dsvn" event={"ID":"f6b39980-f5f7-4709-b768-6cc2b2016cd1","Type":"ContainerStarted","Data":"5d10f4d3da12e432d1727afc3b5fdfff7ddf5237903ebefc5f822217575a5137"} Jul 15 23:37:12.171422 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:12.171133 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-75d9954bb7-rlshl" podStartSLOduration=1.358364951 podStartE2EDuration="5.171109922s" totalImagesPullingTime="3.812744971s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 23:37:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 23:37:12.109165415 +0000 UTC m=+47.482623379" watchObservedRunningTime="2026-07-15 23:37:12.171109922 +0000 UTC m=+47.544567886" Jul 15 23:37:12.171735 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:12.171644 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-b7fc86bbb-vrm2p"] Jul 15 23:37:12.176866 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:37:12.176809 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc051ada_7c71_4701_94cf_914ad0e6c65c.slice/crio-cf50fdee12f98d0285d9a7b88c0dc413dddc575bab9706ad10848c6fa6c5ac39 WatchSource:0}: Error finding container cf50fdee12f98d0285d9a7b88c0dc413dddc575bab9706ad10848c6fa6c5ac39: Status 404 returned error can't find the container with id cf50fdee12f98d0285d9a7b88c0dc413dddc575bab9706ad10848c6fa6c5ac39 Jul 15 23:37:12.522584 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:12.522517 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:37:12.623281 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:12.623201 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:37:12.728024 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:12.727925 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:37:12.824315 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:12.824161 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:37:12.923714 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:12.923679 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:37:13.104589 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:13.104450 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-b7fc86bbb-vrm2p" event={"ID":"cc051ada-7c71-4701-94cf-914ad0e6c65c","Type":"ContainerStarted","Data":"cf50fdee12f98d0285d9a7b88c0dc413dddc575bab9706ad10848c6fa6c5ac39"} Jul 15 23:37:13.967667 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:13.967637 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-7cb86fd7bc-4bphf"] Jul 15 23:37:13.990044 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:13.990007 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-7cb86fd7bc-4bphf"] Jul 15 23:37:13.990192 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:13.990141 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-7cb86fd7bc-4bphf" Jul 15 23:37:13.992761 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:13.992740 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Jul 15 23:37:13.992892 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:13.992807 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Jul 15 23:37:13.992993 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:13.992961 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-sbfb4\"" Jul 15 23:37:13.993371 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:13.993348 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 15 23:37:13.993896 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:13.993875 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-92nvjg67ktfqs\"" Jul 15 23:37:13.994008 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:13.993994 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Jul 15 23:37:14.127428 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:14.127398 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/692b9b9d-1237-4e9c-bf86-0255ab7732b7-metrics-server-audit-profiles\") pod \"metrics-server-7cb86fd7bc-4bphf\" (UID: \"692b9b9d-1237-4e9c-bf86-0255ab7732b7\") " pod="openshift-monitoring/metrics-server-7cb86fd7bc-4bphf" Jul 15 23:37:14.127613 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:14.127447 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/692b9b9d-1237-4e9c-bf86-0255ab7732b7-secret-metrics-server-client-certs\") pod \"metrics-server-7cb86fd7bc-4bphf\" (UID: \"692b9b9d-1237-4e9c-bf86-0255ab7732b7\") " pod="openshift-monitoring/metrics-server-7cb86fd7bc-4bphf" Jul 15 23:37:14.127613 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:14.127486 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mbs47\" (UniqueName: \"kubernetes.io/projected/692b9b9d-1237-4e9c-bf86-0255ab7732b7-kube-api-access-mbs47\") pod \"metrics-server-7cb86fd7bc-4bphf\" (UID: \"692b9b9d-1237-4e9c-bf86-0255ab7732b7\") " pod="openshift-monitoring/metrics-server-7cb86fd7bc-4bphf" Jul 15 23:37:14.127713 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:14.127606 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/692b9b9d-1237-4e9c-bf86-0255ab7732b7-secret-metrics-server-tls\") pod \"metrics-server-7cb86fd7bc-4bphf\" (UID: \"692b9b9d-1237-4e9c-bf86-0255ab7732b7\") " pod="openshift-monitoring/metrics-server-7cb86fd7bc-4bphf" Jul 15 23:37:14.127749 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:14.127718 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/692b9b9d-1237-4e9c-bf86-0255ab7732b7-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7cb86fd7bc-4bphf\" (UID: \"692b9b9d-1237-4e9c-bf86-0255ab7732b7\") " pod="openshift-monitoring/metrics-server-7cb86fd7bc-4bphf" Jul 15 23:37:14.127797 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:14.127748 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/692b9b9d-1237-4e9c-bf86-0255ab7732b7-audit-log\") pod \"metrics-server-7cb86fd7bc-4bphf\" (UID: \"692b9b9d-1237-4e9c-bf86-0255ab7732b7\") " pod="openshift-monitoring/metrics-server-7cb86fd7bc-4bphf" Jul 15 23:37:14.127797 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:14.127784 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/692b9b9d-1237-4e9c-bf86-0255ab7732b7-client-ca-bundle\") pod \"metrics-server-7cb86fd7bc-4bphf\" (UID: \"692b9b9d-1237-4e9c-bf86-0255ab7732b7\") " pod="openshift-monitoring/metrics-server-7cb86fd7bc-4bphf" Jul 15 23:37:14.228482 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:14.228400 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/692b9b9d-1237-4e9c-bf86-0255ab7732b7-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7cb86fd7bc-4bphf\" (UID: \"692b9b9d-1237-4e9c-bf86-0255ab7732b7\") " pod="openshift-monitoring/metrics-server-7cb86fd7bc-4bphf" Jul 15 23:37:14.228482 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:14.228434 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/692b9b9d-1237-4e9c-bf86-0255ab7732b7-audit-log\") pod \"metrics-server-7cb86fd7bc-4bphf\" (UID: \"692b9b9d-1237-4e9c-bf86-0255ab7732b7\") " pod="openshift-monitoring/metrics-server-7cb86fd7bc-4bphf" Jul 15 23:37:14.228482 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:14.228460 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/692b9b9d-1237-4e9c-bf86-0255ab7732b7-client-ca-bundle\") pod \"metrics-server-7cb86fd7bc-4bphf\" (UID: \"692b9b9d-1237-4e9c-bf86-0255ab7732b7\") " pod="openshift-monitoring/metrics-server-7cb86fd7bc-4bphf" Jul 15 23:37:14.229159 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:14.228877 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/692b9b9d-1237-4e9c-bf86-0255ab7732b7-metrics-server-audit-profiles\") pod \"metrics-server-7cb86fd7bc-4bphf\" (UID: \"692b9b9d-1237-4e9c-bf86-0255ab7732b7\") " pod="openshift-monitoring/metrics-server-7cb86fd7bc-4bphf" Jul 15 23:37:14.229159 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:14.228932 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/692b9b9d-1237-4e9c-bf86-0255ab7732b7-secret-metrics-server-client-certs\") pod \"metrics-server-7cb86fd7bc-4bphf\" (UID: \"692b9b9d-1237-4e9c-bf86-0255ab7732b7\") " pod="openshift-monitoring/metrics-server-7cb86fd7bc-4bphf" Jul 15 23:37:14.229159 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:14.228993 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbs47\" (UniqueName: \"kubernetes.io/projected/692b9b9d-1237-4e9c-bf86-0255ab7732b7-kube-api-access-mbs47\") pod \"metrics-server-7cb86fd7bc-4bphf\" (UID: \"692b9b9d-1237-4e9c-bf86-0255ab7732b7\") " pod="openshift-monitoring/metrics-server-7cb86fd7bc-4bphf" Jul 15 23:37:14.229159 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:14.229044 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/692b9b9d-1237-4e9c-bf86-0255ab7732b7-secret-metrics-server-tls\") pod \"metrics-server-7cb86fd7bc-4bphf\" (UID: \"692b9b9d-1237-4e9c-bf86-0255ab7732b7\") " pod="openshift-monitoring/metrics-server-7cb86fd7bc-4bphf" Jul 15 23:37:14.229159 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:14.229123 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/692b9b9d-1237-4e9c-bf86-0255ab7732b7-audit-log\") pod \"metrics-server-7cb86fd7bc-4bphf\" (UID: \"692b9b9d-1237-4e9c-bf86-0255ab7732b7\") " pod="openshift-monitoring/metrics-server-7cb86fd7bc-4bphf" Jul 15 23:37:14.229416 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:14.229236 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/692b9b9d-1237-4e9c-bf86-0255ab7732b7-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7cb86fd7bc-4bphf\" (UID: \"692b9b9d-1237-4e9c-bf86-0255ab7732b7\") " pod="openshift-monitoring/metrics-server-7cb86fd7bc-4bphf" Jul 15 23:37:14.229854 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:14.229806 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/692b9b9d-1237-4e9c-bf86-0255ab7732b7-metrics-server-audit-profiles\") pod \"metrics-server-7cb86fd7bc-4bphf\" (UID: \"692b9b9d-1237-4e9c-bf86-0255ab7732b7\") " pod="openshift-monitoring/metrics-server-7cb86fd7bc-4bphf" Jul 15 23:37:14.233268 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:14.232117 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/692b9b9d-1237-4e9c-bf86-0255ab7732b7-secret-metrics-server-client-certs\") pod \"metrics-server-7cb86fd7bc-4bphf\" (UID: \"692b9b9d-1237-4e9c-bf86-0255ab7732b7\") " pod="openshift-monitoring/metrics-server-7cb86fd7bc-4bphf" Jul 15 23:37:14.233268 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:14.232568 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/692b9b9d-1237-4e9c-bf86-0255ab7732b7-secret-metrics-server-tls\") pod \"metrics-server-7cb86fd7bc-4bphf\" (UID: \"692b9b9d-1237-4e9c-bf86-0255ab7732b7\") " pod="openshift-monitoring/metrics-server-7cb86fd7bc-4bphf" Jul 15 23:37:14.233268 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:14.233181 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/692b9b9d-1237-4e9c-bf86-0255ab7732b7-client-ca-bundle\") pod \"metrics-server-7cb86fd7bc-4bphf\" (UID: \"692b9b9d-1237-4e9c-bf86-0255ab7732b7\") " pod="openshift-monitoring/metrics-server-7cb86fd7bc-4bphf" Jul 15 23:37:14.237680 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:14.237650 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbs47\" (UniqueName: \"kubernetes.io/projected/692b9b9d-1237-4e9c-bf86-0255ab7732b7-kube-api-access-mbs47\") pod \"metrics-server-7cb86fd7bc-4bphf\" (UID: \"692b9b9d-1237-4e9c-bf86-0255ab7732b7\") " pod="openshift-monitoring/metrics-server-7cb86fd7bc-4bphf" Jul 15 23:37:14.302330 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:14.302297 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-7cb86fd7bc-4bphf" Jul 15 23:37:14.456466 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:14.456437 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:37:14.461801 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:14.461779 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-5f68df7f58-gkz55"] Jul 15 23:37:14.485245 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:14.485213 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5f68df7f58-gkz55"] Jul 15 23:37:14.485366 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:14.485303 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-gkz55" Jul 15 23:37:14.487638 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:14.487615 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Jul 15 23:37:14.487763 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:14.487635 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-s4g5d\"" Jul 15 23:37:14.556644 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:14.556618 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:37:14.633358 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:14.633324 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e0f740d6-d99f-48e0-8da0-66e359d9fb48-monitoring-plugin-cert\") pod \"monitoring-plugin-5f68df7f58-gkz55\" (UID: \"e0f740d6-d99f-48e0-8da0-66e359d9fb48\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-gkz55" Jul 15 23:37:14.633478 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:14.633424 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdkcm\" (UniqueName: \"kubernetes.io/projected/e0f740d6-d99f-48e0-8da0-66e359d9fb48-kube-api-access-rdkcm\") pod \"monitoring-plugin-5f68df7f58-gkz55\" (UID: \"e0f740d6-d99f-48e0-8da0-66e359d9fb48\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-gkz55" Jul 15 23:37:14.655712 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:14.655685 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:37:14.733945 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:14.733912 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdkcm\" (UniqueName: \"kubernetes.io/projected/e0f740d6-d99f-48e0-8da0-66e359d9fb48-kube-api-access-rdkcm\") pod \"monitoring-plugin-5f68df7f58-gkz55\" (UID: \"e0f740d6-d99f-48e0-8da0-66e359d9fb48\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-gkz55" Jul 15 23:37:14.734114 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:14.734053 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e0f740d6-d99f-48e0-8da0-66e359d9fb48-monitoring-plugin-cert\") pod \"monitoring-plugin-5f68df7f58-gkz55\" (UID: \"e0f740d6-d99f-48e0-8da0-66e359d9fb48\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-gkz55" Jul 15 23:37:14.736796 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:14.736742 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e0f740d6-d99f-48e0-8da0-66e359d9fb48-monitoring-plugin-cert\") pod \"monitoring-plugin-5f68df7f58-gkz55\" (UID: \"e0f740d6-d99f-48e0-8da0-66e359d9fb48\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-gkz55" Jul 15 23:37:14.742036 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:14.741989 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdkcm\" (UniqueName: \"kubernetes.io/projected/e0f740d6-d99f-48e0-8da0-66e359d9fb48-kube-api-access-rdkcm\") pod \"monitoring-plugin-5f68df7f58-gkz55\" (UID: \"e0f740d6-d99f-48e0-8da0-66e359d9fb48\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-gkz55" Jul 15 23:37:14.794791 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:14.794765 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-gkz55" Jul 15 23:37:14.885472 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:14.885441 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-786d565755-fqg7t"] Jul 15 23:37:14.902812 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:14.902764 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-786d565755-fqg7t"] Jul 15 23:37:14.903012 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:14.902901 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-786d565755-fqg7t" Jul 15 23:37:14.905763 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:14.905741 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Jul 15 23:37:14.905876 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:14.905740 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Jul 15 23:37:14.906353 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:14.906174 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Jul 15 23:37:14.906353 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:14.906204 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-4rdk2\"" Jul 15 23:37:14.906353 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:14.906216 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Jul 15 23:37:14.906353 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:14.906231 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Jul 15 23:37:14.911695 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:14.911675 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Jul 15 23:37:15.036747 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:15.036665 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/7f775c78-eab6-437d-a2e9-291049e51b94-federate-client-tls\") pod \"telemeter-client-786d565755-fqg7t\" (UID: \"7f775c78-eab6-437d-a2e9-291049e51b94\") " pod="openshift-monitoring/telemeter-client-786d565755-fqg7t" Jul 15 23:37:15.036747 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:15.036719 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-99slv\" (UniqueName: \"kubernetes.io/projected/7f775c78-eab6-437d-a2e9-291049e51b94-kube-api-access-99slv\") pod \"telemeter-client-786d565755-fqg7t\" (UID: \"7f775c78-eab6-437d-a2e9-291049e51b94\") " pod="openshift-monitoring/telemeter-client-786d565755-fqg7t" Jul 15 23:37:15.037190 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:15.036801 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/7f775c78-eab6-437d-a2e9-291049e51b94-telemeter-client-tls\") pod \"telemeter-client-786d565755-fqg7t\" (UID: \"7f775c78-eab6-437d-a2e9-291049e51b94\") " pod="openshift-monitoring/telemeter-client-786d565755-fqg7t" Jul 15 23:37:15.037190 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:15.036833 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7f775c78-eab6-437d-a2e9-291049e51b94-metrics-client-ca\") pod \"telemeter-client-786d565755-fqg7t\" (UID: \"7f775c78-eab6-437d-a2e9-291049e51b94\") " pod="openshift-monitoring/telemeter-client-786d565755-fqg7t" Jul 15 23:37:15.037190 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:15.036885 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7f775c78-eab6-437d-a2e9-291049e51b94-serving-certs-ca-bundle\") pod \"telemeter-client-786d565755-fqg7t\" (UID: \"7f775c78-eab6-437d-a2e9-291049e51b94\") " pod="openshift-monitoring/telemeter-client-786d565755-fqg7t" Jul 15 23:37:15.037190 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:15.037006 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7f775c78-eab6-437d-a2e9-291049e51b94-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-786d565755-fqg7t\" (UID: \"7f775c78-eab6-437d-a2e9-291049e51b94\") " pod="openshift-monitoring/telemeter-client-786d565755-fqg7t" Jul 15 23:37:15.037190 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:15.037057 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7f775c78-eab6-437d-a2e9-291049e51b94-telemeter-trusted-ca-bundle\") pod \"telemeter-client-786d565755-fqg7t\" (UID: \"7f775c78-eab6-437d-a2e9-291049e51b94\") " pod="openshift-monitoring/telemeter-client-786d565755-fqg7t" Jul 15 23:37:15.037190 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:15.037106 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/7f775c78-eab6-437d-a2e9-291049e51b94-secret-telemeter-client\") pod \"telemeter-client-786d565755-fqg7t\" (UID: \"7f775c78-eab6-437d-a2e9-291049e51b94\") " pod="openshift-monitoring/telemeter-client-786d565755-fqg7t" Jul 15 23:37:15.137585 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:15.137553 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99slv\" (UniqueName: \"kubernetes.io/projected/7f775c78-eab6-437d-a2e9-291049e51b94-kube-api-access-99slv\") pod \"telemeter-client-786d565755-fqg7t\" (UID: \"7f775c78-eab6-437d-a2e9-291049e51b94\") " pod="openshift-monitoring/telemeter-client-786d565755-fqg7t" Jul 15 23:37:15.137757 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:15.137611 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/7f775c78-eab6-437d-a2e9-291049e51b94-telemeter-client-tls\") pod \"telemeter-client-786d565755-fqg7t\" (UID: \"7f775c78-eab6-437d-a2e9-291049e51b94\") " pod="openshift-monitoring/telemeter-client-786d565755-fqg7t" Jul 15 23:37:15.137757 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:15.137637 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7f775c78-eab6-437d-a2e9-291049e51b94-metrics-client-ca\") pod \"telemeter-client-786d565755-fqg7t\" (UID: \"7f775c78-eab6-437d-a2e9-291049e51b94\") " pod="openshift-monitoring/telemeter-client-786d565755-fqg7t" Jul 15 23:37:15.137757 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:15.137694 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7f775c78-eab6-437d-a2e9-291049e51b94-serving-certs-ca-bundle\") pod \"telemeter-client-786d565755-fqg7t\" (UID: \"7f775c78-eab6-437d-a2e9-291049e51b94\") " pod="openshift-monitoring/telemeter-client-786d565755-fqg7t" Jul 15 23:37:15.137929 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:15.137853 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7f775c78-eab6-437d-a2e9-291049e51b94-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-786d565755-fqg7t\" (UID: \"7f775c78-eab6-437d-a2e9-291049e51b94\") " pod="openshift-monitoring/telemeter-client-786d565755-fqg7t" Jul 15 23:37:15.137929 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:15.137907 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7f775c78-eab6-437d-a2e9-291049e51b94-telemeter-trusted-ca-bundle\") pod \"telemeter-client-786d565755-fqg7t\" (UID: \"7f775c78-eab6-437d-a2e9-291049e51b94\") " pod="openshift-monitoring/telemeter-client-786d565755-fqg7t" Jul 15 23:37:15.138060 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:15.137958 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/7f775c78-eab6-437d-a2e9-291049e51b94-secret-telemeter-client\") pod \"telemeter-client-786d565755-fqg7t\" (UID: \"7f775c78-eab6-437d-a2e9-291049e51b94\") " pod="openshift-monitoring/telemeter-client-786d565755-fqg7t" Jul 15 23:37:15.138060 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:15.138045 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/7f775c78-eab6-437d-a2e9-291049e51b94-federate-client-tls\") pod \"telemeter-client-786d565755-fqg7t\" (UID: \"7f775c78-eab6-437d-a2e9-291049e51b94\") " pod="openshift-monitoring/telemeter-client-786d565755-fqg7t" Jul 15 23:37:15.138526 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:15.138500 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7f775c78-eab6-437d-a2e9-291049e51b94-serving-certs-ca-bundle\") pod \"telemeter-client-786d565755-fqg7t\" (UID: \"7f775c78-eab6-437d-a2e9-291049e51b94\") " pod="openshift-monitoring/telemeter-client-786d565755-fqg7t" Jul 15 23:37:15.138526 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:15.138520 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7f775c78-eab6-437d-a2e9-291049e51b94-metrics-client-ca\") pod \"telemeter-client-786d565755-fqg7t\" (UID: \"7f775c78-eab6-437d-a2e9-291049e51b94\") " pod="openshift-monitoring/telemeter-client-786d565755-fqg7t" Jul 15 23:37:15.138751 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:15.138733 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7f775c78-eab6-437d-a2e9-291049e51b94-telemeter-trusted-ca-bundle\") pod \"telemeter-client-786d565755-fqg7t\" (UID: \"7f775c78-eab6-437d-a2e9-291049e51b94\") " pod="openshift-monitoring/telemeter-client-786d565755-fqg7t" Jul 15 23:37:15.140269 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:15.140246 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/7f775c78-eab6-437d-a2e9-291049e51b94-secret-telemeter-client\") pod \"telemeter-client-786d565755-fqg7t\" (UID: \"7f775c78-eab6-437d-a2e9-291049e51b94\") " pod="openshift-monitoring/telemeter-client-786d565755-fqg7t" Jul 15 23:37:15.140403 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:15.140382 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/7f775c78-eab6-437d-a2e9-291049e51b94-federate-client-tls\") pod \"telemeter-client-786d565755-fqg7t\" (UID: \"7f775c78-eab6-437d-a2e9-291049e51b94\") " pod="openshift-monitoring/telemeter-client-786d565755-fqg7t" Jul 15 23:37:15.140542 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:15.140525 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7f775c78-eab6-437d-a2e9-291049e51b94-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-786d565755-fqg7t\" (UID: \"7f775c78-eab6-437d-a2e9-291049e51b94\") " pod="openshift-monitoring/telemeter-client-786d565755-fqg7t" Jul 15 23:37:15.140578 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:15.140560 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/7f775c78-eab6-437d-a2e9-291049e51b94-telemeter-client-tls\") pod \"telemeter-client-786d565755-fqg7t\" (UID: \"7f775c78-eab6-437d-a2e9-291049e51b94\") " pod="openshift-monitoring/telemeter-client-786d565755-fqg7t" Jul 15 23:37:15.145089 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:15.145068 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-99slv\" (UniqueName: \"kubernetes.io/projected/7f775c78-eab6-437d-a2e9-291049e51b94-kube-api-access-99slv\") pod \"telemeter-client-786d565755-fqg7t\" (UID: \"7f775c78-eab6-437d-a2e9-291049e51b94\") " pod="openshift-monitoring/telemeter-client-786d565755-fqg7t" Jul 15 23:37:15.215194 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:15.215172 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-786d565755-fqg7t" Jul 15 23:37:15.448739 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:15.448624 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:37:15.549945 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:15.549907 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:37:15.649302 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:15.649270 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:37:15.750433 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:15.750346 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:37:15.858527 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:15.858493 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 15 23:37:15.864014 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:15.863982 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:37:15.885524 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:15.885437 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 15 23:37:15.885678 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:15.885578 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 15 23:37:15.888835 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:15.888809 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-4ao1ufe85b9l1\"" Jul 15 23:37:15.889094 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:15.889076 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 15 23:37:15.889621 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:15.889433 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 15 23:37:15.889703 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:15.889296 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 15 23:37:15.890520 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:15.889304 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-z6md5\"" Jul 15 23:37:15.890520 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:15.889356 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 15 23:37:15.890520 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:15.889195 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 15 23:37:15.890520 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:15.889359 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 15 23:37:15.890520 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:15.890133 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 15 23:37:15.890520 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:15.890240 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 15 23:37:15.890520 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:15.890312 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 15 23:37:15.890520 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:15.890242 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 15 23:37:15.890520 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:15.890343 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 15 23:37:15.890520 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:15.890426 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 15 23:37:15.890520 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:15.890466 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 15 23:37:15.892752 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:15.892484 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 15 23:37:16.046495 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:16.046412 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a881fb94-9a0b-4c46-b5fc-d841398d576d-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a881fb94-9a0b-4c46-b5fc-d841398d576d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 23:37:16.046495 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:16.046465 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a881fb94-9a0b-4c46-b5fc-d841398d576d-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a881fb94-9a0b-4c46-b5fc-d841398d576d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 23:37:16.046956 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:16.046560 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a881fb94-9a0b-4c46-b5fc-d841398d576d-config-out\") pod \"prometheus-k8s-0\" (UID: \"a881fb94-9a0b-4c46-b5fc-d841398d576d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 23:37:16.046956 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:16.046595 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a881fb94-9a0b-4c46-b5fc-d841398d576d-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"a881fb94-9a0b-4c46-b5fc-d841398d576d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 23:37:16.046956 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:16.046679 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/a881fb94-9a0b-4c46-b5fc-d841398d576d-config\") pod \"prometheus-k8s-0\" (UID: \"a881fb94-9a0b-4c46-b5fc-d841398d576d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 23:37:16.046956 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:16.046708 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/a881fb94-9a0b-4c46-b5fc-d841398d576d-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"a881fb94-9a0b-4c46-b5fc-d841398d576d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 23:37:16.046956 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:16.046726 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a881fb94-9a0b-4c46-b5fc-d841398d576d-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"a881fb94-9a0b-4c46-b5fc-d841398d576d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 23:37:16.046956 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:16.046754 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a881fb94-9a0b-4c46-b5fc-d841398d576d-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"a881fb94-9a0b-4c46-b5fc-d841398d576d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 23:37:16.046956 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:16.046822 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a881fb94-9a0b-4c46-b5fc-d841398d576d-web-config\") pod \"prometheus-k8s-0\" (UID: \"a881fb94-9a0b-4c46-b5fc-d841398d576d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 23:37:16.046956 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:16.046893 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/a881fb94-9a0b-4c46-b5fc-d841398d576d-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"a881fb94-9a0b-4c46-b5fc-d841398d576d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 23:37:16.046956 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:16.046918 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a881fb94-9a0b-4c46-b5fc-d841398d576d-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"a881fb94-9a0b-4c46-b5fc-d841398d576d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 23:37:16.046956 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:16.046946 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a881fb94-9a0b-4c46-b5fc-d841398d576d-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"a881fb94-9a0b-4c46-b5fc-d841398d576d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 23:37:16.047406 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:16.046988 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a881fb94-9a0b-4c46-b5fc-d841398d576d-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"a881fb94-9a0b-4c46-b5fc-d841398d576d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 23:37:16.047406 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:16.047015 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a881fb94-9a0b-4c46-b5fc-d841398d576d-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"a881fb94-9a0b-4c46-b5fc-d841398d576d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 23:37:16.047406 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:16.047042 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xmvk\" (UniqueName: \"kubernetes.io/projected/a881fb94-9a0b-4c46-b5fc-d841398d576d-kube-api-access-7xmvk\") pod \"prometheus-k8s-0\" (UID: \"a881fb94-9a0b-4c46-b5fc-d841398d576d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 23:37:16.047406 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:16.047097 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/a881fb94-9a0b-4c46-b5fc-d841398d576d-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"a881fb94-9a0b-4c46-b5fc-d841398d576d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 23:37:16.047406 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:16.047126 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a881fb94-9a0b-4c46-b5fc-d841398d576d-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"a881fb94-9a0b-4c46-b5fc-d841398d576d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 23:37:16.047406 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:16.047149 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a881fb94-9a0b-4c46-b5fc-d841398d576d-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a881fb94-9a0b-4c46-b5fc-d841398d576d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 23:37:16.047406 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:16.047205 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a881fb94-9a0b-4c46-b5fc-d841398d576d-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"a881fb94-9a0b-4c46-b5fc-d841398d576d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 23:37:16.047406 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:16.047274 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/a881fb94-9a0b-4c46-b5fc-d841398d576d-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"a881fb94-9a0b-4c46-b5fc-d841398d576d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 23:37:16.048331 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:16.048310 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-wkhxm" Jul 15 23:37:16.149279 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:16.148656 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a881fb94-9a0b-4c46-b5fc-d841398d576d-config-out\") pod \"prometheus-k8s-0\" (UID: \"a881fb94-9a0b-4c46-b5fc-d841398d576d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 23:37:16.149279 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:16.148707 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a881fb94-9a0b-4c46-b5fc-d841398d576d-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"a881fb94-9a0b-4c46-b5fc-d841398d576d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 23:37:16.149279 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:16.148767 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/a881fb94-9a0b-4c46-b5fc-d841398d576d-config\") pod \"prometheus-k8s-0\" (UID: \"a881fb94-9a0b-4c46-b5fc-d841398d576d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 23:37:16.149279 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:16.148792 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/a881fb94-9a0b-4c46-b5fc-d841398d576d-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"a881fb94-9a0b-4c46-b5fc-d841398d576d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 23:37:16.149279 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:16.148815 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a881fb94-9a0b-4c46-b5fc-d841398d576d-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"a881fb94-9a0b-4c46-b5fc-d841398d576d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 23:37:16.149279 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:16.148845 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a881fb94-9a0b-4c46-b5fc-d841398d576d-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"a881fb94-9a0b-4c46-b5fc-d841398d576d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 23:37:16.149279 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:16.148877 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a881fb94-9a0b-4c46-b5fc-d841398d576d-web-config\") pod \"prometheus-k8s-0\" (UID: \"a881fb94-9a0b-4c46-b5fc-d841398d576d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 23:37:16.149279 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:16.148936 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/a881fb94-9a0b-4c46-b5fc-d841398d576d-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"a881fb94-9a0b-4c46-b5fc-d841398d576d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 23:37:16.149279 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:16.148960 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a881fb94-9a0b-4c46-b5fc-d841398d576d-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"a881fb94-9a0b-4c46-b5fc-d841398d576d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 23:37:16.149279 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:16.149058 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a881fb94-9a0b-4c46-b5fc-d841398d576d-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"a881fb94-9a0b-4c46-b5fc-d841398d576d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 23:37:16.149279 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:16.149088 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a881fb94-9a0b-4c46-b5fc-d841398d576d-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"a881fb94-9a0b-4c46-b5fc-d841398d576d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 23:37:16.149279 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:16.149118 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a881fb94-9a0b-4c46-b5fc-d841398d576d-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"a881fb94-9a0b-4c46-b5fc-d841398d576d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 23:37:16.149279 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:16.149145 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xmvk\" (UniqueName: \"kubernetes.io/projected/a881fb94-9a0b-4c46-b5fc-d841398d576d-kube-api-access-7xmvk\") pod \"prometheus-k8s-0\" (UID: \"a881fb94-9a0b-4c46-b5fc-d841398d576d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 23:37:16.149279 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:16.149204 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/a881fb94-9a0b-4c46-b5fc-d841398d576d-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"a881fb94-9a0b-4c46-b5fc-d841398d576d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 23:37:16.149279 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:16.149230 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a881fb94-9a0b-4c46-b5fc-d841398d576d-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"a881fb94-9a0b-4c46-b5fc-d841398d576d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 23:37:16.149279 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:16.149255 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a881fb94-9a0b-4c46-b5fc-d841398d576d-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a881fb94-9a0b-4c46-b5fc-d841398d576d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 23:37:16.150259 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:16.149300 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a881fb94-9a0b-4c46-b5fc-d841398d576d-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"a881fb94-9a0b-4c46-b5fc-d841398d576d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 23:37:16.150259 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:16.149344 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/a881fb94-9a0b-4c46-b5fc-d841398d576d-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"a881fb94-9a0b-4c46-b5fc-d841398d576d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 23:37:16.150259 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:16.149373 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a881fb94-9a0b-4c46-b5fc-d841398d576d-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a881fb94-9a0b-4c46-b5fc-d841398d576d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 23:37:16.150259 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:16.149407 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a881fb94-9a0b-4c46-b5fc-d841398d576d-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a881fb94-9a0b-4c46-b5fc-d841398d576d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 23:37:16.150456 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:16.150336 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a881fb94-9a0b-4c46-b5fc-d841398d576d-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a881fb94-9a0b-4c46-b5fc-d841398d576d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 23:37:16.150456 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:16.150376 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/a881fb94-9a0b-4c46-b5fc-d841398d576d-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"a881fb94-9a0b-4c46-b5fc-d841398d576d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 23:37:16.151479 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:16.150843 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a881fb94-9a0b-4c46-b5fc-d841398d576d-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"a881fb94-9a0b-4c46-b5fc-d841398d576d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 23:37:16.152217 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:16.152193 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a881fb94-9a0b-4c46-b5fc-d841398d576d-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"a881fb94-9a0b-4c46-b5fc-d841398d576d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 23:37:16.153007 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:16.152745 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a881fb94-9a0b-4c46-b5fc-d841398d576d-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"a881fb94-9a0b-4c46-b5fc-d841398d576d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 23:37:16.153007 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:16.152774 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a881fb94-9a0b-4c46-b5fc-d841398d576d-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"a881fb94-9a0b-4c46-b5fc-d841398d576d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 23:37:16.153468 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:16.153443 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a881fb94-9a0b-4c46-b5fc-d841398d576d-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a881fb94-9a0b-4c46-b5fc-d841398d576d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 23:37:16.155949 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:16.155874 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a881fb94-9a0b-4c46-b5fc-d841398d576d-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a881fb94-9a0b-4c46-b5fc-d841398d576d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 23:37:16.155949 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:16.155893 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a881fb94-9a0b-4c46-b5fc-d841398d576d-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"a881fb94-9a0b-4c46-b5fc-d841398d576d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 23:37:16.156132 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:16.155985 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a881fb94-9a0b-4c46-b5fc-d841398d576d-web-config\") pod \"prometheus-k8s-0\" (UID: \"a881fb94-9a0b-4c46-b5fc-d841398d576d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 23:37:16.156132 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:16.156125 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a881fb94-9a0b-4c46-b5fc-d841398d576d-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"a881fb94-9a0b-4c46-b5fc-d841398d576d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 23:37:16.156807 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:16.156763 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/a881fb94-9a0b-4c46-b5fc-d841398d576d-config\") pod \"prometheus-k8s-0\" (UID: \"a881fb94-9a0b-4c46-b5fc-d841398d576d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 23:37:16.157583 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:16.157084 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a881fb94-9a0b-4c46-b5fc-d841398d576d-config-out\") pod \"prometheus-k8s-0\" (UID: \"a881fb94-9a0b-4c46-b5fc-d841398d576d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 23:37:16.157583 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:16.157554 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a881fb94-9a0b-4c46-b5fc-d841398d576d-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"a881fb94-9a0b-4c46-b5fc-d841398d576d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 23:37:16.157583 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:16.157561 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a881fb94-9a0b-4c46-b5fc-d841398d576d-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"a881fb94-9a0b-4c46-b5fc-d841398d576d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 23:37:16.160304 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:16.158001 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/a881fb94-9a0b-4c46-b5fc-d841398d576d-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"a881fb94-9a0b-4c46-b5fc-d841398d576d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 23:37:16.160304 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:16.158561 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/a881fb94-9a0b-4c46-b5fc-d841398d576d-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"a881fb94-9a0b-4c46-b5fc-d841398d576d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 23:37:16.160304 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:16.159110 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/a881fb94-9a0b-4c46-b5fc-d841398d576d-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"a881fb94-9a0b-4c46-b5fc-d841398d576d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 23:37:16.160825 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:16.160806 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a881fb94-9a0b-4c46-b5fc-d841398d576d-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"a881fb94-9a0b-4c46-b5fc-d841398d576d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 23:37:16.161173 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:16.161148 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xmvk\" (UniqueName: \"kubernetes.io/projected/a881fb94-9a0b-4c46-b5fc-d841398d576d-kube-api-access-7xmvk\") pod \"prometheus-k8s-0\" (UID: \"a881fb94-9a0b-4c46-b5fc-d841398d576d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 23:37:16.200176 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:16.200148 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 15 23:37:16.633096 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:16.633042 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:37:16.689640 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:16.689603 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:37:17.429994 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:17.429946 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:37:17.477428 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:17.477396 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:37:17.752174 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:17.752140 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-75d9954bb7-rlshl" Jul 15 23:37:17.752342 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:17.752204 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-75d9954bb7-rlshl" Jul 15 23:37:17.757427 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:17.757403 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-75d9954bb7-rlshl" Jul 15 23:37:18.132004 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:18.131857 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-75d9954bb7-rlshl" Jul 15 23:37:21.818993 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:21.818944 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:37:21.864538 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:21.864506 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:37:24.104099 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:24.104060 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:37:24.151825 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:24.151793 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:37:24.396986 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:24.396858 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:37:24.497761 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:24.497729 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:37:24.555512 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:24.555477 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:37:25.081830 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:25.081794 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:37:25.132653 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:25.132611 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:37:25.957040 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:25.957019 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-786d565755-fqg7t"] Jul 15 23:37:26.269303 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:37:26.268786 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f775c78_eab6_437d_a2e9_291049e51b94.slice/crio-8c8d22a453216c8af679032cc885455be83f407cdea3cb33959eea7c20cb6b02 WatchSource:0}: Error finding container 8c8d22a453216c8af679032cc885455be83f407cdea3cb33959eea7c20cb6b02: Status 404 returned error can't find the container with id 8c8d22a453216c8af679032cc885455be83f407cdea3cb33959eea7c20cb6b02 Jul 15 23:37:26.403487 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:26.403450 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-7cb86fd7bc-4bphf"] Jul 15 23:37:26.635453 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:26.635429 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5f68df7f58-gkz55"] Jul 15 23:37:26.645760 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:26.645733 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 15 23:37:26.733751 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:37:26.733716 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda881fb94_9a0b_4c46_b5fc_d841398d576d.slice/crio-4728431289fcfe070efdba71508337b7ebf686259b1a1db9b984ec034efa7206 WatchSource:0}: Error finding container 4728431289fcfe070efdba71508337b7ebf686259b1a1db9b984ec034efa7206: Status 404 returned error can't find the container with id 4728431289fcfe070efdba71508337b7ebf686259b1a1db9b984ec034efa7206 Jul 15 23:37:26.734767 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:37:26.734728 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod692b9b9d_1237_4e9c_bf86_0255ab7732b7.slice/crio-f13854775b213e37e354e28fcd37c517b68f536f81b109e977ed3f30fbeba83f WatchSource:0}: Error finding container f13854775b213e37e354e28fcd37c517b68f536f81b109e977ed3f30fbeba83f: Status 404 returned error can't find the container with id f13854775b213e37e354e28fcd37c517b68f536f81b109e977ed3f30fbeba83f Jul 15 23:37:26.735767 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:37:26.735742 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0f740d6_d99f_48e0_8da0_66e359d9fb48.slice/crio-494097d64f1c6700bdd3fdde39374e59face4348c86caded368fb3318d3a3ddf WatchSource:0}: Error finding container 494097d64f1c6700bdd3fdde39374e59face4348c86caded368fb3318d3a3ddf: Status 404 returned error can't find the container with id 494097d64f1c6700bdd3fdde39374e59face4348c86caded368fb3318d3a3ddf Jul 15 23:37:27.013471 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:27.013219 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-d8f6k" Jul 15 23:37:27.115638 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:27.115435 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:37:27.179600 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:27.178755 2563 generic.go:356] "Generic (PLEG): container finished" podID="96ec5aef-3317-46d0-be5b-952af4818e44" containerID="d67a3ab25185ac49ac1d6789d28d8296dc08ea48ecb4987680be18c896a172bf" exitCode=0 Jul 15 23:37:27.179600 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:27.178840 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-w5dxc" event={"ID":"96ec5aef-3317-46d0-be5b-952af4818e44","Type":"ContainerDied","Data":"d67a3ab25185ac49ac1d6789d28d8296dc08ea48ecb4987680be18c896a172bf"} Jul 15 23:37:27.201005 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:27.193809 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-8sn4g" event={"ID":"dccc9670-e502-402d-b0cd-63d1c54f136f","Type":"ContainerStarted","Data":"8ca07f9420488e6ad845aa8986b53e34ebb55f70840a501e6493556de3f42bf6"} Jul 15 23:37:27.202537 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:27.202418 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:37:27.206289 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:27.206257 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-gkz55" event={"ID":"e0f740d6-d99f-48e0-8da0-66e359d9fb48","Type":"ContainerStarted","Data":"494097d64f1c6700bdd3fdde39374e59face4348c86caded368fb3318d3a3ddf"} Jul 15 23:37:27.213343 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:27.213268 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-8sn4g" podStartSLOduration=3.908030212 podStartE2EDuration="18.213247331s" totalImagesPullingTime="14.305217119s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 23:37:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 23:37:27.211259408 +0000 UTC m=+62.584717352" watchObservedRunningTime="2026-07-15 23:37:27.213247331 +0000 UTC m=+62.586705295" Jul 15 23:37:27.226071 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:27.225829 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-b7fc86bbb-vrm2p" event={"ID":"cc051ada-7c71-4701-94cf-914ad0e6c65c","Type":"ContainerStarted","Data":"9351a9ae77bc90ccc181691b571b9d85f4c59c5ec6e038d67c27aee19579f158"} Jul 15 23:37:27.226071 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:27.225937 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-b7fc86bbb-vrm2p" event={"ID":"cc051ada-7c71-4701-94cf-914ad0e6c65c","Type":"ContainerStarted","Data":"9b40beea8c20a6a96fbbfaa2cb75ca56281f7ec5f0baa154b1fdae7dcb266e09"} Jul 15 23:37:27.226071 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:27.226005 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-b7fc86bbb-vrm2p" event={"ID":"cc051ada-7c71-4701-94cf-914ad0e6c65c","Type":"ContainerStarted","Data":"f906f7e731e95f32a0d6a1e8b84e6afa373c29bc13bb934156842012cb3a28a6"} Jul 15 23:37:27.227669 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:27.227630 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-786d565755-fqg7t" event={"ID":"7f775c78-eab6-437d-a2e9-291049e51b94","Type":"ContainerStarted","Data":"8c8d22a453216c8af679032cc885455be83f407cdea3cb33959eea7c20cb6b02"} Jul 15 23:37:27.232624 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:27.232591 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-2dsvn" event={"ID":"f6b39980-f5f7-4709-b768-6cc2b2016cd1","Type":"ContainerStarted","Data":"2ca53f6c6d2d151fc8072947a1637a4fa5219419b831ea2cc5797ef2248bdf70"} Jul 15 23:37:27.232736 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:27.232647 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-2dsvn" event={"ID":"f6b39980-f5f7-4709-b768-6cc2b2016cd1","Type":"ContainerStarted","Data":"7c99d4dc222ba05df645aa37f22489c84bc2d563650376adf70b06aba911772b"} Jul 15 23:37:27.232736 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:27.232678 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-2dsvn" event={"ID":"f6b39980-f5f7-4709-b768-6cc2b2016cd1","Type":"ContainerStarted","Data":"cda3a6476c945d7d5dc67ab90e51cf6844623d5412c2db7c82ed7f35940c3dec"} Jul 15 23:37:27.237134 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:27.236902 2563 generic.go:356] "Generic (PLEG): container finished" podID="a881fb94-9a0b-4c46-b5fc-d841398d576d" containerID="8f7819dd3ffe1271d15f5961909f0a003400c7c166497136c73d2661416e5a06" exitCode=0 Jul 15 23:37:27.237134 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:27.237063 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a881fb94-9a0b-4c46-b5fc-d841398d576d","Type":"ContainerDied","Data":"8f7819dd3ffe1271d15f5961909f0a003400c7c166497136c73d2661416e5a06"} Jul 15 23:37:27.237321 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:27.237159 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a881fb94-9a0b-4c46-b5fc-d841398d576d","Type":"ContainerStarted","Data":"4728431289fcfe070efdba71508337b7ebf686259b1a1db9b984ec034efa7206"} Jul 15 23:37:27.241225 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:27.240555 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-7cb86fd7bc-4bphf" event={"ID":"692b9b9d-1237-4e9c-bf86-0255ab7732b7","Type":"ContainerStarted","Data":"f13854775b213e37e354e28fcd37c517b68f536f81b109e977ed3f30fbeba83f"} Jul 15 23:37:27.244125 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:27.244074 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-68c9cbb497-4xmmb" event={"ID":"59e5d97d-b146-4ca9-96c3-a2702d1d36ae","Type":"ContainerStarted","Data":"3d5f3a8a46a8a27c1857d0b2ea407aabc4dd9377cc51721aee278435e1f276ad"} Jul 15 23:37:27.244269 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:27.244227 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-68c9cbb497-4xmmb" Jul 15 23:37:27.247497 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:27.247476 2563 generic.go:356] "Generic (PLEG): container finished" podID="5e3e5aa3-8bc5-49ff-8706-6107d3e3d2bb" containerID="baa174551f012dbe92f015d620924fd4ab07cf84fbe2158b0919fd5626754bc6" exitCode=0 Jul 15 23:37:27.247640 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:27.247550 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5e3e5aa3-8bc5-49ff-8706-6107d3e3d2bb","Type":"ContainerDied","Data":"baa174551f012dbe92f015d620924fd4ab07cf84fbe2158b0919fd5626754bc6"} Jul 15 23:37:27.252990 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:27.252930 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-785778ff74-2dsvn" podStartSLOduration=4.073910939 podStartE2EDuration="18.252899031s" totalImagesPullingTime="14.178988092s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 23:37:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 23:37:27.250748466 +0000 UTC m=+62.624206430" watchObservedRunningTime="2026-07-15 23:37:27.252899031 +0000 UTC m=+62.626356993" Jul 15 23:37:27.262145 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:27.262108 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-68c9cbb497-4xmmb" Jul 15 23:37:27.264322 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:27.264122 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:37:27.324888 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:27.324820 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-68c9cbb497-4xmmb" podStartSLOduration=2.241765004 podStartE2EDuration="26.324797677s" totalImagesPullingTime="24.083032673s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 23:37:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 23:37:27.322700916 +0000 UTC m=+62.696158917" watchObservedRunningTime="2026-07-15 23:37:27.324797677 +0000 UTC m=+62.698255635" Jul 15 23:37:27.549267 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:27.548509 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-75d9954bb7-rlshl"] Jul 15 23:37:28.274321 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:28.274203 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5q8md" event={"ID":"409f6854-b231-41aa-8715-a300ee9aa09e","Type":"ContainerStarted","Data":"0db75cdd1ee3d44fe72770cb25ac33b6889d985de2acca308b7bf3b28ee30a71"} Jul 15 23:37:28.281979 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:28.281123 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-w5dxc" event={"ID":"96ec5aef-3317-46d0-be5b-952af4818e44","Type":"ContainerStarted","Data":"04b83e6b3cda89fe9f6e975ee69f19c13543125ae69634b82aa64a266d923d61"} Jul 15 23:37:28.281979 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:28.281177 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-w5dxc" event={"ID":"96ec5aef-3317-46d0-be5b-952af4818e44","Type":"ContainerStarted","Data":"862e4262170afcc7f1512c77ecb99352edf12d0bf56f58654c7f746813a3ce5a"} Jul 15 23:37:28.299941 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:28.299859 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-w5dxc" podStartSLOduration=4.856968995 podStartE2EDuration="19.299837088s" totalImagesPullingTime="14.334582802s" totalInitContainerRuntime="108.285291ms" isStatefulPod=true podCreationTimestamp="2026-07-15 23:37:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 23:37:28.299252425 +0000 UTC m=+63.672710396" watchObservedRunningTime="2026-07-15 23:37:28.299837088 +0000 UTC m=+63.673295047" Jul 15 23:37:29.625951 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:29.625902 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:37:29.724667 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:29.724578 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:37:31.730034 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:31.730005 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d1c182db-bcdc-40a1-bb82-4f8b64173e43-metrics-certs\") pod \"network-metrics-daemon-7n4t9\" (UID: \"d1c182db-bcdc-40a1-bb82-4f8b64173e43\") " pod="openshift-multus/network-metrics-daemon-7n4t9" Jul 15 23:37:31.732503 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:31.732477 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 15 23:37:31.743725 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:31.743700 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d1c182db-bcdc-40a1-bb82-4f8b64173e43-metrics-certs\") pod \"network-metrics-daemon-7n4t9\" (UID: \"d1c182db-bcdc-40a1-bb82-4f8b64173e43\") " pod="openshift-multus/network-metrics-daemon-7n4t9" Jul 15 23:37:31.831389 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:31.831354 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v47j5\" (UniqueName: \"kubernetes.io/projected/b0b0bc56-c9c9-4a96-a97b-15f00f3b2099-kube-api-access-v47j5\") pod \"network-check-target-mp4pr\" (UID: \"b0b0bc56-c9c9-4a96-a97b-15f00f3b2099\") " pod="openshift-network-diagnostics/network-check-target-mp4pr" Jul 15 23:37:31.831570 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:31.831425 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/10a402bd-5fc8-41c2-852a-5df876f74bdb-original-pull-secret\") pod \"global-pull-secret-syncer-4zw2v\" (UID: \"10a402bd-5fc8-41c2-852a-5df876f74bdb\") " pod="kube-system/global-pull-secret-syncer-4zw2v" Jul 15 23:37:31.833904 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:31.833881 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 15 23:37:31.834046 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:31.833879 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 15 23:37:31.844732 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:31.843874 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 15 23:37:31.844732 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:31.844573 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/10a402bd-5fc8-41c2-852a-5df876f74bdb-original-pull-secret\") pod \"global-pull-secret-syncer-4zw2v\" (UID: \"10a402bd-5fc8-41c2-852a-5df876f74bdb\") " pod="kube-system/global-pull-secret-syncer-4zw2v" Jul 15 23:37:31.855194 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:31.855166 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v47j5\" (UniqueName: \"kubernetes.io/projected/b0b0bc56-c9c9-4a96-a97b-15f00f3b2099-kube-api-access-v47j5\") pod \"network-check-target-mp4pr\" (UID: \"b0b0bc56-c9c9-4a96-a97b-15f00f3b2099\") " pod="openshift-network-diagnostics/network-check-target-mp4pr" Jul 15 23:37:32.009631 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:32.009549 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-qkb4z\"" Jul 15 23:37:32.017530 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:32.017467 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mp4pr" Jul 15 23:37:32.018308 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:32.018286 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-s2fsj\"" Jul 15 23:37:32.024135 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:32.024110 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4zw2v" Jul 15 23:37:32.027254 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:32.027238 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7n4t9" Jul 15 23:37:33.224188 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:33.224148 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:37:33.470980 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:33.470931 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:37:33.573143 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:33.573094 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:37:33.825158 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:33.825053 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:37:33.879601 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:33.879545 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:37:37.932285 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:37.932257 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-4zw2v"] Jul 15 23:37:37.981749 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:37:37.981713 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10a402bd_5fc8_41c2_852a_5df876f74bdb.slice/crio-fa174c532c304bbb57db1cec4f539559ae0a11a7272e6c96adcb809f43479302 WatchSource:0}: Error finding container fa174c532c304bbb57db1cec4f539559ae0a11a7272e6c96adcb809f43479302: Status 404 returned error can't find the container with id fa174c532c304bbb57db1cec4f539559ae0a11a7272e6c96adcb809f43479302 Jul 15 23:37:38.119832 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:38.119726 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-7n4t9"] Jul 15 23:37:38.128541 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:37:38.128508 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1c182db_bcdc_40a1_bb82_4f8b64173e43.slice/crio-ee8f2a94b612cc33ae609b6cb49c7f085c25540e084e19f2d24bb244368f0e8a WatchSource:0}: Error finding container ee8f2a94b612cc33ae609b6cb49c7f085c25540e084e19f2d24bb244368f0e8a: Status 404 returned error can't find the container with id ee8f2a94b612cc33ae609b6cb49c7f085c25540e084e19f2d24bb244368f0e8a Jul 15 23:37:38.143376 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:38.143350 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-mp4pr"] Jul 15 23:37:38.145548 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:37:38.145525 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0b0bc56_c9c9_4a96_a97b_15f00f3b2099.slice/crio-5d6c3f9f5eb7d675c33a18eae67ca51d5ad2092111838d99d2215c3d3f67e1d5 WatchSource:0}: Error finding container 5d6c3f9f5eb7d675c33a18eae67ca51d5ad2092111838d99d2215c3d3f67e1d5: Status 404 returned error can't find the container with id 5d6c3f9f5eb7d675c33a18eae67ca51d5ad2092111838d99d2215c3d3f67e1d5 Jul 15 23:37:38.341664 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:38.341586 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-b7fc86bbb-vrm2p" event={"ID":"cc051ada-7c71-4701-94cf-914ad0e6c65c","Type":"ContainerStarted","Data":"8e97c991c422d85fc543667a11757085ac49eb66b5eebfe5b6fdc3ec6b0e8112"} Jul 15 23:37:38.343747 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:38.343717 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a881fb94-9a0b-4c46-b5fc-d841398d576d","Type":"ContainerStarted","Data":"a4a887aaf513cd8202fe2c033d87ef3ba8df280b24c5113b3738377bcf5a5ee8"} Jul 15 23:37:38.345356 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:38.345330 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-7cb86fd7bc-4bphf" event={"ID":"692b9b9d-1237-4e9c-bf86-0255ab7732b7","Type":"ContainerStarted","Data":"d3952b5d146b49d61bfb3aef292b7a5a041858e13138ecc7dbd516f0f9879211"} Jul 15 23:37:38.349133 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:38.349105 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5e3e5aa3-8bc5-49ff-8706-6107d3e3d2bb","Type":"ContainerStarted","Data":"fb9dbad6497ba7894d5fb5c72ac48f4219df84a9c0863367dead5834e92eb7e5"} Jul 15 23:37:38.350839 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:38.350812 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-gkz55" event={"ID":"e0f740d6-d99f-48e0-8da0-66e359d9fb48","Type":"ContainerStarted","Data":"c5e7687fe01775f0c6414e5184a043e1c25a3b801d2c3dd227cbd9726a2bda2a"} Jul 15 23:37:38.351160 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:38.351138 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-gkz55" Jul 15 23:37:38.352201 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:38.352178 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-4zw2v" event={"ID":"10a402bd-5fc8-41c2-852a-5df876f74bdb","Type":"ContainerStarted","Data":"fa174c532c304bbb57db1cec4f539559ae0a11a7272e6c96adcb809f43479302"} Jul 15 23:37:38.354209 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:38.354186 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-786d565755-fqg7t" event={"ID":"7f775c78-eab6-437d-a2e9-291049e51b94","Type":"ContainerStarted","Data":"79edaadf97c8b9e68a9a31312cbaf08364e3525628578003b632f2ff520fb963"} Jul 15 23:37:38.354316 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:38.354238 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-786d565755-fqg7t" event={"ID":"7f775c78-eab6-437d-a2e9-291049e51b94","Type":"ContainerStarted","Data":"48abf601fdf03e5f7fb67e0473e02b75fb8d70a8c14c15a7a9d0504568b5c155"} Jul 15 23:37:38.355428 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:38.355406 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-mp4pr" event={"ID":"b0b0bc56-c9c9-4a96-a97b-15f00f3b2099","Type":"ContainerStarted","Data":"5d6c3f9f5eb7d675c33a18eae67ca51d5ad2092111838d99d2215c3d3f67e1d5"} Jul 15 23:37:38.356780 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:38.356739 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-gkz55" Jul 15 23:37:38.357102 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:38.357081 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-7n4t9" event={"ID":"d1c182db-bcdc-40a1-bb82-4f8b64173e43","Type":"ContainerStarted","Data":"ee8f2a94b612cc33ae609b6cb49c7f085c25540e084e19f2d24bb244368f0e8a"} Jul 15 23:37:38.383535 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:38.383483 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-gkz55" podStartSLOduration=14.470931591 podStartE2EDuration="24.383465751s" totalImagesPullingTime="9.91253416s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 23:37:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 23:37:38.381678509 +0000 UTC m=+73.755136472" watchObservedRunningTime="2026-07-15 23:37:38.383465751 +0000 UTC m=+73.756923714" Jul 15 23:37:38.383680 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:38.383622 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-7cb86fd7bc-4bphf" podStartSLOduration=14.319629943 podStartE2EDuration="25.383617707s" totalImagesPullingTime="11.063987764s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 23:37:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 23:37:38.366210238 +0000 UTC m=+73.739668203" watchObservedRunningTime="2026-07-15 23:37:38.383617707 +0000 UTC m=+73.757075669" Jul 15 23:37:39.370268 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:39.370109 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5e3e5aa3-8bc5-49ff-8706-6107d3e3d2bb","Type":"ContainerStarted","Data":"a3cfe9a8d9b94f2fb69268dc3f0dd81c5e1290f049e4cd8358062fbf08639053"} Jul 15 23:37:39.370268 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:39.370230 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5e3e5aa3-8bc5-49ff-8706-6107d3e3d2bb","Type":"ContainerStarted","Data":"e10c2697bb2dd4dc1576fda5ed268bf9256c3e78c729a38a70587f488dc73c8b"} Jul 15 23:37:39.370803 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:39.370274 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5e3e5aa3-8bc5-49ff-8706-6107d3e3d2bb","Type":"ContainerStarted","Data":"15e9b037670f1c0e45c820cb92270698797d7139a20d765789f519f54f8ead77"} Jul 15 23:37:39.370803 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:39.370313 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5e3e5aa3-8bc5-49ff-8706-6107d3e3d2bb","Type":"ContainerStarted","Data":"dcfd2775981fa0a98baffb79b279dd05931faeaf553ddb4c2e211978d4ca909c"} Jul 15 23:37:39.370803 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:39.370351 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5e3e5aa3-8bc5-49ff-8706-6107d3e3d2bb","Type":"ContainerStarted","Data":"7c12b9be4a933b3f011075e9b9a97af663bcc7f93257c38bdac3ccfa931c7758"} Jul 15 23:37:39.374015 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:39.373983 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5q8md" event={"ID":"409f6854-b231-41aa-8715-a300ee9aa09e","Type":"ContainerStarted","Data":"1b9e3cd691690a6366c660052193bc2049450fcd2dd972c2be6e3cd620106c50"} Jul 15 23:37:39.379033 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:39.379007 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-786d565755-fqg7t" event={"ID":"7f775c78-eab6-437d-a2e9-291049e51b94","Type":"ContainerStarted","Data":"6f82a03f5dd7f310c0a4958a4ae893cc8860509a352f0a1c046ab95664c312f7"} Jul 15 23:37:39.384440 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:39.384413 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-b7fc86bbb-vrm2p" event={"ID":"cc051ada-7c71-4701-94cf-914ad0e6c65c","Type":"ContainerStarted","Data":"ddfc886d04db678df35707c4af137e778a6c7087226adc2b343ccc7ec03d1b42"} Jul 15 23:37:39.384541 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:39.384487 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-b7fc86bbb-vrm2p" event={"ID":"cc051ada-7c71-4701-94cf-914ad0e6c65c","Type":"ContainerStarted","Data":"82a8cb3cf06de09feb568ec86b7caa3e06ac32812554cdc5cfed48132ff3c580"} Jul 15 23:37:39.385808 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:39.385788 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-b7fc86bbb-vrm2p" Jul 15 23:37:39.391836 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:39.391339 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a881fb94-9a0b-4c46-b5fc-d841398d576d","Type":"ContainerStarted","Data":"1555d6bba92a06a023fbe9901c90b4d739ced83bc3b4e3c5ba2acdff57649475"} Jul 15 23:37:39.391836 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:39.391402 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a881fb94-9a0b-4c46-b5fc-d841398d576d","Type":"ContainerStarted","Data":"f2eef3071405d6109930b1123dc53e48dee6c71643c6b82219f38fdf07154180"} Jul 15 23:37:39.391836 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:39.391442 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a881fb94-9a0b-4c46-b5fc-d841398d576d","Type":"ContainerStarted","Data":"63c7e833b7f4dd49f6b4b2057181362d71af0d8fdd0d48039672e403ee88c365"} Jul 15 23:37:39.391836 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:39.391469 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a881fb94-9a0b-4c46-b5fc-d841398d576d","Type":"ContainerStarted","Data":"623e6174e579792885b393eef036d66bec7d8fe27570aa09b14e3d7cf266d7a5"} Jul 15 23:37:39.391836 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:39.391506 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a881fb94-9a0b-4c46-b5fc-d841398d576d","Type":"ContainerStarted","Data":"cd1c919f29a0f532795c23a6f3e6172ab0cdff76e2d54de23e34d939a47dec1b"} Jul 15 23:37:39.393598 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:39.393577 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-b7fc86bbb-vrm2p" Jul 15 23:37:39.398691 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:39.398643 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=4.70420252 podStartE2EDuration="29.398624699s" totalImagesPullingTime="24.681029006s" totalInitContainerRuntime="13.393173ms" isStatefulPod=false podCreationTimestamp="2026-07-15 23:37:10 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 23:37:39.395264289 +0000 UTC m=+74.768722289" watchObservedRunningTime="2026-07-15 23:37:39.398624699 +0000 UTC m=+74.772082664" Jul 15 23:37:39.432161 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:39.432102 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=13.652322165 podStartE2EDuration="24.432083017s" totalImagesPullingTime="10.744145852s" totalInitContainerRuntime="35.615ms" isStatefulPod=false podCreationTimestamp="2026-07-15 23:37:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 23:37:39.430112255 +0000 UTC m=+74.803570219" watchObservedRunningTime="2026-07-15 23:37:39.432083017 +0000 UTC m=+74.805540981" Jul 15 23:37:39.453142 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:39.452111 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-b7fc86bbb-vrm2p" podStartSLOduration=5.776676626 podStartE2EDuration="28.452091866s" totalImagesPullingTime="22.67541524s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 23:37:11 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 23:37:39.450392071 +0000 UTC m=+74.823850053" watchObservedRunningTime="2026-07-15 23:37:39.452091866 +0000 UTC m=+74.825549829" Jul 15 23:37:39.470984 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:39.470913 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-786d565755-fqg7t" podStartSLOduration=15.563422286 podStartE2EDuration="25.47089224s" totalImagesPullingTime="9.907469954s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 23:37:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 23:37:39.468487725 +0000 UTC m=+74.841945703" watchObservedRunningTime="2026-07-15 23:37:39.47089224 +0000 UTC m=+74.844350206" Jul 15 23:37:40.090412 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:40.088866 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:37:40.139462 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:40.139430 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:37:40.273038 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:40.273006 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7d6d44479b-g6lw7"] Jul 15 23:37:40.301494 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:40.301060 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7d6d44479b-g6lw7"] Jul 15 23:37:40.301494 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:40.301180 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7d6d44479b-g6lw7" Jul 15 23:37:40.315145 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:40.314960 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:37:40.319239 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:40.319212 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 15 23:37:40.407007 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:40.406564 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-7n4t9" event={"ID":"d1c182db-bcdc-40a1-bb82-4f8b64173e43","Type":"ContainerStarted","Data":"b1fd49421578807decf3ff4c9d1cf0ee4eb438c19df650dbf9c13c9b3da51563"} Jul 15 23:37:40.407007 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:40.406734 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-7n4t9" event={"ID":"d1c182db-bcdc-40a1-bb82-4f8b64173e43","Type":"ContainerStarted","Data":"815f3f816b3eb43a8b4954a2aa97e78c8438d31164facfbfedd64d97f3955634"} Jul 15 23:37:40.423269 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:40.423231 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4a3b0780-bf04-4522-92df-8b09e2f29bcb-console-config\") pod \"console-7d6d44479b-g6lw7\" (UID: \"4a3b0780-bf04-4522-92df-8b09e2f29bcb\") " pod="openshift-console/console-7d6d44479b-g6lw7" Jul 15 23:37:40.423451 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:40.423367 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4a3b0780-bf04-4522-92df-8b09e2f29bcb-trusted-ca-bundle\") pod \"console-7d6d44479b-g6lw7\" (UID: \"4a3b0780-bf04-4522-92df-8b09e2f29bcb\") " pod="openshift-console/console-7d6d44479b-g6lw7" Jul 15 23:37:40.424018 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:40.423712 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4a3b0780-bf04-4522-92df-8b09e2f29bcb-console-oauth-config\") pod \"console-7d6d44479b-g6lw7\" (UID: \"4a3b0780-bf04-4522-92df-8b09e2f29bcb\") " pod="openshift-console/console-7d6d44479b-g6lw7" Jul 15 23:37:40.424018 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:40.423754 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4a3b0780-bf04-4522-92df-8b09e2f29bcb-oauth-serving-cert\") pod \"console-7d6d44479b-g6lw7\" (UID: \"4a3b0780-bf04-4522-92df-8b09e2f29bcb\") " pod="openshift-console/console-7d6d44479b-g6lw7" Jul 15 23:37:40.424018 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:40.423785 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5sc9\" (UniqueName: \"kubernetes.io/projected/4a3b0780-bf04-4522-92df-8b09e2f29bcb-kube-api-access-c5sc9\") pod \"console-7d6d44479b-g6lw7\" (UID: \"4a3b0780-bf04-4522-92df-8b09e2f29bcb\") " pod="openshift-console/console-7d6d44479b-g6lw7" Jul 15 23:37:40.424018 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:40.423870 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4a3b0780-bf04-4522-92df-8b09e2f29bcb-console-serving-cert\") pod \"console-7d6d44479b-g6lw7\" (UID: \"4a3b0780-bf04-4522-92df-8b09e2f29bcb\") " pod="openshift-console/console-7d6d44479b-g6lw7" Jul 15 23:37:40.424018 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:40.423898 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4a3b0780-bf04-4522-92df-8b09e2f29bcb-service-ca\") pod \"console-7d6d44479b-g6lw7\" (UID: \"4a3b0780-bf04-4522-92df-8b09e2f29bcb\") " pod="openshift-console/console-7d6d44479b-g6lw7" Jul 15 23:37:40.429474 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:40.429351 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-7n4t9" podStartSLOduration=73.56200584 podStartE2EDuration="1m15.429331935s" totalImagesPullingTime="1.867326095s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 23:36:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 23:37:40.42902355 +0000 UTC m=+75.802481514" watchObservedRunningTime="2026-07-15 23:37:40.429331935 +0000 UTC m=+75.802789899" Jul 15 23:37:40.524375 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:40.524218 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:37:40.524583 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:40.524375 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4a3b0780-bf04-4522-92df-8b09e2f29bcb-service-ca\") pod \"console-7d6d44479b-g6lw7\" (UID: \"4a3b0780-bf04-4522-92df-8b09e2f29bcb\") " pod="openshift-console/console-7d6d44479b-g6lw7" Jul 15 23:37:40.526045 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:40.525150 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4a3b0780-bf04-4522-92df-8b09e2f29bcb-service-ca\") pod \"console-7d6d44479b-g6lw7\" (UID: \"4a3b0780-bf04-4522-92df-8b09e2f29bcb\") " pod="openshift-console/console-7d6d44479b-g6lw7" Jul 15 23:37:40.526045 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:40.525698 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4a3b0780-bf04-4522-92df-8b09e2f29bcb-console-config\") pod \"console-7d6d44479b-g6lw7\" (UID: \"4a3b0780-bf04-4522-92df-8b09e2f29bcb\") " pod="openshift-console/console-7d6d44479b-g6lw7" Jul 15 23:37:40.526045 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:40.525740 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4a3b0780-bf04-4522-92df-8b09e2f29bcb-trusted-ca-bundle\") pod \"console-7d6d44479b-g6lw7\" (UID: \"4a3b0780-bf04-4522-92df-8b09e2f29bcb\") " pod="openshift-console/console-7d6d44479b-g6lw7" Jul 15 23:37:40.526561 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:40.526535 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4a3b0780-bf04-4522-92df-8b09e2f29bcb-console-oauth-config\") pod \"console-7d6d44479b-g6lw7\" (UID: \"4a3b0780-bf04-4522-92df-8b09e2f29bcb\") " pod="openshift-console/console-7d6d44479b-g6lw7" Jul 15 23:37:40.526657 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:40.526599 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4a3b0780-bf04-4522-92df-8b09e2f29bcb-trusted-ca-bundle\") pod \"console-7d6d44479b-g6lw7\" (UID: \"4a3b0780-bf04-4522-92df-8b09e2f29bcb\") " pod="openshift-console/console-7d6d44479b-g6lw7" Jul 15 23:37:40.526657 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:40.526622 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4a3b0780-bf04-4522-92df-8b09e2f29bcb-console-config\") pod \"console-7d6d44479b-g6lw7\" (UID: \"4a3b0780-bf04-4522-92df-8b09e2f29bcb\") " pod="openshift-console/console-7d6d44479b-g6lw7" Jul 15 23:37:40.527648 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:40.527380 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4a3b0780-bf04-4522-92df-8b09e2f29bcb-oauth-serving-cert\") pod \"console-7d6d44479b-g6lw7\" (UID: \"4a3b0780-bf04-4522-92df-8b09e2f29bcb\") " pod="openshift-console/console-7d6d44479b-g6lw7" Jul 15 23:37:40.529455 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:40.527775 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5sc9\" (UniqueName: \"kubernetes.io/projected/4a3b0780-bf04-4522-92df-8b09e2f29bcb-kube-api-access-c5sc9\") pod \"console-7d6d44479b-g6lw7\" (UID: \"4a3b0780-bf04-4522-92df-8b09e2f29bcb\") " pod="openshift-console/console-7d6d44479b-g6lw7" Jul 15 23:37:40.529455 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:40.528050 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4a3b0780-bf04-4522-92df-8b09e2f29bcb-oauth-serving-cert\") pod \"console-7d6d44479b-g6lw7\" (UID: \"4a3b0780-bf04-4522-92df-8b09e2f29bcb\") " pod="openshift-console/console-7d6d44479b-g6lw7" Jul 15 23:37:40.529455 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:40.528256 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4a3b0780-bf04-4522-92df-8b09e2f29bcb-console-serving-cert\") pod \"console-7d6d44479b-g6lw7\" (UID: \"4a3b0780-bf04-4522-92df-8b09e2f29bcb\") " pod="openshift-console/console-7d6d44479b-g6lw7" Jul 15 23:37:40.529809 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:40.529787 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4a3b0780-bf04-4522-92df-8b09e2f29bcb-console-oauth-config\") pod \"console-7d6d44479b-g6lw7\" (UID: \"4a3b0780-bf04-4522-92df-8b09e2f29bcb\") " pod="openshift-console/console-7d6d44479b-g6lw7" Jul 15 23:37:40.533982 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:40.533940 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4a3b0780-bf04-4522-92df-8b09e2f29bcb-console-serving-cert\") pod \"console-7d6d44479b-g6lw7\" (UID: \"4a3b0780-bf04-4522-92df-8b09e2f29bcb\") " pod="openshift-console/console-7d6d44479b-g6lw7" Jul 15 23:37:40.537234 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:40.537210 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5sc9\" (UniqueName: \"kubernetes.io/projected/4a3b0780-bf04-4522-92df-8b09e2f29bcb-kube-api-access-c5sc9\") pod \"console-7d6d44479b-g6lw7\" (UID: \"4a3b0780-bf04-4522-92df-8b09e2f29bcb\") " pod="openshift-console/console-7d6d44479b-g6lw7" Jul 15 23:37:40.581692 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:40.581655 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:37:40.619554 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:40.619524 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7d6d44479b-g6lw7" Jul 15 23:37:40.852580 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:40.852159 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7d6d44479b-g6lw7"] Jul 15 23:37:40.858740 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:37:40.858711 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a3b0780_bf04_4522_92df_8b09e2f29bcb.slice/crio-832bf46e1a027bc5db0a3627e5ac0e2812614c5deee619fbc72937988da8703b WatchSource:0}: Error finding container 832bf46e1a027bc5db0a3627e5ac0e2812614c5deee619fbc72937988da8703b: Status 404 returned error can't find the container with id 832bf46e1a027bc5db0a3627e5ac0e2812614c5deee619fbc72937988da8703b Jul 15 23:37:41.050470 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:41.050392 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:37:41.106074 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:41.105994 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:37:41.201999 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:41.200704 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 15 23:37:41.417723 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:41.417641 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7d6d44479b-g6lw7" event={"ID":"4a3b0780-bf04-4522-92df-8b09e2f29bcb","Type":"ContainerStarted","Data":"644c17f4386873fb31adbc1e5c7543804a12939ae20f4db42d46c2c843d4815a"} Jul 15 23:37:41.417723 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:41.417715 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7d6d44479b-g6lw7" event={"ID":"4a3b0780-bf04-4522-92df-8b09e2f29bcb","Type":"ContainerStarted","Data":"832bf46e1a027bc5db0a3627e5ac0e2812614c5deee619fbc72937988da8703b"} Jul 15 23:37:41.435166 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:41.435109 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7d6d44479b-g6lw7" podStartSLOduration=1.435089893 podStartE2EDuration="1.435089893s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 23:37:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 23:37:41.433000077 +0000 UTC m=+76.806458042" watchObservedRunningTime="2026-07-15 23:37:41.435089893 +0000 UTC m=+76.808547858" Jul 15 23:37:47.451117 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:47.451081 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-4zw2v" event={"ID":"10a402bd-5fc8-41c2-852a-5df876f74bdb","Type":"ContainerStarted","Data":"d6131d7cd478a1f3ecf018aa608605fe5323a71cca926733054c0460c44b6b90"} Jul 15 23:37:47.453309 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:47.453285 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5q8md" event={"ID":"409f6854-b231-41aa-8715-a300ee9aa09e","Type":"ContainerStarted","Data":"bb031ed5acf1a21468d64531c07a5d601d0450511717492085d27e7d91d3e8ef"} Jul 15 23:37:47.454939 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:47.454916 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-mp4pr" event={"ID":"b0b0bc56-c9c9-4a96-a97b-15f00f3b2099","Type":"ContainerStarted","Data":"922056201e422b25cb1083eced9d76de0c502d02208245e1c727d4420c739fe1"} Jul 15 23:37:47.455056 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:47.455000 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-mp4pr" Jul 15 23:37:47.465234 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:47.465192 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-4zw2v" podStartSLOduration=74.020991631 podStartE2EDuration="1m22.465177s" totalImagesPullingTime="8.444185369s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 23:36:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 23:37:47.464889244 +0000 UTC m=+82.838347204" watchObservedRunningTime="2026-07-15 23:37:47.465177 +0000 UTC m=+82.838634963" Jul 15 23:37:47.479107 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:47.479067 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-mp4pr" podStartSLOduration=74.209131696 podStartE2EDuration="1m22.479052738s" totalImagesPullingTime="8.269921042s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 23:36:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 23:37:47.477457904 +0000 UTC m=+82.850915867" watchObservedRunningTime="2026-07-15 23:37:47.479052738 +0000 UTC m=+82.852510737" Jul 15 23:37:47.494571 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:47.494516 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-5q8md" podStartSLOduration=9.832386894999999 podStartE2EDuration="46.494480434s" totalImagesPullingTime="36.662093539s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 23:37:01 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 23:37:47.493071509 +0000 UTC m=+82.866529472" watchObservedRunningTime="2026-07-15 23:37:47.494480434 +0000 UTC m=+82.867938398" Jul 15 23:37:50.620844 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:50.620808 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7d6d44479b-g6lw7" Jul 15 23:37:50.621245 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:50.620867 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7d6d44479b-g6lw7" Jul 15 23:37:50.625587 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:50.625566 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7d6d44479b-g6lw7" Jul 15 23:37:51.474456 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:51.474430 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7d6d44479b-g6lw7" Jul 15 23:37:51.618245 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:51.618214 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:37:51.668097 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:51.668073 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:37:51.719021 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:51.718992 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:37:52.583650 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:52.583594 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-75d9954bb7-rlshl" podUID="5efd853d-ff2f-43d9-bbe8-fd8ee02465ec" containerName="console" containerID="cri-o://90ed71e26eb77a56336d1451a0c80808798e2e4a46fc387193582c8282b05c04" gracePeriod=15 Jul 15 23:37:52.858449 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:52.858391 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-75d9954bb7-rlshl" Jul 15 23:37:52.946411 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:52.946383 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qs7xp\" (UniqueName: \"kubernetes.io/projected/5efd853d-ff2f-43d9-bbe8-fd8ee02465ec-kube-api-access-qs7xp\") pod \"5efd853d-ff2f-43d9-bbe8-fd8ee02465ec\" (UID: \"5efd853d-ff2f-43d9-bbe8-fd8ee02465ec\") " Jul 15 23:37:52.946580 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:52.946444 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5efd853d-ff2f-43d9-bbe8-fd8ee02465ec-console-oauth-config\") pod \"5efd853d-ff2f-43d9-bbe8-fd8ee02465ec\" (UID: \"5efd853d-ff2f-43d9-bbe8-fd8ee02465ec\") " Jul 15 23:37:52.946580 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:52.946500 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5efd853d-ff2f-43d9-bbe8-fd8ee02465ec-console-serving-cert\") pod \"5efd853d-ff2f-43d9-bbe8-fd8ee02465ec\" (UID: \"5efd853d-ff2f-43d9-bbe8-fd8ee02465ec\") " Jul 15 23:37:52.946580 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:52.946533 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5efd853d-ff2f-43d9-bbe8-fd8ee02465ec-oauth-serving-cert\") pod \"5efd853d-ff2f-43d9-bbe8-fd8ee02465ec\" (UID: \"5efd853d-ff2f-43d9-bbe8-fd8ee02465ec\") " Jul 15 23:37:52.946580 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:52.946554 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5efd853d-ff2f-43d9-bbe8-fd8ee02465ec-service-ca\") pod \"5efd853d-ff2f-43d9-bbe8-fd8ee02465ec\" (UID: \"5efd853d-ff2f-43d9-bbe8-fd8ee02465ec\") " Jul 15 23:37:52.946779 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:52.946588 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5efd853d-ff2f-43d9-bbe8-fd8ee02465ec-console-config\") pod \"5efd853d-ff2f-43d9-bbe8-fd8ee02465ec\" (UID: \"5efd853d-ff2f-43d9-bbe8-fd8ee02465ec\") " Jul 15 23:37:52.946960 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:52.946925 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5efd853d-ff2f-43d9-bbe8-fd8ee02465ec-service-ca" (OuterVolumeSpecName: "service-ca") pod "5efd853d-ff2f-43d9-bbe8-fd8ee02465ec" (UID: "5efd853d-ff2f-43d9-bbe8-fd8ee02465ec"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 23:37:52.947134 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:52.946953 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5efd853d-ff2f-43d9-bbe8-fd8ee02465ec-console-config" (OuterVolumeSpecName: "console-config") pod "5efd853d-ff2f-43d9-bbe8-fd8ee02465ec" (UID: "5efd853d-ff2f-43d9-bbe8-fd8ee02465ec"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 23:37:52.947134 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:52.946937 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5efd853d-ff2f-43d9-bbe8-fd8ee02465ec-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "5efd853d-ff2f-43d9-bbe8-fd8ee02465ec" (UID: "5efd853d-ff2f-43d9-bbe8-fd8ee02465ec"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 23:37:52.948764 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:52.948740 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5efd853d-ff2f-43d9-bbe8-fd8ee02465ec-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "5efd853d-ff2f-43d9-bbe8-fd8ee02465ec" (UID: "5efd853d-ff2f-43d9-bbe8-fd8ee02465ec"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 23:37:52.948993 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:52.948831 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5efd853d-ff2f-43d9-bbe8-fd8ee02465ec-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "5efd853d-ff2f-43d9-bbe8-fd8ee02465ec" (UID: "5efd853d-ff2f-43d9-bbe8-fd8ee02465ec"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 23:37:52.948993 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:52.948877 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5efd853d-ff2f-43d9-bbe8-fd8ee02465ec-kube-api-access-qs7xp" (OuterVolumeSpecName: "kube-api-access-qs7xp") pod "5efd853d-ff2f-43d9-bbe8-fd8ee02465ec" (UID: "5efd853d-ff2f-43d9-bbe8-fd8ee02465ec"). InnerVolumeSpecName "kube-api-access-qs7xp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 23:37:53.048188 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:53.048162 2563 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5efd853d-ff2f-43d9-bbe8-fd8ee02465ec-console-oauth-config\") on node \"ip-10-0-128-197.ec2.internal\" DevicePath \"\"" Jul 15 23:37:53.048188 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:53.048186 2563 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5efd853d-ff2f-43d9-bbe8-fd8ee02465ec-console-serving-cert\") on node \"ip-10-0-128-197.ec2.internal\" DevicePath \"\"" Jul 15 23:37:53.048340 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:53.048196 2563 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5efd853d-ff2f-43d9-bbe8-fd8ee02465ec-oauth-serving-cert\") on node \"ip-10-0-128-197.ec2.internal\" DevicePath \"\"" Jul 15 23:37:53.048340 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:53.048205 2563 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5efd853d-ff2f-43d9-bbe8-fd8ee02465ec-service-ca\") on node \"ip-10-0-128-197.ec2.internal\" DevicePath \"\"" Jul 15 23:37:53.048340 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:53.048213 2563 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5efd853d-ff2f-43d9-bbe8-fd8ee02465ec-console-config\") on node \"ip-10-0-128-197.ec2.internal\" DevicePath \"\"" Jul 15 23:37:53.048340 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:53.048220 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qs7xp\" (UniqueName: \"kubernetes.io/projected/5efd853d-ff2f-43d9-bbe8-fd8ee02465ec-kube-api-access-qs7xp\") on node \"ip-10-0-128-197.ec2.internal\" DevicePath \"\"" Jul 15 23:37:53.482803 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:53.482770 2563 generic.go:356] "Generic (PLEG): container finished" podID="5efd853d-ff2f-43d9-bbe8-fd8ee02465ec" containerID="90ed71e26eb77a56336d1451a0c80808798e2e4a46fc387193582c8282b05c04" exitCode=0 Jul 15 23:37:53.483084 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:53.482821 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-75d9954bb7-rlshl" Jul 15 23:37:53.483084 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:53.482835 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-75d9954bb7-rlshl" event={"ID":"5efd853d-ff2f-43d9-bbe8-fd8ee02465ec","Type":"ContainerDied","Data":"90ed71e26eb77a56336d1451a0c80808798e2e4a46fc387193582c8282b05c04"} Jul 15 23:37:53.483084 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:53.482876 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-75d9954bb7-rlshl" event={"ID":"5efd853d-ff2f-43d9-bbe8-fd8ee02465ec","Type":"ContainerDied","Data":"3c01c2651f37f6ee2e194cd291c4d352173f6c4d363c57db1b9932bbaf21a3d3"} Jul 15 23:37:53.483084 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:53.482896 2563 scope.go:117] "RemoveContainer" containerID="90ed71e26eb77a56336d1451a0c80808798e2e4a46fc387193582c8282b05c04" Jul 15 23:37:53.491421 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:53.491401 2563 scope.go:117] "RemoveContainer" containerID="90ed71e26eb77a56336d1451a0c80808798e2e4a46fc387193582c8282b05c04" Jul 15 23:37:53.491699 ip-10-0-128-197 kubenswrapper[2563]: E0715 23:37:53.491678 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"90ed71e26eb77a56336d1451a0c80808798e2e4a46fc387193582c8282b05c04\": container with ID starting with 90ed71e26eb77a56336d1451a0c80808798e2e4a46fc387193582c8282b05c04 not found: ID does not exist" containerID="90ed71e26eb77a56336d1451a0c80808798e2e4a46fc387193582c8282b05c04" Jul 15 23:37:53.491781 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:53.491710 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90ed71e26eb77a56336d1451a0c80808798e2e4a46fc387193582c8282b05c04"} err="failed to get container status \"90ed71e26eb77a56336d1451a0c80808798e2e4a46fc387193582c8282b05c04\": rpc error: code = NotFound desc = could not find container \"90ed71e26eb77a56336d1451a0c80808798e2e4a46fc387193582c8282b05c04\": container with ID starting with 90ed71e26eb77a56336d1451a0c80808798e2e4a46fc387193582c8282b05c04 not found: ID does not exist" Jul 15 23:37:53.497757 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:53.497732 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-75d9954bb7-rlshl"] Jul 15 23:37:53.504767 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:53.504742 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-75d9954bb7-rlshl"] Jul 15 23:37:54.302701 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:54.302666 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-7cb86fd7bc-4bphf" Jul 15 23:37:54.303289 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:54.302731 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-7cb86fd7bc-4bphf" Jul 15 23:37:55.169640 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:55.169608 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5efd853d-ff2f-43d9-bbe8-fd8ee02465ec" path="/var/lib/kubelet/pods/5efd853d-ff2f-43d9-bbe8-fd8ee02465ec/volumes" Jul 15 23:37:59.192667 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:59.192633 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 15 23:37:59.251220 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:59.251196 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 15 23:37:59.526412 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:37:59.526377 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 15 23:38:06.422919 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:06.422873 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:38:12.829483 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:12.829448 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:38:14.307907 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:14.307879 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-7cb86fd7bc-4bphf" Jul 15 23:38:14.311734 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:14.311713 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-7cb86fd7bc-4bphf" Jul 15 23:38:15.053445 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:15.053407 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:38:15.106565 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:15.106531 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:38:18.461214 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:18.461187 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-mp4pr" Jul 15 23:38:19.292303 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:19.292267 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 15 23:38:19.292560 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:19.292538 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="5e3e5aa3-8bc5-49ff-8706-6107d3e3d2bb" containerName="alertmanager" containerID="cri-o://fb9dbad6497ba7894d5fb5c72ac48f4219df84a9c0863367dead5834e92eb7e5" gracePeriod=120 Jul 15 23:38:19.292661 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:19.292621 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="5e3e5aa3-8bc5-49ff-8706-6107d3e3d2bb" containerName="kube-rbac-proxy-metric" containerID="cri-o://e10c2697bb2dd4dc1576fda5ed268bf9256c3e78c729a38a70587f488dc73c8b" gracePeriod=120 Jul 15 23:38:19.292795 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:19.292656 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="5e3e5aa3-8bc5-49ff-8706-6107d3e3d2bb" containerName="prom-label-proxy" containerID="cri-o://a3cfe9a8d9b94f2fb69268dc3f0dd81c5e1290f049e4cd8358062fbf08639053" gracePeriod=120 Jul 15 23:38:19.292795 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:19.292687 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="5e3e5aa3-8bc5-49ff-8706-6107d3e3d2bb" containerName="config-reloader" containerID="cri-o://7c12b9be4a933b3f011075e9b9a97af663bcc7f93257c38bdac3ccfa931c7758" gracePeriod=120 Jul 15 23:38:19.292795 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:19.292733 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="5e3e5aa3-8bc5-49ff-8706-6107d3e3d2bb" containerName="kube-rbac-proxy" containerID="cri-o://15e9b037670f1c0e45c820cb92270698797d7139a20d765789f519f54f8ead77" gracePeriod=120 Jul 15 23:38:19.292941 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:19.292686 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="5e3e5aa3-8bc5-49ff-8706-6107d3e3d2bb" containerName="kube-rbac-proxy-web" containerID="cri-o://dcfd2775981fa0a98baffb79b279dd05931faeaf553ddb4c2e211978d4ca909c" gracePeriod=120 Jul 15 23:38:19.591250 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:19.591221 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 15 23:38:19.594760 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:19.594734 2563 generic.go:356] "Generic (PLEG): container finished" podID="5e3e5aa3-8bc5-49ff-8706-6107d3e3d2bb" containerID="a3cfe9a8d9b94f2fb69268dc3f0dd81c5e1290f049e4cd8358062fbf08639053" exitCode=0 Jul 15 23:38:19.594760 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:19.594755 2563 generic.go:356] "Generic (PLEG): container finished" podID="5e3e5aa3-8bc5-49ff-8706-6107d3e3d2bb" containerID="e10c2697bb2dd4dc1576fda5ed268bf9256c3e78c729a38a70587f488dc73c8b" exitCode=0 Jul 15 23:38:19.594760 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:19.594761 2563 generic.go:356] "Generic (PLEG): container finished" podID="5e3e5aa3-8bc5-49ff-8706-6107d3e3d2bb" containerID="15e9b037670f1c0e45c820cb92270698797d7139a20d765789f519f54f8ead77" exitCode=0 Jul 15 23:38:19.594760 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:19.594767 2563 generic.go:356] "Generic (PLEG): container finished" podID="5e3e5aa3-8bc5-49ff-8706-6107d3e3d2bb" containerID="dcfd2775981fa0a98baffb79b279dd05931faeaf553ddb4c2e211978d4ca909c" exitCode=0 Jul 15 23:38:19.595103 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:19.594772 2563 generic.go:356] "Generic (PLEG): container finished" podID="5e3e5aa3-8bc5-49ff-8706-6107d3e3d2bb" containerID="7c12b9be4a933b3f011075e9b9a97af663bcc7f93257c38bdac3ccfa931c7758" exitCode=0 Jul 15 23:38:19.595103 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:19.594780 2563 generic.go:356] "Generic (PLEG): container finished" podID="5e3e5aa3-8bc5-49ff-8706-6107d3e3d2bb" containerID="fb9dbad6497ba7894d5fb5c72ac48f4219df84a9c0863367dead5834e92eb7e5" exitCode=0 Jul 15 23:38:19.595103 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:19.594801 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 15 23:38:19.595103 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:19.594804 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5e3e5aa3-8bc5-49ff-8706-6107d3e3d2bb","Type":"ContainerDied","Data":"a3cfe9a8d9b94f2fb69268dc3f0dd81c5e1290f049e4cd8358062fbf08639053"} Jul 15 23:38:19.595103 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:19.594883 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5e3e5aa3-8bc5-49ff-8706-6107d3e3d2bb","Type":"ContainerDied","Data":"e10c2697bb2dd4dc1576fda5ed268bf9256c3e78c729a38a70587f488dc73c8b"} Jul 15 23:38:19.595103 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:19.594913 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5e3e5aa3-8bc5-49ff-8706-6107d3e3d2bb","Type":"ContainerDied","Data":"15e9b037670f1c0e45c820cb92270698797d7139a20d765789f519f54f8ead77"} Jul 15 23:38:19.595103 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:19.594943 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5e3e5aa3-8bc5-49ff-8706-6107d3e3d2bb","Type":"ContainerDied","Data":"dcfd2775981fa0a98baffb79b279dd05931faeaf553ddb4c2e211978d4ca909c"} Jul 15 23:38:19.595103 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:19.594954 2563 scope.go:117] "RemoveContainer" containerID="a3cfe9a8d9b94f2fb69268dc3f0dd81c5e1290f049e4cd8358062fbf08639053" Jul 15 23:38:19.595103 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:19.595006 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5e3e5aa3-8bc5-49ff-8706-6107d3e3d2bb","Type":"ContainerDied","Data":"7c12b9be4a933b3f011075e9b9a97af663bcc7f93257c38bdac3ccfa931c7758"} Jul 15 23:38:19.595103 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:19.595047 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5e3e5aa3-8bc5-49ff-8706-6107d3e3d2bb","Type":"ContainerDied","Data":"fb9dbad6497ba7894d5fb5c72ac48f4219df84a9c0863367dead5834e92eb7e5"} Jul 15 23:38:19.595103 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:19.595086 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5e3e5aa3-8bc5-49ff-8706-6107d3e3d2bb","Type":"ContainerDied","Data":"5cf03839cfe4212e881a3fe9ca9f7b3f27ac1cc32d685dbc3db8649f98d0e9f4"} Jul 15 23:38:19.602380 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:19.602362 2563 scope.go:117] "RemoveContainer" containerID="e10c2697bb2dd4dc1576fda5ed268bf9256c3e78c729a38a70587f488dc73c8b" Jul 15 23:38:19.609142 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:19.609128 2563 scope.go:117] "RemoveContainer" containerID="15e9b037670f1c0e45c820cb92270698797d7139a20d765789f519f54f8ead77" Jul 15 23:38:19.617628 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:19.617610 2563 scope.go:117] "RemoveContainer" containerID="dcfd2775981fa0a98baffb79b279dd05931faeaf553ddb4c2e211978d4ca909c" Jul 15 23:38:19.625655 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:19.625637 2563 scope.go:117] "RemoveContainer" containerID="7c12b9be4a933b3f011075e9b9a97af663bcc7f93257c38bdac3ccfa931c7758" Jul 15 23:38:19.635054 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:19.635031 2563 scope.go:117] "RemoveContainer" containerID="fb9dbad6497ba7894d5fb5c72ac48f4219df84a9c0863367dead5834e92eb7e5" Jul 15 23:38:19.642421 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:19.642393 2563 scope.go:117] "RemoveContainer" containerID="baa174551f012dbe92f015d620924fd4ab07cf84fbe2158b0919fd5626754bc6" Jul 15 23:38:19.651025 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:19.651008 2563 scope.go:117] "RemoveContainer" containerID="a3cfe9a8d9b94f2fb69268dc3f0dd81c5e1290f049e4cd8358062fbf08639053" Jul 15 23:38:19.651311 ip-10-0-128-197 kubenswrapper[2563]: E0715 23:38:19.651286 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3cfe9a8d9b94f2fb69268dc3f0dd81c5e1290f049e4cd8358062fbf08639053\": container with ID starting with a3cfe9a8d9b94f2fb69268dc3f0dd81c5e1290f049e4cd8358062fbf08639053 not found: ID does not exist" containerID="a3cfe9a8d9b94f2fb69268dc3f0dd81c5e1290f049e4cd8358062fbf08639053" Jul 15 23:38:19.651374 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:19.651314 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3cfe9a8d9b94f2fb69268dc3f0dd81c5e1290f049e4cd8358062fbf08639053"} err="failed to get container status \"a3cfe9a8d9b94f2fb69268dc3f0dd81c5e1290f049e4cd8358062fbf08639053\": rpc error: code = NotFound desc = could not find container \"a3cfe9a8d9b94f2fb69268dc3f0dd81c5e1290f049e4cd8358062fbf08639053\": container with ID starting with a3cfe9a8d9b94f2fb69268dc3f0dd81c5e1290f049e4cd8358062fbf08639053 not found: ID does not exist" Jul 15 23:38:19.651374 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:19.651332 2563 scope.go:117] "RemoveContainer" containerID="e10c2697bb2dd4dc1576fda5ed268bf9256c3e78c729a38a70587f488dc73c8b" Jul 15 23:38:19.651597 ip-10-0-128-197 kubenswrapper[2563]: E0715 23:38:19.651580 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e10c2697bb2dd4dc1576fda5ed268bf9256c3e78c729a38a70587f488dc73c8b\": container with ID starting with e10c2697bb2dd4dc1576fda5ed268bf9256c3e78c729a38a70587f488dc73c8b not found: ID does not exist" containerID="e10c2697bb2dd4dc1576fda5ed268bf9256c3e78c729a38a70587f488dc73c8b" Jul 15 23:38:19.651654 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:19.651606 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e10c2697bb2dd4dc1576fda5ed268bf9256c3e78c729a38a70587f488dc73c8b"} err="failed to get container status \"e10c2697bb2dd4dc1576fda5ed268bf9256c3e78c729a38a70587f488dc73c8b\": rpc error: code = NotFound desc = could not find container \"e10c2697bb2dd4dc1576fda5ed268bf9256c3e78c729a38a70587f488dc73c8b\": container with ID starting with e10c2697bb2dd4dc1576fda5ed268bf9256c3e78c729a38a70587f488dc73c8b not found: ID does not exist" Jul 15 23:38:19.651654 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:19.651628 2563 scope.go:117] "RemoveContainer" containerID="15e9b037670f1c0e45c820cb92270698797d7139a20d765789f519f54f8ead77" Jul 15 23:38:19.651921 ip-10-0-128-197 kubenswrapper[2563]: E0715 23:38:19.651901 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15e9b037670f1c0e45c820cb92270698797d7139a20d765789f519f54f8ead77\": container with ID starting with 15e9b037670f1c0e45c820cb92270698797d7139a20d765789f519f54f8ead77 not found: ID does not exist" containerID="15e9b037670f1c0e45c820cb92270698797d7139a20d765789f519f54f8ead77" Jul 15 23:38:19.651985 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:19.651925 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15e9b037670f1c0e45c820cb92270698797d7139a20d765789f519f54f8ead77"} err="failed to get container status \"15e9b037670f1c0e45c820cb92270698797d7139a20d765789f519f54f8ead77\": rpc error: code = NotFound desc = could not find container \"15e9b037670f1c0e45c820cb92270698797d7139a20d765789f519f54f8ead77\": container with ID starting with 15e9b037670f1c0e45c820cb92270698797d7139a20d765789f519f54f8ead77 not found: ID does not exist" Jul 15 23:38:19.651985 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:19.651938 2563 scope.go:117] "RemoveContainer" containerID="dcfd2775981fa0a98baffb79b279dd05931faeaf553ddb4c2e211978d4ca909c" Jul 15 23:38:19.652188 ip-10-0-128-197 kubenswrapper[2563]: E0715 23:38:19.652169 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dcfd2775981fa0a98baffb79b279dd05931faeaf553ddb4c2e211978d4ca909c\": container with ID starting with dcfd2775981fa0a98baffb79b279dd05931faeaf553ddb4c2e211978d4ca909c not found: ID does not exist" containerID="dcfd2775981fa0a98baffb79b279dd05931faeaf553ddb4c2e211978d4ca909c" Jul 15 23:38:19.652274 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:19.652193 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dcfd2775981fa0a98baffb79b279dd05931faeaf553ddb4c2e211978d4ca909c"} err="failed to get container status \"dcfd2775981fa0a98baffb79b279dd05931faeaf553ddb4c2e211978d4ca909c\": rpc error: code = NotFound desc = could not find container \"dcfd2775981fa0a98baffb79b279dd05931faeaf553ddb4c2e211978d4ca909c\": container with ID starting with dcfd2775981fa0a98baffb79b279dd05931faeaf553ddb4c2e211978d4ca909c not found: ID does not exist" Jul 15 23:38:19.652274 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:19.652207 2563 scope.go:117] "RemoveContainer" containerID="7c12b9be4a933b3f011075e9b9a97af663bcc7f93257c38bdac3ccfa931c7758" Jul 15 23:38:19.652463 ip-10-0-128-197 kubenswrapper[2563]: E0715 23:38:19.652447 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c12b9be4a933b3f011075e9b9a97af663bcc7f93257c38bdac3ccfa931c7758\": container with ID starting with 7c12b9be4a933b3f011075e9b9a97af663bcc7f93257c38bdac3ccfa931c7758 not found: ID does not exist" containerID="7c12b9be4a933b3f011075e9b9a97af663bcc7f93257c38bdac3ccfa931c7758" Jul 15 23:38:19.652511 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:19.652466 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c12b9be4a933b3f011075e9b9a97af663bcc7f93257c38bdac3ccfa931c7758"} err="failed to get container status \"7c12b9be4a933b3f011075e9b9a97af663bcc7f93257c38bdac3ccfa931c7758\": rpc error: code = NotFound desc = could not find container \"7c12b9be4a933b3f011075e9b9a97af663bcc7f93257c38bdac3ccfa931c7758\": container with ID starting with 7c12b9be4a933b3f011075e9b9a97af663bcc7f93257c38bdac3ccfa931c7758 not found: ID does not exist" Jul 15 23:38:19.652511 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:19.652478 2563 scope.go:117] "RemoveContainer" containerID="fb9dbad6497ba7894d5fb5c72ac48f4219df84a9c0863367dead5834e92eb7e5" Jul 15 23:38:19.652708 ip-10-0-128-197 kubenswrapper[2563]: E0715 23:38:19.652691 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb9dbad6497ba7894d5fb5c72ac48f4219df84a9c0863367dead5834e92eb7e5\": container with ID starting with fb9dbad6497ba7894d5fb5c72ac48f4219df84a9c0863367dead5834e92eb7e5 not found: ID does not exist" containerID="fb9dbad6497ba7894d5fb5c72ac48f4219df84a9c0863367dead5834e92eb7e5" Jul 15 23:38:19.652752 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:19.652712 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb9dbad6497ba7894d5fb5c72ac48f4219df84a9c0863367dead5834e92eb7e5"} err="failed to get container status \"fb9dbad6497ba7894d5fb5c72ac48f4219df84a9c0863367dead5834e92eb7e5\": rpc error: code = NotFound desc = could not find container \"fb9dbad6497ba7894d5fb5c72ac48f4219df84a9c0863367dead5834e92eb7e5\": container with ID starting with fb9dbad6497ba7894d5fb5c72ac48f4219df84a9c0863367dead5834e92eb7e5 not found: ID does not exist" Jul 15 23:38:19.652752 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:19.652725 2563 scope.go:117] "RemoveContainer" containerID="baa174551f012dbe92f015d620924fd4ab07cf84fbe2158b0919fd5626754bc6" Jul 15 23:38:19.653082 ip-10-0-128-197 kubenswrapper[2563]: E0715 23:38:19.652954 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"baa174551f012dbe92f015d620924fd4ab07cf84fbe2158b0919fd5626754bc6\": container with ID starting with baa174551f012dbe92f015d620924fd4ab07cf84fbe2158b0919fd5626754bc6 not found: ID does not exist" containerID="baa174551f012dbe92f015d620924fd4ab07cf84fbe2158b0919fd5626754bc6" Jul 15 23:38:19.653130 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:19.653093 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"baa174551f012dbe92f015d620924fd4ab07cf84fbe2158b0919fd5626754bc6"} err="failed to get container status \"baa174551f012dbe92f015d620924fd4ab07cf84fbe2158b0919fd5626754bc6\": rpc error: code = NotFound desc = could not find container \"baa174551f012dbe92f015d620924fd4ab07cf84fbe2158b0919fd5626754bc6\": container with ID starting with baa174551f012dbe92f015d620924fd4ab07cf84fbe2158b0919fd5626754bc6 not found: ID does not exist" Jul 15 23:38:19.653130 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:19.653108 2563 scope.go:117] "RemoveContainer" containerID="a3cfe9a8d9b94f2fb69268dc3f0dd81c5e1290f049e4cd8358062fbf08639053" Jul 15 23:38:19.653376 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:19.653356 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3cfe9a8d9b94f2fb69268dc3f0dd81c5e1290f049e4cd8358062fbf08639053"} err="failed to get container status \"a3cfe9a8d9b94f2fb69268dc3f0dd81c5e1290f049e4cd8358062fbf08639053\": rpc error: code = NotFound desc = could not find container \"a3cfe9a8d9b94f2fb69268dc3f0dd81c5e1290f049e4cd8358062fbf08639053\": container with ID starting with a3cfe9a8d9b94f2fb69268dc3f0dd81c5e1290f049e4cd8358062fbf08639053 not found: ID does not exist" Jul 15 23:38:19.653416 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:19.653377 2563 scope.go:117] "RemoveContainer" containerID="e10c2697bb2dd4dc1576fda5ed268bf9256c3e78c729a38a70587f488dc73c8b" Jul 15 23:38:19.653593 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:19.653568 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e10c2697bb2dd4dc1576fda5ed268bf9256c3e78c729a38a70587f488dc73c8b"} err="failed to get container status \"e10c2697bb2dd4dc1576fda5ed268bf9256c3e78c729a38a70587f488dc73c8b\": rpc error: code = NotFound desc = could not find container \"e10c2697bb2dd4dc1576fda5ed268bf9256c3e78c729a38a70587f488dc73c8b\": container with ID starting with e10c2697bb2dd4dc1576fda5ed268bf9256c3e78c729a38a70587f488dc73c8b not found: ID does not exist" Jul 15 23:38:19.653593 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:19.653590 2563 scope.go:117] "RemoveContainer" containerID="15e9b037670f1c0e45c820cb92270698797d7139a20d765789f519f54f8ead77" Jul 15 23:38:19.653801 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:19.653782 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15e9b037670f1c0e45c820cb92270698797d7139a20d765789f519f54f8ead77"} err="failed to get container status \"15e9b037670f1c0e45c820cb92270698797d7139a20d765789f519f54f8ead77\": rpc error: code = NotFound desc = could not find container \"15e9b037670f1c0e45c820cb92270698797d7139a20d765789f519f54f8ead77\": container with ID starting with 15e9b037670f1c0e45c820cb92270698797d7139a20d765789f519f54f8ead77 not found: ID does not exist" Jul 15 23:38:19.653872 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:19.653803 2563 scope.go:117] "RemoveContainer" containerID="dcfd2775981fa0a98baffb79b279dd05931faeaf553ddb4c2e211978d4ca909c" Jul 15 23:38:19.654071 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:19.654053 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dcfd2775981fa0a98baffb79b279dd05931faeaf553ddb4c2e211978d4ca909c"} err="failed to get container status \"dcfd2775981fa0a98baffb79b279dd05931faeaf553ddb4c2e211978d4ca909c\": rpc error: code = NotFound desc = could not find container \"dcfd2775981fa0a98baffb79b279dd05931faeaf553ddb4c2e211978d4ca909c\": container with ID starting with dcfd2775981fa0a98baffb79b279dd05931faeaf553ddb4c2e211978d4ca909c not found: ID does not exist" Jul 15 23:38:19.654071 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:19.654070 2563 scope.go:117] "RemoveContainer" containerID="7c12b9be4a933b3f011075e9b9a97af663bcc7f93257c38bdac3ccfa931c7758" Jul 15 23:38:19.654303 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:19.654282 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c12b9be4a933b3f011075e9b9a97af663bcc7f93257c38bdac3ccfa931c7758"} err="failed to get container status \"7c12b9be4a933b3f011075e9b9a97af663bcc7f93257c38bdac3ccfa931c7758\": rpc error: code = NotFound desc = could not find container \"7c12b9be4a933b3f011075e9b9a97af663bcc7f93257c38bdac3ccfa931c7758\": container with ID starting with 7c12b9be4a933b3f011075e9b9a97af663bcc7f93257c38bdac3ccfa931c7758 not found: ID does not exist" Jul 15 23:38:19.654303 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:19.654302 2563 scope.go:117] "RemoveContainer" containerID="fb9dbad6497ba7894d5fb5c72ac48f4219df84a9c0863367dead5834e92eb7e5" Jul 15 23:38:19.654497 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:19.654482 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb9dbad6497ba7894d5fb5c72ac48f4219df84a9c0863367dead5834e92eb7e5"} err="failed to get container status \"fb9dbad6497ba7894d5fb5c72ac48f4219df84a9c0863367dead5834e92eb7e5\": rpc error: code = NotFound desc = could not find container \"fb9dbad6497ba7894d5fb5c72ac48f4219df84a9c0863367dead5834e92eb7e5\": container with ID starting with fb9dbad6497ba7894d5fb5c72ac48f4219df84a9c0863367dead5834e92eb7e5 not found: ID does not exist" Jul 15 23:38:19.654497 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:19.654496 2563 scope.go:117] "RemoveContainer" containerID="baa174551f012dbe92f015d620924fd4ab07cf84fbe2158b0919fd5626754bc6" Jul 15 23:38:19.654674 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:19.654657 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"baa174551f012dbe92f015d620924fd4ab07cf84fbe2158b0919fd5626754bc6"} err="failed to get container status \"baa174551f012dbe92f015d620924fd4ab07cf84fbe2158b0919fd5626754bc6\": rpc error: code = NotFound desc = could not find container \"baa174551f012dbe92f015d620924fd4ab07cf84fbe2158b0919fd5626754bc6\": container with ID starting with baa174551f012dbe92f015d620924fd4ab07cf84fbe2158b0919fd5626754bc6 not found: ID does not exist" Jul 15 23:38:19.654733 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:19.654675 2563 scope.go:117] "RemoveContainer" containerID="a3cfe9a8d9b94f2fb69268dc3f0dd81c5e1290f049e4cd8358062fbf08639053" Jul 15 23:38:19.654889 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:19.654873 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3cfe9a8d9b94f2fb69268dc3f0dd81c5e1290f049e4cd8358062fbf08639053"} err="failed to get container status \"a3cfe9a8d9b94f2fb69268dc3f0dd81c5e1290f049e4cd8358062fbf08639053\": rpc error: code = NotFound desc = could not find container \"a3cfe9a8d9b94f2fb69268dc3f0dd81c5e1290f049e4cd8358062fbf08639053\": container with ID starting with a3cfe9a8d9b94f2fb69268dc3f0dd81c5e1290f049e4cd8358062fbf08639053 not found: ID does not exist" Jul 15 23:38:19.654889 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:19.654889 2563 scope.go:117] "RemoveContainer" containerID="e10c2697bb2dd4dc1576fda5ed268bf9256c3e78c729a38a70587f488dc73c8b" Jul 15 23:38:19.655144 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:19.655124 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e10c2697bb2dd4dc1576fda5ed268bf9256c3e78c729a38a70587f488dc73c8b"} err="failed to get container status \"e10c2697bb2dd4dc1576fda5ed268bf9256c3e78c729a38a70587f488dc73c8b\": rpc error: code = NotFound desc = could not find container \"e10c2697bb2dd4dc1576fda5ed268bf9256c3e78c729a38a70587f488dc73c8b\": container with ID starting with e10c2697bb2dd4dc1576fda5ed268bf9256c3e78c729a38a70587f488dc73c8b not found: ID does not exist" Jul 15 23:38:19.655186 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:19.655147 2563 scope.go:117] "RemoveContainer" containerID="15e9b037670f1c0e45c820cb92270698797d7139a20d765789f519f54f8ead77" Jul 15 23:38:19.655406 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:19.655375 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15e9b037670f1c0e45c820cb92270698797d7139a20d765789f519f54f8ead77"} err="failed to get container status \"15e9b037670f1c0e45c820cb92270698797d7139a20d765789f519f54f8ead77\": rpc error: code = NotFound desc = could not find container \"15e9b037670f1c0e45c820cb92270698797d7139a20d765789f519f54f8ead77\": container with ID starting with 15e9b037670f1c0e45c820cb92270698797d7139a20d765789f519f54f8ead77 not found: ID does not exist" Jul 15 23:38:19.655487 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:19.655409 2563 scope.go:117] "RemoveContainer" containerID="dcfd2775981fa0a98baffb79b279dd05931faeaf553ddb4c2e211978d4ca909c" Jul 15 23:38:19.655651 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:19.655635 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dcfd2775981fa0a98baffb79b279dd05931faeaf553ddb4c2e211978d4ca909c"} err="failed to get container status \"dcfd2775981fa0a98baffb79b279dd05931faeaf553ddb4c2e211978d4ca909c\": rpc error: code = NotFound desc = could not find container \"dcfd2775981fa0a98baffb79b279dd05931faeaf553ddb4c2e211978d4ca909c\": container with ID starting with dcfd2775981fa0a98baffb79b279dd05931faeaf553ddb4c2e211978d4ca909c not found: ID does not exist" Jul 15 23:38:19.655691 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:19.655653 2563 scope.go:117] "RemoveContainer" containerID="7c12b9be4a933b3f011075e9b9a97af663bcc7f93257c38bdac3ccfa931c7758" Jul 15 23:38:19.655892 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:19.655874 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c12b9be4a933b3f011075e9b9a97af663bcc7f93257c38bdac3ccfa931c7758"} err="failed to get container status \"7c12b9be4a933b3f011075e9b9a97af663bcc7f93257c38bdac3ccfa931c7758\": rpc error: code = NotFound desc = could not find container \"7c12b9be4a933b3f011075e9b9a97af663bcc7f93257c38bdac3ccfa931c7758\": container with ID starting with 7c12b9be4a933b3f011075e9b9a97af663bcc7f93257c38bdac3ccfa931c7758 not found: ID does not exist" Jul 15 23:38:19.655936 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:19.655893 2563 scope.go:117] "RemoveContainer" containerID="fb9dbad6497ba7894d5fb5c72ac48f4219df84a9c0863367dead5834e92eb7e5" Jul 15 23:38:19.656133 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:19.656115 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb9dbad6497ba7894d5fb5c72ac48f4219df84a9c0863367dead5834e92eb7e5"} err="failed to get container status \"fb9dbad6497ba7894d5fb5c72ac48f4219df84a9c0863367dead5834e92eb7e5\": rpc error: code = NotFound desc = could not find container \"fb9dbad6497ba7894d5fb5c72ac48f4219df84a9c0863367dead5834e92eb7e5\": container with ID starting with fb9dbad6497ba7894d5fb5c72ac48f4219df84a9c0863367dead5834e92eb7e5 not found: ID does not exist" Jul 15 23:38:19.656185 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:19.656135 2563 scope.go:117] "RemoveContainer" containerID="baa174551f012dbe92f015d620924fd4ab07cf84fbe2158b0919fd5626754bc6" Jul 15 23:38:19.656332 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:19.656317 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"baa174551f012dbe92f015d620924fd4ab07cf84fbe2158b0919fd5626754bc6"} err="failed to get container status \"baa174551f012dbe92f015d620924fd4ab07cf84fbe2158b0919fd5626754bc6\": rpc error: code = NotFound desc = could not find container \"baa174551f012dbe92f015d620924fd4ab07cf84fbe2158b0919fd5626754bc6\": container with ID starting with baa174551f012dbe92f015d620924fd4ab07cf84fbe2158b0919fd5626754bc6 not found: ID does not exist" Jul 15 23:38:19.656371 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:19.656334 2563 scope.go:117] "RemoveContainer" containerID="a3cfe9a8d9b94f2fb69268dc3f0dd81c5e1290f049e4cd8358062fbf08639053" Jul 15 23:38:19.656520 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:19.656505 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3cfe9a8d9b94f2fb69268dc3f0dd81c5e1290f049e4cd8358062fbf08639053"} err="failed to get container status \"a3cfe9a8d9b94f2fb69268dc3f0dd81c5e1290f049e4cd8358062fbf08639053\": rpc error: code = NotFound desc = could not find container \"a3cfe9a8d9b94f2fb69268dc3f0dd81c5e1290f049e4cd8358062fbf08639053\": container with ID starting with a3cfe9a8d9b94f2fb69268dc3f0dd81c5e1290f049e4cd8358062fbf08639053 not found: ID does not exist" Jul 15 23:38:19.656572 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:19.656520 2563 scope.go:117] "RemoveContainer" containerID="e10c2697bb2dd4dc1576fda5ed268bf9256c3e78c729a38a70587f488dc73c8b" Jul 15 23:38:19.656713 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:19.656697 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e10c2697bb2dd4dc1576fda5ed268bf9256c3e78c729a38a70587f488dc73c8b"} err="failed to get container status \"e10c2697bb2dd4dc1576fda5ed268bf9256c3e78c729a38a70587f488dc73c8b\": rpc error: code = NotFound desc = could not find container \"e10c2697bb2dd4dc1576fda5ed268bf9256c3e78c729a38a70587f488dc73c8b\": container with ID starting with e10c2697bb2dd4dc1576fda5ed268bf9256c3e78c729a38a70587f488dc73c8b not found: ID does not exist" Jul 15 23:38:19.656754 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:19.656712 2563 scope.go:117] "RemoveContainer" containerID="15e9b037670f1c0e45c820cb92270698797d7139a20d765789f519f54f8ead77" Jul 15 23:38:19.656890 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:19.656874 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15e9b037670f1c0e45c820cb92270698797d7139a20d765789f519f54f8ead77"} err="failed to get container status \"15e9b037670f1c0e45c820cb92270698797d7139a20d765789f519f54f8ead77\": rpc error: code = NotFound desc = could not find container \"15e9b037670f1c0e45c820cb92270698797d7139a20d765789f519f54f8ead77\": container with ID starting with 15e9b037670f1c0e45c820cb92270698797d7139a20d765789f519f54f8ead77 not found: ID does not exist" Jul 15 23:38:19.656928 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:19.656890 2563 scope.go:117] "RemoveContainer" containerID="dcfd2775981fa0a98baffb79b279dd05931faeaf553ddb4c2e211978d4ca909c" Jul 15 23:38:19.657132 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:19.657109 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dcfd2775981fa0a98baffb79b279dd05931faeaf553ddb4c2e211978d4ca909c"} err="failed to get container status \"dcfd2775981fa0a98baffb79b279dd05931faeaf553ddb4c2e211978d4ca909c\": rpc error: code = NotFound desc = could not find container \"dcfd2775981fa0a98baffb79b279dd05931faeaf553ddb4c2e211978d4ca909c\": container with ID starting with dcfd2775981fa0a98baffb79b279dd05931faeaf553ddb4c2e211978d4ca909c not found: ID does not exist" Jul 15 23:38:19.657132 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:19.657131 2563 scope.go:117] "RemoveContainer" containerID="7c12b9be4a933b3f011075e9b9a97af663bcc7f93257c38bdac3ccfa931c7758" Jul 15 23:38:19.657371 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:19.657356 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c12b9be4a933b3f011075e9b9a97af663bcc7f93257c38bdac3ccfa931c7758"} err="failed to get container status \"7c12b9be4a933b3f011075e9b9a97af663bcc7f93257c38bdac3ccfa931c7758\": rpc error: code = NotFound desc = could not find container \"7c12b9be4a933b3f011075e9b9a97af663bcc7f93257c38bdac3ccfa931c7758\": container with ID starting with 7c12b9be4a933b3f011075e9b9a97af663bcc7f93257c38bdac3ccfa931c7758 not found: ID does not exist" Jul 15 23:38:19.657420 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:19.657371 2563 scope.go:117] "RemoveContainer" containerID="fb9dbad6497ba7894d5fb5c72ac48f4219df84a9c0863367dead5834e92eb7e5" Jul 15 23:38:19.657565 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:19.657549 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb9dbad6497ba7894d5fb5c72ac48f4219df84a9c0863367dead5834e92eb7e5"} err="failed to get container status \"fb9dbad6497ba7894d5fb5c72ac48f4219df84a9c0863367dead5834e92eb7e5\": rpc error: code = NotFound desc = could not find container \"fb9dbad6497ba7894d5fb5c72ac48f4219df84a9c0863367dead5834e92eb7e5\": container with ID starting with fb9dbad6497ba7894d5fb5c72ac48f4219df84a9c0863367dead5834e92eb7e5 not found: ID does not exist" Jul 15 23:38:19.657606 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:19.657567 2563 scope.go:117] "RemoveContainer" containerID="baa174551f012dbe92f015d620924fd4ab07cf84fbe2158b0919fd5626754bc6" Jul 15 23:38:19.657789 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:19.657772 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"baa174551f012dbe92f015d620924fd4ab07cf84fbe2158b0919fd5626754bc6"} err="failed to get container status \"baa174551f012dbe92f015d620924fd4ab07cf84fbe2158b0919fd5626754bc6\": rpc error: code = NotFound desc = could not find container \"baa174551f012dbe92f015d620924fd4ab07cf84fbe2158b0919fd5626754bc6\": container with ID starting with baa174551f012dbe92f015d620924fd4ab07cf84fbe2158b0919fd5626754bc6 not found: ID does not exist" Jul 15 23:38:19.657789 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:19.657788 2563 scope.go:117] "RemoveContainer" containerID="a3cfe9a8d9b94f2fb69268dc3f0dd81c5e1290f049e4cd8358062fbf08639053" Jul 15 23:38:19.658026 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:19.658009 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3cfe9a8d9b94f2fb69268dc3f0dd81c5e1290f049e4cd8358062fbf08639053"} err="failed to get container status \"a3cfe9a8d9b94f2fb69268dc3f0dd81c5e1290f049e4cd8358062fbf08639053\": rpc error: code = NotFound desc = could not find container \"a3cfe9a8d9b94f2fb69268dc3f0dd81c5e1290f049e4cd8358062fbf08639053\": container with ID starting with a3cfe9a8d9b94f2fb69268dc3f0dd81c5e1290f049e4cd8358062fbf08639053 not found: ID does not exist" Jul 15 23:38:19.658079 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:19.658026 2563 scope.go:117] "RemoveContainer" containerID="e10c2697bb2dd4dc1576fda5ed268bf9256c3e78c729a38a70587f488dc73c8b" Jul 15 23:38:19.658239 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:19.658223 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e10c2697bb2dd4dc1576fda5ed268bf9256c3e78c729a38a70587f488dc73c8b"} err="failed to get container status \"e10c2697bb2dd4dc1576fda5ed268bf9256c3e78c729a38a70587f488dc73c8b\": rpc error: code = NotFound desc = could not find container \"e10c2697bb2dd4dc1576fda5ed268bf9256c3e78c729a38a70587f488dc73c8b\": container with ID starting with e10c2697bb2dd4dc1576fda5ed268bf9256c3e78c729a38a70587f488dc73c8b not found: ID does not exist" Jul 15 23:38:19.658290 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:19.658240 2563 scope.go:117] "RemoveContainer" containerID="15e9b037670f1c0e45c820cb92270698797d7139a20d765789f519f54f8ead77" Jul 15 23:38:19.658454 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:19.658436 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15e9b037670f1c0e45c820cb92270698797d7139a20d765789f519f54f8ead77"} err="failed to get container status \"15e9b037670f1c0e45c820cb92270698797d7139a20d765789f519f54f8ead77\": rpc error: code = NotFound desc = could not find container \"15e9b037670f1c0e45c820cb92270698797d7139a20d765789f519f54f8ead77\": container with ID starting with 15e9b037670f1c0e45c820cb92270698797d7139a20d765789f519f54f8ead77 not found: ID does not exist" Jul 15 23:38:19.658511 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:19.658458 2563 scope.go:117] "RemoveContainer" containerID="dcfd2775981fa0a98baffb79b279dd05931faeaf553ddb4c2e211978d4ca909c" Jul 15 23:38:19.658663 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:19.658641 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dcfd2775981fa0a98baffb79b279dd05931faeaf553ddb4c2e211978d4ca909c"} err="failed to get container status \"dcfd2775981fa0a98baffb79b279dd05931faeaf553ddb4c2e211978d4ca909c\": rpc error: code = NotFound desc = could not find container \"dcfd2775981fa0a98baffb79b279dd05931faeaf553ddb4c2e211978d4ca909c\": container with ID starting with dcfd2775981fa0a98baffb79b279dd05931faeaf553ddb4c2e211978d4ca909c not found: ID does not exist" Jul 15 23:38:19.658706 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:19.658666 2563 scope.go:117] "RemoveContainer" containerID="7c12b9be4a933b3f011075e9b9a97af663bcc7f93257c38bdac3ccfa931c7758" Jul 15 23:38:19.658903 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:19.658887 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c12b9be4a933b3f011075e9b9a97af663bcc7f93257c38bdac3ccfa931c7758"} err="failed to get container status \"7c12b9be4a933b3f011075e9b9a97af663bcc7f93257c38bdac3ccfa931c7758\": rpc error: code = NotFound desc = could not find container \"7c12b9be4a933b3f011075e9b9a97af663bcc7f93257c38bdac3ccfa931c7758\": container with ID starting with 7c12b9be4a933b3f011075e9b9a97af663bcc7f93257c38bdac3ccfa931c7758 not found: ID does not exist" Jul 15 23:38:19.658949 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:19.658904 2563 scope.go:117] "RemoveContainer" containerID="fb9dbad6497ba7894d5fb5c72ac48f4219df84a9c0863367dead5834e92eb7e5" Jul 15 23:38:19.659131 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:19.659115 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb9dbad6497ba7894d5fb5c72ac48f4219df84a9c0863367dead5834e92eb7e5"} err="failed to get container status \"fb9dbad6497ba7894d5fb5c72ac48f4219df84a9c0863367dead5834e92eb7e5\": rpc error: code = NotFound desc = could not find container \"fb9dbad6497ba7894d5fb5c72ac48f4219df84a9c0863367dead5834e92eb7e5\": container with ID starting with fb9dbad6497ba7894d5fb5c72ac48f4219df84a9c0863367dead5834e92eb7e5 not found: ID does not exist" Jul 15 23:38:19.659175 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:19.659132 2563 scope.go:117] "RemoveContainer" containerID="baa174551f012dbe92f015d620924fd4ab07cf84fbe2158b0919fd5626754bc6" Jul 15 23:38:19.659389 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:19.659364 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"baa174551f012dbe92f015d620924fd4ab07cf84fbe2158b0919fd5626754bc6"} err="failed to get container status \"baa174551f012dbe92f015d620924fd4ab07cf84fbe2158b0919fd5626754bc6\": rpc error: code = NotFound desc = could not find container \"baa174551f012dbe92f015d620924fd4ab07cf84fbe2158b0919fd5626754bc6\": container with ID starting with baa174551f012dbe92f015d620924fd4ab07cf84fbe2158b0919fd5626754bc6 not found: ID does not exist" Jul 15 23:38:19.659446 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:19.659391 2563 scope.go:117] "RemoveContainer" containerID="a3cfe9a8d9b94f2fb69268dc3f0dd81c5e1290f049e4cd8358062fbf08639053" Jul 15 23:38:19.659640 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:19.659623 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3cfe9a8d9b94f2fb69268dc3f0dd81c5e1290f049e4cd8358062fbf08639053"} err="failed to get container status \"a3cfe9a8d9b94f2fb69268dc3f0dd81c5e1290f049e4cd8358062fbf08639053\": rpc error: code = NotFound desc = could not find container \"a3cfe9a8d9b94f2fb69268dc3f0dd81c5e1290f049e4cd8358062fbf08639053\": container with ID starting with a3cfe9a8d9b94f2fb69268dc3f0dd81c5e1290f049e4cd8358062fbf08639053 not found: ID does not exist" Jul 15 23:38:19.659697 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:19.659641 2563 scope.go:117] "RemoveContainer" containerID="e10c2697bb2dd4dc1576fda5ed268bf9256c3e78c729a38a70587f488dc73c8b" Jul 15 23:38:19.659878 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:19.659849 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e10c2697bb2dd4dc1576fda5ed268bf9256c3e78c729a38a70587f488dc73c8b"} err="failed to get container status \"e10c2697bb2dd4dc1576fda5ed268bf9256c3e78c729a38a70587f488dc73c8b\": rpc error: code = NotFound desc = could not find container \"e10c2697bb2dd4dc1576fda5ed268bf9256c3e78c729a38a70587f488dc73c8b\": container with ID starting with e10c2697bb2dd4dc1576fda5ed268bf9256c3e78c729a38a70587f488dc73c8b not found: ID does not exist" Jul 15 23:38:19.659878 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:19.659869 2563 scope.go:117] "RemoveContainer" containerID="15e9b037670f1c0e45c820cb92270698797d7139a20d765789f519f54f8ead77" Jul 15 23:38:19.660148 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:19.660126 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15e9b037670f1c0e45c820cb92270698797d7139a20d765789f519f54f8ead77"} err="failed to get container status \"15e9b037670f1c0e45c820cb92270698797d7139a20d765789f519f54f8ead77\": rpc error: code = NotFound desc = could not find container \"15e9b037670f1c0e45c820cb92270698797d7139a20d765789f519f54f8ead77\": container with ID starting with 15e9b037670f1c0e45c820cb92270698797d7139a20d765789f519f54f8ead77 not found: ID does not exist" Jul 15 23:38:19.660196 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:19.660150 2563 scope.go:117] "RemoveContainer" containerID="dcfd2775981fa0a98baffb79b279dd05931faeaf553ddb4c2e211978d4ca909c" Jul 15 23:38:19.660389 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:19.660370 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dcfd2775981fa0a98baffb79b279dd05931faeaf553ddb4c2e211978d4ca909c"} err="failed to get container status \"dcfd2775981fa0a98baffb79b279dd05931faeaf553ddb4c2e211978d4ca909c\": rpc error: code = NotFound desc = could not find container \"dcfd2775981fa0a98baffb79b279dd05931faeaf553ddb4c2e211978d4ca909c\": container with ID starting with dcfd2775981fa0a98baffb79b279dd05931faeaf553ddb4c2e211978d4ca909c not found: ID does not exist" Jul 15 23:38:19.660433 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:19.660390 2563 scope.go:117] "RemoveContainer" containerID="7c12b9be4a933b3f011075e9b9a97af663bcc7f93257c38bdac3ccfa931c7758" Jul 15 23:38:19.660632 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:19.660608 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c12b9be4a933b3f011075e9b9a97af663bcc7f93257c38bdac3ccfa931c7758"} err="failed to get container status \"7c12b9be4a933b3f011075e9b9a97af663bcc7f93257c38bdac3ccfa931c7758\": rpc error: code = NotFound desc = could not find container \"7c12b9be4a933b3f011075e9b9a97af663bcc7f93257c38bdac3ccfa931c7758\": container with ID starting with 7c12b9be4a933b3f011075e9b9a97af663bcc7f93257c38bdac3ccfa931c7758 not found: ID does not exist" Jul 15 23:38:19.660632 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:19.660630 2563 scope.go:117] "RemoveContainer" containerID="fb9dbad6497ba7894d5fb5c72ac48f4219df84a9c0863367dead5834e92eb7e5" Jul 15 23:38:19.660851 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:19.660834 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb9dbad6497ba7894d5fb5c72ac48f4219df84a9c0863367dead5834e92eb7e5"} err="failed to get container status \"fb9dbad6497ba7894d5fb5c72ac48f4219df84a9c0863367dead5834e92eb7e5\": rpc error: code = NotFound desc = could not find container \"fb9dbad6497ba7894d5fb5c72ac48f4219df84a9c0863367dead5834e92eb7e5\": container with ID starting with fb9dbad6497ba7894d5fb5c72ac48f4219df84a9c0863367dead5834e92eb7e5 not found: ID does not exist" Jul 15 23:38:19.660851 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:19.660851 2563 scope.go:117] "RemoveContainer" containerID="baa174551f012dbe92f015d620924fd4ab07cf84fbe2158b0919fd5626754bc6" Jul 15 23:38:19.661095 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:19.661079 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"baa174551f012dbe92f015d620924fd4ab07cf84fbe2158b0919fd5626754bc6"} err="failed to get container status \"baa174551f012dbe92f015d620924fd4ab07cf84fbe2158b0919fd5626754bc6\": rpc error: code = NotFound desc = could not find container \"baa174551f012dbe92f015d620924fd4ab07cf84fbe2158b0919fd5626754bc6\": container with ID starting with baa174551f012dbe92f015d620924fd4ab07cf84fbe2158b0919fd5626754bc6 not found: ID does not exist" Jul 15 23:38:19.687444 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:19.687415 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cwzrf\" (UniqueName: \"kubernetes.io/projected/5e3e5aa3-8bc5-49ff-8706-6107d3e3d2bb-kube-api-access-cwzrf\") pod \"5e3e5aa3-8bc5-49ff-8706-6107d3e3d2bb\" (UID: \"5e3e5aa3-8bc5-49ff-8706-6107d3e3d2bb\") " Jul 15 23:38:19.687552 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:19.687467 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5e3e5aa3-8bc5-49ff-8706-6107d3e3d2bb-web-config\") pod \"5e3e5aa3-8bc5-49ff-8706-6107d3e3d2bb\" (UID: \"5e3e5aa3-8bc5-49ff-8706-6107d3e3d2bb\") " Jul 15 23:38:19.687552 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:19.687499 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5e3e5aa3-8bc5-49ff-8706-6107d3e3d2bb-secret-alertmanager-kube-rbac-proxy-web\") pod \"5e3e5aa3-8bc5-49ff-8706-6107d3e3d2bb\" (UID: \"5e3e5aa3-8bc5-49ff-8706-6107d3e3d2bb\") " Jul 15 23:38:19.687552 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:19.687544 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5e3e5aa3-8bc5-49ff-8706-6107d3e3d2bb-tls-assets\") pod \"5e3e5aa3-8bc5-49ff-8706-6107d3e3d2bb\" (UID: \"5e3e5aa3-8bc5-49ff-8706-6107d3e3d2bb\") " Jul 15 23:38:19.687703 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:19.687569 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/5e3e5aa3-8bc5-49ff-8706-6107d3e3d2bb-secret-alertmanager-main-tls\") pod \"5e3e5aa3-8bc5-49ff-8706-6107d3e3d2bb\" (UID: \"5e3e5aa3-8bc5-49ff-8706-6107d3e3d2bb\") " Jul 15 23:38:19.687762 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:19.687744 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5e3e5aa3-8bc5-49ff-8706-6107d3e3d2bb-config-out\") pod \"5e3e5aa3-8bc5-49ff-8706-6107d3e3d2bb\" (UID: \"5e3e5aa3-8bc5-49ff-8706-6107d3e3d2bb\") " Jul 15 23:38:19.687817 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:19.687802 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5e3e5aa3-8bc5-49ff-8706-6107d3e3d2bb-secret-alertmanager-kube-rbac-proxy\") pod \"5e3e5aa3-8bc5-49ff-8706-6107d3e3d2bb\" (UID: \"5e3e5aa3-8bc5-49ff-8706-6107d3e3d2bb\") " Jul 15 23:38:19.687871 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:19.687861 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5e3e5aa3-8bc5-49ff-8706-6107d3e3d2bb-alertmanager-trusted-ca-bundle\") pod \"5e3e5aa3-8bc5-49ff-8706-6107d3e3d2bb\" (UID: \"5e3e5aa3-8bc5-49ff-8706-6107d3e3d2bb\") " Jul 15 23:38:19.687934 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:19.687899 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5e3e5aa3-8bc5-49ff-8706-6107d3e3d2bb-metrics-client-ca\") pod \"5e3e5aa3-8bc5-49ff-8706-6107d3e3d2bb\" (UID: \"5e3e5aa3-8bc5-49ff-8706-6107d3e3d2bb\") " Jul 15 23:38:19.688014 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:19.687938 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/5e3e5aa3-8bc5-49ff-8706-6107d3e3d2bb-cluster-tls-config\") pod \"5e3e5aa3-8bc5-49ff-8706-6107d3e3d2bb\" (UID: \"5e3e5aa3-8bc5-49ff-8706-6107d3e3d2bb\") " Jul 15 23:38:19.688081 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:19.688018 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/5e3e5aa3-8bc5-49ff-8706-6107d3e3d2bb-config-volume\") pod \"5e3e5aa3-8bc5-49ff-8706-6107d3e3d2bb\" (UID: \"5e3e5aa3-8bc5-49ff-8706-6107d3e3d2bb\") " Jul 15 23:38:19.688081 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:19.688047 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/5e3e5aa3-8bc5-49ff-8706-6107d3e3d2bb-alertmanager-main-db\") pod \"5e3e5aa3-8bc5-49ff-8706-6107d3e3d2bb\" (UID: \"5e3e5aa3-8bc5-49ff-8706-6107d3e3d2bb\") " Jul 15 23:38:19.688081 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:19.688075 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/5e3e5aa3-8bc5-49ff-8706-6107d3e3d2bb-secret-alertmanager-kube-rbac-proxy-metric\") pod \"5e3e5aa3-8bc5-49ff-8706-6107d3e3d2bb\" (UID: \"5e3e5aa3-8bc5-49ff-8706-6107d3e3d2bb\") " Jul 15 23:38:19.688792 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:19.688724 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e3e5aa3-8bc5-49ff-8706-6107d3e3d2bb-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "5e3e5aa3-8bc5-49ff-8706-6107d3e3d2bb" (UID: "5e3e5aa3-8bc5-49ff-8706-6107d3e3d2bb"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 23:38:19.688792 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:19.688762 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e3e5aa3-8bc5-49ff-8706-6107d3e3d2bb-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "5e3e5aa3-8bc5-49ff-8706-6107d3e3d2bb" (UID: "5e3e5aa3-8bc5-49ff-8706-6107d3e3d2bb"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 23:38:19.688947 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:19.688791 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e3e5aa3-8bc5-49ff-8706-6107d3e3d2bb-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "5e3e5aa3-8bc5-49ff-8706-6107d3e3d2bb" (UID: "5e3e5aa3-8bc5-49ff-8706-6107d3e3d2bb"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 23:38:19.690731 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:19.690700 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e3e5aa3-8bc5-49ff-8706-6107d3e3d2bb-kube-api-access-cwzrf" (OuterVolumeSpecName: "kube-api-access-cwzrf") pod "5e3e5aa3-8bc5-49ff-8706-6107d3e3d2bb" (UID: "5e3e5aa3-8bc5-49ff-8706-6107d3e3d2bb"). InnerVolumeSpecName "kube-api-access-cwzrf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 23:38:19.690864 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:19.690844 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e3e5aa3-8bc5-49ff-8706-6107d3e3d2bb-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "5e3e5aa3-8bc5-49ff-8706-6107d3e3d2bb" (UID: "5e3e5aa3-8bc5-49ff-8706-6107d3e3d2bb"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 23:38:19.690935 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:19.690869 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e3e5aa3-8bc5-49ff-8706-6107d3e3d2bb-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "5e3e5aa3-8bc5-49ff-8706-6107d3e3d2bb" (UID: "5e3e5aa3-8bc5-49ff-8706-6107d3e3d2bb"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 23:38:19.691163 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:19.691135 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e3e5aa3-8bc5-49ff-8706-6107d3e3d2bb-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "5e3e5aa3-8bc5-49ff-8706-6107d3e3d2bb" (UID: "5e3e5aa3-8bc5-49ff-8706-6107d3e3d2bb"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 23:38:19.691260 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:19.691189 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e3e5aa3-8bc5-49ff-8706-6107d3e3d2bb-config-out" (OuterVolumeSpecName: "config-out") pod "5e3e5aa3-8bc5-49ff-8706-6107d3e3d2bb" (UID: "5e3e5aa3-8bc5-49ff-8706-6107d3e3d2bb"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 23:38:19.691322 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:19.691263 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e3e5aa3-8bc5-49ff-8706-6107d3e3d2bb-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "5e3e5aa3-8bc5-49ff-8706-6107d3e3d2bb" (UID: "5e3e5aa3-8bc5-49ff-8706-6107d3e3d2bb"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 23:38:19.691442 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:19.691414 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e3e5aa3-8bc5-49ff-8706-6107d3e3d2bb-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "5e3e5aa3-8bc5-49ff-8706-6107d3e3d2bb" (UID: "5e3e5aa3-8bc5-49ff-8706-6107d3e3d2bb"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 23:38:19.691948 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:19.691928 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e3e5aa3-8bc5-49ff-8706-6107d3e3d2bb-config-volume" (OuterVolumeSpecName: "config-volume") pod "5e3e5aa3-8bc5-49ff-8706-6107d3e3d2bb" (UID: "5e3e5aa3-8bc5-49ff-8706-6107d3e3d2bb"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 23:38:19.695509 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:19.695487 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e3e5aa3-8bc5-49ff-8706-6107d3e3d2bb-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "5e3e5aa3-8bc5-49ff-8706-6107d3e3d2bb" (UID: "5e3e5aa3-8bc5-49ff-8706-6107d3e3d2bb"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 23:38:19.701461 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:19.701437 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e3e5aa3-8bc5-49ff-8706-6107d3e3d2bb-web-config" (OuterVolumeSpecName: "web-config") pod "5e3e5aa3-8bc5-49ff-8706-6107d3e3d2bb" (UID: "5e3e5aa3-8bc5-49ff-8706-6107d3e3d2bb"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 23:38:19.789167 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:19.789136 2563 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5e3e5aa3-8bc5-49ff-8706-6107d3e3d2bb-tls-assets\") on node \"ip-10-0-128-197.ec2.internal\" DevicePath \"\"" Jul 15 23:38:19.789167 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:19.789161 2563 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/5e3e5aa3-8bc5-49ff-8706-6107d3e3d2bb-secret-alertmanager-main-tls\") on node \"ip-10-0-128-197.ec2.internal\" DevicePath \"\"" Jul 15 23:38:19.789167 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:19.789172 2563 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5e3e5aa3-8bc5-49ff-8706-6107d3e3d2bb-config-out\") on node \"ip-10-0-128-197.ec2.internal\" DevicePath \"\"" Jul 15 23:38:19.789363 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:19.789182 2563 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5e3e5aa3-8bc5-49ff-8706-6107d3e3d2bb-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-128-197.ec2.internal\" DevicePath \"\"" Jul 15 23:38:19.789363 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:19.789191 2563 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5e3e5aa3-8bc5-49ff-8706-6107d3e3d2bb-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-128-197.ec2.internal\" DevicePath \"\"" Jul 15 23:38:19.789363 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:19.789200 2563 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5e3e5aa3-8bc5-49ff-8706-6107d3e3d2bb-metrics-client-ca\") on node \"ip-10-0-128-197.ec2.internal\" DevicePath \"\"" Jul 15 23:38:19.789363 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:19.789208 2563 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/5e3e5aa3-8bc5-49ff-8706-6107d3e3d2bb-cluster-tls-config\") on node \"ip-10-0-128-197.ec2.internal\" DevicePath \"\"" Jul 15 23:38:19.789363 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:19.789217 2563 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/5e3e5aa3-8bc5-49ff-8706-6107d3e3d2bb-config-volume\") on node \"ip-10-0-128-197.ec2.internal\" DevicePath \"\"" Jul 15 23:38:19.789363 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:19.789225 2563 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/5e3e5aa3-8bc5-49ff-8706-6107d3e3d2bb-alertmanager-main-db\") on node \"ip-10-0-128-197.ec2.internal\" DevicePath \"\"" Jul 15 23:38:19.789363 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:19.789234 2563 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/5e3e5aa3-8bc5-49ff-8706-6107d3e3d2bb-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-128-197.ec2.internal\" DevicePath \"\"" Jul 15 23:38:19.789363 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:19.789243 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cwzrf\" (UniqueName: \"kubernetes.io/projected/5e3e5aa3-8bc5-49ff-8706-6107d3e3d2bb-kube-api-access-cwzrf\") on node \"ip-10-0-128-197.ec2.internal\" DevicePath \"\"" Jul 15 23:38:19.789363 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:19.789251 2563 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5e3e5aa3-8bc5-49ff-8706-6107d3e3d2bb-web-config\") on node \"ip-10-0-128-197.ec2.internal\" DevicePath \"\"" Jul 15 23:38:19.789363 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:19.789259 2563 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5e3e5aa3-8bc5-49ff-8706-6107d3e3d2bb-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-128-197.ec2.internal\" DevicePath \"\"" Jul 15 23:38:19.917214 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:19.917187 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 15 23:38:19.921023 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:19.921000 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 15 23:38:19.945289 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:19.945269 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 15 23:38:19.945563 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:19.945551 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e3e5aa3-8bc5-49ff-8706-6107d3e3d2bb" containerName="alertmanager" Jul 15 23:38:19.945604 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:19.945564 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e3e5aa3-8bc5-49ff-8706-6107d3e3d2bb" containerName="alertmanager" Jul 15 23:38:19.945604 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:19.945580 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5efd853d-ff2f-43d9-bbe8-fd8ee02465ec" containerName="console" Jul 15 23:38:19.945604 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:19.945585 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="5efd853d-ff2f-43d9-bbe8-fd8ee02465ec" containerName="console" Jul 15 23:38:19.945604 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:19.945597 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e3e5aa3-8bc5-49ff-8706-6107d3e3d2bb" containerName="config-reloader" Jul 15 23:38:19.945604 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:19.945603 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e3e5aa3-8bc5-49ff-8706-6107d3e3d2bb" containerName="config-reloader" Jul 15 23:38:19.945743 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:19.945610 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e3e5aa3-8bc5-49ff-8706-6107d3e3d2bb" containerName="prom-label-proxy" Jul 15 23:38:19.945743 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:19.945616 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e3e5aa3-8bc5-49ff-8706-6107d3e3d2bb" containerName="prom-label-proxy" Jul 15 23:38:19.945743 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:19.945624 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e3e5aa3-8bc5-49ff-8706-6107d3e3d2bb" containerName="kube-rbac-proxy-metric" Jul 15 23:38:19.945743 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:19.945630 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e3e5aa3-8bc5-49ff-8706-6107d3e3d2bb" containerName="kube-rbac-proxy-metric" Jul 15 23:38:19.945743 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:19.945637 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e3e5aa3-8bc5-49ff-8706-6107d3e3d2bb" containerName="kube-rbac-proxy" Jul 15 23:38:19.945743 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:19.945641 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e3e5aa3-8bc5-49ff-8706-6107d3e3d2bb" containerName="kube-rbac-proxy" Jul 15 23:38:19.945743 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:19.945648 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e3e5aa3-8bc5-49ff-8706-6107d3e3d2bb" containerName="init-config-reloader" Jul 15 23:38:19.945743 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:19.945654 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e3e5aa3-8bc5-49ff-8706-6107d3e3d2bb" containerName="init-config-reloader" Jul 15 23:38:19.945743 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:19.945661 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e3e5aa3-8bc5-49ff-8706-6107d3e3d2bb" containerName="kube-rbac-proxy-web" Jul 15 23:38:19.945743 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:19.945665 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e3e5aa3-8bc5-49ff-8706-6107d3e3d2bb" containerName="kube-rbac-proxy-web" Jul 15 23:38:19.945743 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:19.945711 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="5efd853d-ff2f-43d9-bbe8-fd8ee02465ec" containerName="console" Jul 15 23:38:19.945743 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:19.945719 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="5e3e5aa3-8bc5-49ff-8706-6107d3e3d2bb" containerName="config-reloader" Jul 15 23:38:19.945743 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:19.945724 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="5e3e5aa3-8bc5-49ff-8706-6107d3e3d2bb" containerName="kube-rbac-proxy" Jul 15 23:38:19.945743 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:19.945731 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="5e3e5aa3-8bc5-49ff-8706-6107d3e3d2bb" containerName="kube-rbac-proxy-metric" Jul 15 23:38:19.945743 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:19.945736 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="5e3e5aa3-8bc5-49ff-8706-6107d3e3d2bb" containerName="kube-rbac-proxy-web" Jul 15 23:38:19.945743 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:19.945742 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="5e3e5aa3-8bc5-49ff-8706-6107d3e3d2bb" containerName="prom-label-proxy" Jul 15 23:38:19.945743 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:19.945748 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="5e3e5aa3-8bc5-49ff-8706-6107d3e3d2bb" containerName="alertmanager" Jul 15 23:38:19.975810 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:19.975788 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 15 23:38:19.975924 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:19.975911 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 15 23:38:19.978361 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:19.978339 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 15 23:38:19.978361 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:19.978348 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 15 23:38:19.978549 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:19.978348 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 15 23:38:19.978549 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:19.978433 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 15 23:38:19.978744 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:19.978694 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 15 23:38:19.978857 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:19.978745 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 15 23:38:19.978857 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:19.978774 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 15 23:38:19.978857 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:19.978812 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-6xzdv\"" Jul 15 23:38:19.979116 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:19.979047 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 15 23:38:19.983047 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:19.983028 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 15 23:38:20.092145 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:20.092115 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/76c2c664-fc68-45f0-90d7-839b197c0fac-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"76c2c664-fc68-45f0-90d7-839b197c0fac\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 23:38:20.092145 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:20.092149 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2wk4\" (UniqueName: \"kubernetes.io/projected/76c2c664-fc68-45f0-90d7-839b197c0fac-kube-api-access-g2wk4\") pod \"alertmanager-main-0\" (UID: \"76c2c664-fc68-45f0-90d7-839b197c0fac\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 23:38:20.092337 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:20.092173 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/76c2c664-fc68-45f0-90d7-839b197c0fac-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"76c2c664-fc68-45f0-90d7-839b197c0fac\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 23:38:20.092337 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:20.092246 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/76c2c664-fc68-45f0-90d7-839b197c0fac-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"76c2c664-fc68-45f0-90d7-839b197c0fac\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 23:38:20.092337 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:20.092283 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/76c2c664-fc68-45f0-90d7-839b197c0fac-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"76c2c664-fc68-45f0-90d7-839b197c0fac\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 23:38:20.092337 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:20.092306 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/76c2c664-fc68-45f0-90d7-839b197c0fac-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"76c2c664-fc68-45f0-90d7-839b197c0fac\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 23:38:20.092337 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:20.092327 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/76c2c664-fc68-45f0-90d7-839b197c0fac-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"76c2c664-fc68-45f0-90d7-839b197c0fac\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 23:38:20.092486 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:20.092399 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/76c2c664-fc68-45f0-90d7-839b197c0fac-config-out\") pod \"alertmanager-main-0\" (UID: \"76c2c664-fc68-45f0-90d7-839b197c0fac\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 23:38:20.092486 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:20.092426 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/76c2c664-fc68-45f0-90d7-839b197c0fac-config-volume\") pod \"alertmanager-main-0\" (UID: \"76c2c664-fc68-45f0-90d7-839b197c0fac\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 23:38:20.092486 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:20.092450 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/76c2c664-fc68-45f0-90d7-839b197c0fac-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"76c2c664-fc68-45f0-90d7-839b197c0fac\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 23:38:20.092486 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:20.092467 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/76c2c664-fc68-45f0-90d7-839b197c0fac-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"76c2c664-fc68-45f0-90d7-839b197c0fac\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 23:38:20.092622 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:20.092488 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/76c2c664-fc68-45f0-90d7-839b197c0fac-web-config\") pod \"alertmanager-main-0\" (UID: \"76c2c664-fc68-45f0-90d7-839b197c0fac\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 23:38:20.092622 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:20.092513 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/76c2c664-fc68-45f0-90d7-839b197c0fac-tls-assets\") pod \"alertmanager-main-0\" (UID: \"76c2c664-fc68-45f0-90d7-839b197c0fac\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 23:38:20.193515 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:20.193444 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/76c2c664-fc68-45f0-90d7-839b197c0fac-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"76c2c664-fc68-45f0-90d7-839b197c0fac\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 23:38:20.193515 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:20.193476 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2wk4\" (UniqueName: \"kubernetes.io/projected/76c2c664-fc68-45f0-90d7-839b197c0fac-kube-api-access-g2wk4\") pod \"alertmanager-main-0\" (UID: \"76c2c664-fc68-45f0-90d7-839b197c0fac\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 23:38:20.193515 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:20.193503 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/76c2c664-fc68-45f0-90d7-839b197c0fac-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"76c2c664-fc68-45f0-90d7-839b197c0fac\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 23:38:20.193778 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:20.193755 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/76c2c664-fc68-45f0-90d7-839b197c0fac-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"76c2c664-fc68-45f0-90d7-839b197c0fac\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 23:38:20.193836 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:20.193801 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/76c2c664-fc68-45f0-90d7-839b197c0fac-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"76c2c664-fc68-45f0-90d7-839b197c0fac\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 23:38:20.193871 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:20.193846 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/76c2c664-fc68-45f0-90d7-839b197c0fac-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"76c2c664-fc68-45f0-90d7-839b197c0fac\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 23:38:20.193871 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:20.193857 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/76c2c664-fc68-45f0-90d7-839b197c0fac-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"76c2c664-fc68-45f0-90d7-839b197c0fac\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 23:38:20.193947 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:20.193900 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/76c2c664-fc68-45f0-90d7-839b197c0fac-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"76c2c664-fc68-45f0-90d7-839b197c0fac\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 23:38:20.194017 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:20.193957 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/76c2c664-fc68-45f0-90d7-839b197c0fac-config-out\") pod \"alertmanager-main-0\" (UID: \"76c2c664-fc68-45f0-90d7-839b197c0fac\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 23:38:20.194017 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:20.194011 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/76c2c664-fc68-45f0-90d7-839b197c0fac-config-volume\") pod \"alertmanager-main-0\" (UID: \"76c2c664-fc68-45f0-90d7-839b197c0fac\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 23:38:20.194113 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:20.194048 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/76c2c664-fc68-45f0-90d7-839b197c0fac-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"76c2c664-fc68-45f0-90d7-839b197c0fac\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 23:38:20.194113 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:20.194071 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/76c2c664-fc68-45f0-90d7-839b197c0fac-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"76c2c664-fc68-45f0-90d7-839b197c0fac\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 23:38:20.194113 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:20.194103 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/76c2c664-fc68-45f0-90d7-839b197c0fac-web-config\") pod \"alertmanager-main-0\" (UID: \"76c2c664-fc68-45f0-90d7-839b197c0fac\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 23:38:20.194257 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:20.194132 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/76c2c664-fc68-45f0-90d7-839b197c0fac-tls-assets\") pod \"alertmanager-main-0\" (UID: \"76c2c664-fc68-45f0-90d7-839b197c0fac\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 23:38:20.195002 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:20.194677 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/76c2c664-fc68-45f0-90d7-839b197c0fac-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"76c2c664-fc68-45f0-90d7-839b197c0fac\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 23:38:20.195002 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:20.194727 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/76c2c664-fc68-45f0-90d7-839b197c0fac-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"76c2c664-fc68-45f0-90d7-839b197c0fac\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 23:38:20.196599 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:20.196572 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/76c2c664-fc68-45f0-90d7-839b197c0fac-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"76c2c664-fc68-45f0-90d7-839b197c0fac\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 23:38:20.196812 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:20.196756 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/76c2c664-fc68-45f0-90d7-839b197c0fac-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"76c2c664-fc68-45f0-90d7-839b197c0fac\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 23:38:20.196812 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:20.196773 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/76c2c664-fc68-45f0-90d7-839b197c0fac-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"76c2c664-fc68-45f0-90d7-839b197c0fac\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 23:38:20.196812 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:20.196775 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/76c2c664-fc68-45f0-90d7-839b197c0fac-config-volume\") pod \"alertmanager-main-0\" (UID: \"76c2c664-fc68-45f0-90d7-839b197c0fac\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 23:38:20.196812 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:20.196808 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/76c2c664-fc68-45f0-90d7-839b197c0fac-tls-assets\") pod \"alertmanager-main-0\" (UID: \"76c2c664-fc68-45f0-90d7-839b197c0fac\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 23:38:20.197030 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:20.196807 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/76c2c664-fc68-45f0-90d7-839b197c0fac-config-out\") pod \"alertmanager-main-0\" (UID: \"76c2c664-fc68-45f0-90d7-839b197c0fac\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 23:38:20.197357 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:20.197340 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/76c2c664-fc68-45f0-90d7-839b197c0fac-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"76c2c664-fc68-45f0-90d7-839b197c0fac\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 23:38:20.197701 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:20.197688 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/76c2c664-fc68-45f0-90d7-839b197c0fac-web-config\") pod \"alertmanager-main-0\" (UID: \"76c2c664-fc68-45f0-90d7-839b197c0fac\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 23:38:20.198518 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:20.198500 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/76c2c664-fc68-45f0-90d7-839b197c0fac-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"76c2c664-fc68-45f0-90d7-839b197c0fac\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 23:38:20.200742 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:20.200722 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2wk4\" (UniqueName: \"kubernetes.io/projected/76c2c664-fc68-45f0-90d7-839b197c0fac-kube-api-access-g2wk4\") pod \"alertmanager-main-0\" (UID: \"76c2c664-fc68-45f0-90d7-839b197c0fac\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 23:38:20.285792 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:20.285760 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 15 23:38:20.442279 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:20.441777 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 15 23:38:20.606697 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:20.606663 2563 generic.go:356] "Generic (PLEG): container finished" podID="76c2c664-fc68-45f0-90d7-839b197c0fac" containerID="bd8487372ab5e471416700a5e6de92d6eb33931d507d3faed8da1e49097f8a9d" exitCode=0 Jul 15 23:38:20.607014 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:20.606700 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"76c2c664-fc68-45f0-90d7-839b197c0fac","Type":"ContainerDied","Data":"bd8487372ab5e471416700a5e6de92d6eb33931d507d3faed8da1e49097f8a9d"} Jul 15 23:38:20.607014 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:20.606787 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"76c2c664-fc68-45f0-90d7-839b197c0fac","Type":"ContainerStarted","Data":"ed4e053e757f66c5a3268bda882dbe7da32e7fa755536c1c85d622996b4941d1"} Jul 15 23:38:21.169323 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:21.169297 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5e3e5aa3-8bc5-49ff-8706-6107d3e3d2bb" path="/var/lib/kubelet/pods/5e3e5aa3-8bc5-49ff-8706-6107d3e3d2bb/volumes" Jul 15 23:38:21.614644 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:21.614559 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"76c2c664-fc68-45f0-90d7-839b197c0fac","Type":"ContainerStarted","Data":"2f31701206bcf06604297c0d06137d3c4afd2348a5da8e984fee2f926ddb12e0"} Jul 15 23:38:21.614644 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:21.614620 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"76c2c664-fc68-45f0-90d7-839b197c0fac","Type":"ContainerStarted","Data":"b3372320778a77638f56c23ad7a0c650e66e5f819ee4ff70b7adbd243edf4176"} Jul 15 23:38:21.615109 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:21.614651 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"76c2c664-fc68-45f0-90d7-839b197c0fac","Type":"ContainerStarted","Data":"1e8927fbf0bc1d9c20bede3083e297a92e50e698b32b61fe10a00192fad86798"} Jul 15 23:38:21.615109 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:21.614685 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"76c2c664-fc68-45f0-90d7-839b197c0fac","Type":"ContainerStarted","Data":"5faab0b1f5781488479f0d78d8c41c4456f9407143c5f6b7ceb7a0f22a0ab17c"} Jul 15 23:38:21.615109 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:21.614719 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"76c2c664-fc68-45f0-90d7-839b197c0fac","Type":"ContainerStarted","Data":"c1848fd6b9b482b4a296830288005e1a8cf41b26b73d08d9e05f1ae47467d34e"} Jul 15 23:38:21.615109 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:21.614744 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"76c2c664-fc68-45f0-90d7-839b197c0fac","Type":"ContainerStarted","Data":"cf5ded942486ed23fe7e06a83e0895d8e109d08b607af98f4c90d73332748039"} Jul 15 23:38:21.638561 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:21.638512 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.6260257510000002 podStartE2EDuration="2.63849649s" totalImagesPullingTime="0s" totalInitContainerRuntime="12.470739ms" isStatefulPod=false podCreationTimestamp="2026-07-15 23:38:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 23:38:21.636105303 +0000 UTC m=+117.009563266" watchObservedRunningTime="2026-07-15 23:38:21.63849649 +0000 UTC m=+117.011954452" Jul 15 23:38:23.522789 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:23.522757 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 15 23:38:23.523242 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:23.523106 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="a881fb94-9a0b-4c46-b5fc-d841398d576d" containerName="thanos-sidecar" containerID="cri-o://623e6174e579792885b393eef036d66bec7d8fe27570aa09b14e3d7cf266d7a5" gracePeriod=600 Jul 15 23:38:23.523242 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:23.523114 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="a881fb94-9a0b-4c46-b5fc-d841398d576d" containerName="config-reloader" containerID="cri-o://cd1c919f29a0f532795c23a6f3e6172ab0cdff76e2d54de23e34d939a47dec1b" gracePeriod=600 Jul 15 23:38:23.523242 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:23.523128 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="a881fb94-9a0b-4c46-b5fc-d841398d576d" containerName="kube-rbac-proxy-web" containerID="cri-o://63c7e833b7f4dd49f6b4b2057181362d71af0d8fdd0d48039672e403ee88c365" gracePeriod=600 Jul 15 23:38:23.523242 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:23.523217 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="a881fb94-9a0b-4c46-b5fc-d841398d576d" containerName="kube-rbac-proxy" containerID="cri-o://f2eef3071405d6109930b1123dc53e48dee6c71643c6b82219f38fdf07154180" gracePeriod=600 Jul 15 23:38:23.523478 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:23.523266 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="a881fb94-9a0b-4c46-b5fc-d841398d576d" containerName="kube-rbac-proxy-thanos" containerID="cri-o://1555d6bba92a06a023fbe9901c90b4d739ced83bc3b4e3c5ba2acdff57649475" gracePeriod=600 Jul 15 23:38:23.523562 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:23.523285 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="a881fb94-9a0b-4c46-b5fc-d841398d576d" containerName="prometheus" containerID="cri-o://a4a887aaf513cd8202fe2c033d87ef3ba8df280b24c5113b3738377bcf5a5ee8" gracePeriod=600 Jul 15 23:38:24.513191 ip-10-0-128-197 kubenswrapper[2563]: E0715 23:38:24.513144 2563 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" containerID="a4a887aaf513cd8202fe2c033d87ef3ba8df280b24c5113b3738377bcf5a5ee8" cmd=["sh","-c","if [ -x \"$(command -v curl)\" ]; then exec curl --fail http://localhost:9090/-/ready; elif [ -x \"$(command -v wget)\" ]; then exec wget -q -O /dev/null http://localhost:9090/-/ready; else exit 1; fi"] Jul 15 23:38:24.514121 ip-10-0-128-197 kubenswrapper[2563]: E0715 23:38:24.514088 2563 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" containerID="a4a887aaf513cd8202fe2c033d87ef3ba8df280b24c5113b3738377bcf5a5ee8" cmd=["sh","-c","if [ -x \"$(command -v curl)\" ]; then exec curl --fail http://localhost:9090/-/ready; elif [ -x \"$(command -v wget)\" ]; then exec wget -q -O /dev/null http://localhost:9090/-/ready; else exit 1; fi"] Jul 15 23:38:24.515012 ip-10-0-128-197 kubenswrapper[2563]: E0715 23:38:24.514984 2563 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" containerID="a4a887aaf513cd8202fe2c033d87ef3ba8df280b24c5113b3738377bcf5a5ee8" cmd=["sh","-c","if [ -x \"$(command -v curl)\" ]; then exec curl --fail http://localhost:9090/-/ready; elif [ -x \"$(command -v wget)\" ]; then exec wget -q -O /dev/null http://localhost:9090/-/ready; else exit 1; fi"] Jul 15 23:38:24.515104 ip-10-0-128-197 kubenswrapper[2563]: E0715 23:38:24.515023 2563 prober.go:104] "Probe errored" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" probeType="Readiness" pod="openshift-monitoring/prometheus-k8s-0" podUID="a881fb94-9a0b-4c46-b5fc-d841398d576d" containerName="prometheus" probeResult="unknown" Jul 15 23:38:24.631445 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:24.631413 2563 generic.go:356] "Generic (PLEG): container finished" podID="a881fb94-9a0b-4c46-b5fc-d841398d576d" containerID="1555d6bba92a06a023fbe9901c90b4d739ced83bc3b4e3c5ba2acdff57649475" exitCode=0 Jul 15 23:38:24.631445 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:24.631441 2563 generic.go:356] "Generic (PLEG): container finished" podID="a881fb94-9a0b-4c46-b5fc-d841398d576d" containerID="f2eef3071405d6109930b1123dc53e48dee6c71643c6b82219f38fdf07154180" exitCode=0 Jul 15 23:38:24.631844 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:24.631452 2563 generic.go:356] "Generic (PLEG): container finished" podID="a881fb94-9a0b-4c46-b5fc-d841398d576d" containerID="63c7e833b7f4dd49f6b4b2057181362d71af0d8fdd0d48039672e403ee88c365" exitCode=0 Jul 15 23:38:24.631844 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:24.631460 2563 generic.go:356] "Generic (PLEG): container finished" podID="a881fb94-9a0b-4c46-b5fc-d841398d576d" containerID="623e6174e579792885b393eef036d66bec7d8fe27570aa09b14e3d7cf266d7a5" exitCode=0 Jul 15 23:38:24.631844 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:24.631468 2563 generic.go:356] "Generic (PLEG): container finished" podID="a881fb94-9a0b-4c46-b5fc-d841398d576d" containerID="cd1c919f29a0f532795c23a6f3e6172ab0cdff76e2d54de23e34d939a47dec1b" exitCode=0 Jul 15 23:38:24.631844 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:24.631482 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a881fb94-9a0b-4c46-b5fc-d841398d576d","Type":"ContainerDied","Data":"1555d6bba92a06a023fbe9901c90b4d739ced83bc3b4e3c5ba2acdff57649475"} Jul 15 23:38:24.631844 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:24.631575 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a881fb94-9a0b-4c46-b5fc-d841398d576d","Type":"ContainerDied","Data":"f2eef3071405d6109930b1123dc53e48dee6c71643c6b82219f38fdf07154180"} Jul 15 23:38:24.631844 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:24.631605 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a881fb94-9a0b-4c46-b5fc-d841398d576d","Type":"ContainerDied","Data":"63c7e833b7f4dd49f6b4b2057181362d71af0d8fdd0d48039672e403ee88c365"} Jul 15 23:38:24.631844 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:24.631629 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a881fb94-9a0b-4c46-b5fc-d841398d576d","Type":"ContainerDied","Data":"623e6174e579792885b393eef036d66bec7d8fe27570aa09b14e3d7cf266d7a5"} Jul 15 23:38:24.631844 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:24.631652 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a881fb94-9a0b-4c46-b5fc-d841398d576d","Type":"ContainerDied","Data":"cd1c919f29a0f532795c23a6f3e6172ab0cdff76e2d54de23e34d939a47dec1b"} Jul 15 23:38:26.829718 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:26.829689 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:38:26.874999 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:26.874953 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:38:29.129550 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:29.129515 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:38:29.185190 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:29.185160 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:38:29.273368 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:29.273335 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:38:29.512816 ip-10-0-128-197 kubenswrapper[2563]: E0715 23:38:29.512776 2563 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" containerID="a4a887aaf513cd8202fe2c033d87ef3ba8df280b24c5113b3738377bcf5a5ee8" cmd=["sh","-c","if [ -x \"$(command -v curl)\" ]; then exec curl --fail http://localhost:9090/-/ready; elif [ -x \"$(command -v wget)\" ]; then exec wget -q -O /dev/null http://localhost:9090/-/ready; else exit 1; fi"] Jul 15 23:38:29.513810 ip-10-0-128-197 kubenswrapper[2563]: E0715 23:38:29.513769 2563 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" containerID="a4a887aaf513cd8202fe2c033d87ef3ba8df280b24c5113b3738377bcf5a5ee8" cmd=["sh","-c","if [ -x \"$(command -v curl)\" ]; then exec curl --fail http://localhost:9090/-/ready; elif [ -x \"$(command -v wget)\" ]; then exec wget -q -O /dev/null http://localhost:9090/-/ready; else exit 1; fi"] Jul 15 23:38:29.514690 ip-10-0-128-197 kubenswrapper[2563]: E0715 23:38:29.514655 2563 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" containerID="a4a887aaf513cd8202fe2c033d87ef3ba8df280b24c5113b3738377bcf5a5ee8" cmd=["sh","-c","if [ -x \"$(command -v curl)\" ]; then exec curl --fail http://localhost:9090/-/ready; elif [ -x \"$(command -v wget)\" ]; then exec wget -q -O /dev/null http://localhost:9090/-/ready; else exit 1; fi"] Jul 15 23:38:29.514771 ip-10-0-128-197 kubenswrapper[2563]: E0715 23:38:29.514705 2563 prober.go:104] "Probe errored" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" probeType="Readiness" pod="openshift-monitoring/prometheus-k8s-0" podUID="a881fb94-9a0b-4c46-b5fc-d841398d576d" containerName="prometheus" probeResult="unknown" Jul 15 23:38:29.873153 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:29.873072 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:38:29.927113 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:29.927085 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:38:30.661898 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:30.661818 2563 generic.go:356] "Generic (PLEG): container finished" podID="a881fb94-9a0b-4c46-b5fc-d841398d576d" containerID="a4a887aaf513cd8202fe2c033d87ef3ba8df280b24c5113b3738377bcf5a5ee8" exitCode=0 Jul 15 23:38:30.662247 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:30.661889 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a881fb94-9a0b-4c46-b5fc-d841398d576d","Type":"ContainerDied","Data":"a4a887aaf513cd8202fe2c033d87ef3ba8df280b24c5113b3738377bcf5a5ee8"} Jul 15 23:38:30.920737 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:30.920681 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:38:30.968908 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:30.968880 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:38:30.995975 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:30.995944 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 15 23:38:31.083670 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:31.083643 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a881fb94-9a0b-4c46-b5fc-d841398d576d-configmap-metrics-client-ca\") pod \"a881fb94-9a0b-4c46-b5fc-d841398d576d\" (UID: \"a881fb94-9a0b-4c46-b5fc-d841398d576d\") " Jul 15 23:38:31.083670 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:31.083681 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/a881fb94-9a0b-4c46-b5fc-d841398d576d-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"a881fb94-9a0b-4c46-b5fc-d841398d576d\" (UID: \"a881fb94-9a0b-4c46-b5fc-d841398d576d\") " Jul 15 23:38:31.083875 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:31.083700 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a881fb94-9a0b-4c46-b5fc-d841398d576d-prometheus-trusted-ca-bundle\") pod \"a881fb94-9a0b-4c46-b5fc-d841398d576d\" (UID: \"a881fb94-9a0b-4c46-b5fc-d841398d576d\") " Jul 15 23:38:31.083875 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:31.083725 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a881fb94-9a0b-4c46-b5fc-d841398d576d-configmap-kubelet-serving-ca-bundle\") pod \"a881fb94-9a0b-4c46-b5fc-d841398d576d\" (UID: \"a881fb94-9a0b-4c46-b5fc-d841398d576d\") " Jul 15 23:38:31.083875 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:31.083756 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/a881fb94-9a0b-4c46-b5fc-d841398d576d-secret-prometheus-k8s-tls\") pod \"a881fb94-9a0b-4c46-b5fc-d841398d576d\" (UID: \"a881fb94-9a0b-4c46-b5fc-d841398d576d\") " Jul 15 23:38:31.083875 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:31.083779 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7xmvk\" (UniqueName: \"kubernetes.io/projected/a881fb94-9a0b-4c46-b5fc-d841398d576d-kube-api-access-7xmvk\") pod \"a881fb94-9a0b-4c46-b5fc-d841398d576d\" (UID: \"a881fb94-9a0b-4c46-b5fc-d841398d576d\") " Jul 15 23:38:31.083875 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:31.083797 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a881fb94-9a0b-4c46-b5fc-d841398d576d-prometheus-k8s-rulefiles-2\") pod \"a881fb94-9a0b-4c46-b5fc-d841398d576d\" (UID: \"a881fb94-9a0b-4c46-b5fc-d841398d576d\") " Jul 15 23:38:31.083875 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:31.083827 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a881fb94-9a0b-4c46-b5fc-d841398d576d-web-config\") pod \"a881fb94-9a0b-4c46-b5fc-d841398d576d\" (UID: \"a881fb94-9a0b-4c46-b5fc-d841398d576d\") " Jul 15 23:38:31.084146 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:31.083881 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/a881fb94-9a0b-4c46-b5fc-d841398d576d-secret-metrics-client-certs\") pod \"a881fb94-9a0b-4c46-b5fc-d841398d576d\" (UID: \"a881fb94-9a0b-4c46-b5fc-d841398d576d\") " Jul 15 23:38:31.084146 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:31.083921 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a881fb94-9a0b-4c46-b5fc-d841398d576d-tls-assets\") pod \"a881fb94-9a0b-4c46-b5fc-d841398d576d\" (UID: \"a881fb94-9a0b-4c46-b5fc-d841398d576d\") " Jul 15 23:38:31.084146 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:31.083951 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/a881fb94-9a0b-4c46-b5fc-d841398d576d-prometheus-k8s-db\") pod \"a881fb94-9a0b-4c46-b5fc-d841398d576d\" (UID: \"a881fb94-9a0b-4c46-b5fc-d841398d576d\") " Jul 15 23:38:31.084146 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:31.083991 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a881fb94-9a0b-4c46-b5fc-d841398d576d-prometheus-k8s-rulefiles-0\") pod \"a881fb94-9a0b-4c46-b5fc-d841398d576d\" (UID: \"a881fb94-9a0b-4c46-b5fc-d841398d576d\") " Jul 15 23:38:31.084146 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:31.084027 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a881fb94-9a0b-4c46-b5fc-d841398d576d-configmap-serving-certs-ca-bundle\") pod \"a881fb94-9a0b-4c46-b5fc-d841398d576d\" (UID: \"a881fb94-9a0b-4c46-b5fc-d841398d576d\") " Jul 15 23:38:31.084146 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:31.084084 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a881fb94-9a0b-4c46-b5fc-d841398d576d-prometheus-k8s-rulefiles-1\") pod \"a881fb94-9a0b-4c46-b5fc-d841398d576d\" (UID: \"a881fb94-9a0b-4c46-b5fc-d841398d576d\") " Jul 15 23:38:31.084146 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:31.084092 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a881fb94-9a0b-4c46-b5fc-d841398d576d-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "a881fb94-9a0b-4c46-b5fc-d841398d576d" (UID: "a881fb94-9a0b-4c46-b5fc-d841398d576d"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 23:38:31.084146 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:31.084107 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a881fb94-9a0b-4c46-b5fc-d841398d576d-secret-grpc-tls\") pod \"a881fb94-9a0b-4c46-b5fc-d841398d576d\" (UID: \"a881fb94-9a0b-4c46-b5fc-d841398d576d\") " Jul 15 23:38:31.084526 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:31.084174 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a881fb94-9a0b-4c46-b5fc-d841398d576d-config-out\") pod \"a881fb94-9a0b-4c46-b5fc-d841398d576d\" (UID: \"a881fb94-9a0b-4c46-b5fc-d841398d576d\") " Jul 15 23:38:31.084526 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:31.084290 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/a881fb94-9a0b-4c46-b5fc-d841398d576d-config\") pod \"a881fb94-9a0b-4c46-b5fc-d841398d576d\" (UID: \"a881fb94-9a0b-4c46-b5fc-d841398d576d\") " Jul 15 23:38:31.084526 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:31.084320 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a881fb94-9a0b-4c46-b5fc-d841398d576d-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"a881fb94-9a0b-4c46-b5fc-d841398d576d\" (UID: \"a881fb94-9a0b-4c46-b5fc-d841398d576d\") " Jul 15 23:38:31.084526 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:31.084350 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a881fb94-9a0b-4c46-b5fc-d841398d576d-secret-kube-rbac-proxy\") pod \"a881fb94-9a0b-4c46-b5fc-d841398d576d\" (UID: \"a881fb94-9a0b-4c46-b5fc-d841398d576d\") " Jul 15 23:38:31.084526 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:31.084374 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a881fb94-9a0b-4c46-b5fc-d841398d576d-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "a881fb94-9a0b-4c46-b5fc-d841398d576d" (UID: "a881fb94-9a0b-4c46-b5fc-d841398d576d"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 23:38:31.084526 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:31.084381 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a881fb94-9a0b-4c46-b5fc-d841398d576d-thanos-prometheus-http-client-file\") pod \"a881fb94-9a0b-4c46-b5fc-d841398d576d\" (UID: \"a881fb94-9a0b-4c46-b5fc-d841398d576d\") " Jul 15 23:38:31.084526 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:31.084489 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a881fb94-9a0b-4c46-b5fc-d841398d576d-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "a881fb94-9a0b-4c46-b5fc-d841398d576d" (UID: "a881fb94-9a0b-4c46-b5fc-d841398d576d"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 23:38:31.084849 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:31.084749 2563 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a881fb94-9a0b-4c46-b5fc-d841398d576d-configmap-metrics-client-ca\") on node \"ip-10-0-128-197.ec2.internal\" DevicePath \"\"" Jul 15 23:38:31.084849 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:31.084771 2563 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a881fb94-9a0b-4c46-b5fc-d841398d576d-prometheus-trusted-ca-bundle\") on node \"ip-10-0-128-197.ec2.internal\" DevicePath \"\"" Jul 15 23:38:31.084849 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:31.084787 2563 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a881fb94-9a0b-4c46-b5fc-d841398d576d-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-128-197.ec2.internal\" DevicePath \"\"" Jul 15 23:38:31.085615 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:31.085393 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a881fb94-9a0b-4c46-b5fc-d841398d576d-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "a881fb94-9a0b-4c46-b5fc-d841398d576d" (UID: "a881fb94-9a0b-4c46-b5fc-d841398d576d"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 23:38:31.085945 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:31.085786 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a881fb94-9a0b-4c46-b5fc-d841398d576d-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "a881fb94-9a0b-4c46-b5fc-d841398d576d" (UID: "a881fb94-9a0b-4c46-b5fc-d841398d576d"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 23:38:31.086382 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:31.086329 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a881fb94-9a0b-4c46-b5fc-d841398d576d-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "a881fb94-9a0b-4c46-b5fc-d841398d576d" (UID: "a881fb94-9a0b-4c46-b5fc-d841398d576d"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 23:38:31.087327 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:31.087290 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a881fb94-9a0b-4c46-b5fc-d841398d576d-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "a881fb94-9a0b-4c46-b5fc-d841398d576d" (UID: "a881fb94-9a0b-4c46-b5fc-d841398d576d"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 23:38:31.087762 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:31.087734 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a881fb94-9a0b-4c46-b5fc-d841398d576d-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "a881fb94-9a0b-4c46-b5fc-d841398d576d" (UID: "a881fb94-9a0b-4c46-b5fc-d841398d576d"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 23:38:31.088042 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:31.088017 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a881fb94-9a0b-4c46-b5fc-d841398d576d-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "a881fb94-9a0b-4c46-b5fc-d841398d576d" (UID: "a881fb94-9a0b-4c46-b5fc-d841398d576d"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 23:38:31.088042 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:31.088029 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a881fb94-9a0b-4c46-b5fc-d841398d576d-kube-api-access-7xmvk" (OuterVolumeSpecName: "kube-api-access-7xmvk") pod "a881fb94-9a0b-4c46-b5fc-d841398d576d" (UID: "a881fb94-9a0b-4c46-b5fc-d841398d576d"). InnerVolumeSpecName "kube-api-access-7xmvk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 23:38:31.088444 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:31.088411 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a881fb94-9a0b-4c46-b5fc-d841398d576d-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "a881fb94-9a0b-4c46-b5fc-d841398d576d" (UID: "a881fb94-9a0b-4c46-b5fc-d841398d576d"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 23:38:31.088685 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:31.088665 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a881fb94-9a0b-4c46-b5fc-d841398d576d-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "a881fb94-9a0b-4c46-b5fc-d841398d576d" (UID: "a881fb94-9a0b-4c46-b5fc-d841398d576d"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 23:38:31.088848 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:31.088823 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a881fb94-9a0b-4c46-b5fc-d841398d576d-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "a881fb94-9a0b-4c46-b5fc-d841398d576d" (UID: "a881fb94-9a0b-4c46-b5fc-d841398d576d"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 23:38:31.089254 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:31.089205 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a881fb94-9a0b-4c46-b5fc-d841398d576d-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "a881fb94-9a0b-4c46-b5fc-d841398d576d" (UID: "a881fb94-9a0b-4c46-b5fc-d841398d576d"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 23:38:31.089254 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:31.089267 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a881fb94-9a0b-4c46-b5fc-d841398d576d-config-out" (OuterVolumeSpecName: "config-out") pod "a881fb94-9a0b-4c46-b5fc-d841398d576d" (UID: "a881fb94-9a0b-4c46-b5fc-d841398d576d"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 23:38:31.089254 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:31.089299 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a881fb94-9a0b-4c46-b5fc-d841398d576d-config" (OuterVolumeSpecName: "config") pod "a881fb94-9a0b-4c46-b5fc-d841398d576d" (UID: "a881fb94-9a0b-4c46-b5fc-d841398d576d"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 23:38:31.089539 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:31.089346 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a881fb94-9a0b-4c46-b5fc-d841398d576d-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "a881fb94-9a0b-4c46-b5fc-d841398d576d" (UID: "a881fb94-9a0b-4c46-b5fc-d841398d576d"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 23:38:31.089934 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:31.089905 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a881fb94-9a0b-4c46-b5fc-d841398d576d-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "a881fb94-9a0b-4c46-b5fc-d841398d576d" (UID: "a881fb94-9a0b-4c46-b5fc-d841398d576d"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 23:38:31.090686 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:31.090663 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a881fb94-9a0b-4c46-b5fc-d841398d576d-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "a881fb94-9a0b-4c46-b5fc-d841398d576d" (UID: "a881fb94-9a0b-4c46-b5fc-d841398d576d"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 23:38:31.099498 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:31.099475 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a881fb94-9a0b-4c46-b5fc-d841398d576d-web-config" (OuterVolumeSpecName: "web-config") pod "a881fb94-9a0b-4c46-b5fc-d841398d576d" (UID: "a881fb94-9a0b-4c46-b5fc-d841398d576d"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 23:38:31.185907 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:31.185884 2563 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/a881fb94-9a0b-4c46-b5fc-d841398d576d-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-128-197.ec2.internal\" DevicePath \"\"" Jul 15 23:38:31.185907 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:31.185906 2563 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/a881fb94-9a0b-4c46-b5fc-d841398d576d-secret-prometheus-k8s-tls\") on node \"ip-10-0-128-197.ec2.internal\" DevicePath \"\"" Jul 15 23:38:31.186043 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:31.185920 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7xmvk\" (UniqueName: \"kubernetes.io/projected/a881fb94-9a0b-4c46-b5fc-d841398d576d-kube-api-access-7xmvk\") on node \"ip-10-0-128-197.ec2.internal\" DevicePath \"\"" Jul 15 23:38:31.186043 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:31.185928 2563 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a881fb94-9a0b-4c46-b5fc-d841398d576d-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-128-197.ec2.internal\" DevicePath \"\"" Jul 15 23:38:31.186043 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:31.185938 2563 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a881fb94-9a0b-4c46-b5fc-d841398d576d-web-config\") on node \"ip-10-0-128-197.ec2.internal\" DevicePath \"\"" Jul 15 23:38:31.186043 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:31.185946 2563 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/a881fb94-9a0b-4c46-b5fc-d841398d576d-secret-metrics-client-certs\") on node \"ip-10-0-128-197.ec2.internal\" DevicePath \"\"" Jul 15 23:38:31.186043 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:31.185955 2563 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a881fb94-9a0b-4c46-b5fc-d841398d576d-tls-assets\") on node \"ip-10-0-128-197.ec2.internal\" DevicePath \"\"" Jul 15 23:38:31.186043 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:31.185983 2563 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/a881fb94-9a0b-4c46-b5fc-d841398d576d-prometheus-k8s-db\") on node \"ip-10-0-128-197.ec2.internal\" DevicePath \"\"" Jul 15 23:38:31.186043 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:31.185996 2563 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a881fb94-9a0b-4c46-b5fc-d841398d576d-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-128-197.ec2.internal\" DevicePath \"\"" Jul 15 23:38:31.186043 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:31.186006 2563 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a881fb94-9a0b-4c46-b5fc-d841398d576d-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-128-197.ec2.internal\" DevicePath \"\"" Jul 15 23:38:31.186043 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:31.186015 2563 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a881fb94-9a0b-4c46-b5fc-d841398d576d-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-128-197.ec2.internal\" DevicePath \"\"" Jul 15 23:38:31.186043 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:31.186023 2563 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a881fb94-9a0b-4c46-b5fc-d841398d576d-secret-grpc-tls\") on node \"ip-10-0-128-197.ec2.internal\" DevicePath \"\"" Jul 15 23:38:31.186043 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:31.186031 2563 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a881fb94-9a0b-4c46-b5fc-d841398d576d-config-out\") on node \"ip-10-0-128-197.ec2.internal\" DevicePath \"\"" Jul 15 23:38:31.186043 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:31.186039 2563 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/a881fb94-9a0b-4c46-b5fc-d841398d576d-config\") on node \"ip-10-0-128-197.ec2.internal\" DevicePath \"\"" Jul 15 23:38:31.186043 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:31.186046 2563 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a881fb94-9a0b-4c46-b5fc-d841398d576d-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-128-197.ec2.internal\" DevicePath \"\"" Jul 15 23:38:31.186414 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:31.186054 2563 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a881fb94-9a0b-4c46-b5fc-d841398d576d-secret-kube-rbac-proxy\") on node \"ip-10-0-128-197.ec2.internal\" DevicePath \"\"" Jul 15 23:38:31.186414 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:31.186062 2563 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a881fb94-9a0b-4c46-b5fc-d841398d576d-thanos-prometheus-http-client-file\") on node \"ip-10-0-128-197.ec2.internal\" DevicePath \"\"" Jul 15 23:38:31.672051 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:31.672028 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 15 23:38:31.672479 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:31.672023 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a881fb94-9a0b-4c46-b5fc-d841398d576d","Type":"ContainerDied","Data":"4728431289fcfe070efdba71508337b7ebf686259b1a1db9b984ec034efa7206"} Jul 15 23:38:31.672479 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:31.672178 2563 scope.go:117] "RemoveContainer" containerID="1555d6bba92a06a023fbe9901c90b4d739ced83bc3b4e3c5ba2acdff57649475" Jul 15 23:38:31.680626 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:31.680610 2563 scope.go:117] "RemoveContainer" containerID="f2eef3071405d6109930b1123dc53e48dee6c71643c6b82219f38fdf07154180" Jul 15 23:38:31.691130 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:31.691110 2563 scope.go:117] "RemoveContainer" containerID="63c7e833b7f4dd49f6b4b2057181362d71af0d8fdd0d48039672e403ee88c365" Jul 15 23:38:31.694010 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:31.693991 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 15 23:38:31.697652 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:31.697628 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 15 23:38:31.698836 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:31.698820 2563 scope.go:117] "RemoveContainer" containerID="623e6174e579792885b393eef036d66bec7d8fe27570aa09b14e3d7cf266d7a5" Jul 15 23:38:31.705375 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:31.705359 2563 scope.go:117] "RemoveContainer" containerID="cd1c919f29a0f532795c23a6f3e6172ab0cdff76e2d54de23e34d939a47dec1b" Jul 15 23:38:31.711749 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:31.711729 2563 scope.go:117] "RemoveContainer" containerID="a4a887aaf513cd8202fe2c033d87ef3ba8df280b24c5113b3738377bcf5a5ee8" Jul 15 23:38:31.718895 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:31.718871 2563 scope.go:117] "RemoveContainer" containerID="8f7819dd3ffe1271d15f5961909f0a003400c7c166497136c73d2661416e5a06" Jul 15 23:38:31.720249 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:31.720231 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 15 23:38:31.720565 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:31.720553 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a881fb94-9a0b-4c46-b5fc-d841398d576d" containerName="init-config-reloader" Jul 15 23:38:31.720605 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:31.720567 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="a881fb94-9a0b-4c46-b5fc-d841398d576d" containerName="init-config-reloader" Jul 15 23:38:31.720605 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:31.720577 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a881fb94-9a0b-4c46-b5fc-d841398d576d" containerName="prometheus" Jul 15 23:38:31.720605 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:31.720584 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="a881fb94-9a0b-4c46-b5fc-d841398d576d" containerName="prometheus" Jul 15 23:38:31.720605 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:31.720598 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a881fb94-9a0b-4c46-b5fc-d841398d576d" containerName="config-reloader" Jul 15 23:38:31.720605 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:31.720603 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="a881fb94-9a0b-4c46-b5fc-d841398d576d" containerName="config-reloader" Jul 15 23:38:31.720752 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:31.720612 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a881fb94-9a0b-4c46-b5fc-d841398d576d" containerName="thanos-sidecar" Jul 15 23:38:31.720752 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:31.720620 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="a881fb94-9a0b-4c46-b5fc-d841398d576d" containerName="thanos-sidecar" Jul 15 23:38:31.720752 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:31.720631 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a881fb94-9a0b-4c46-b5fc-d841398d576d" containerName="kube-rbac-proxy" Jul 15 23:38:31.720752 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:31.720636 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="a881fb94-9a0b-4c46-b5fc-d841398d576d" containerName="kube-rbac-proxy" Jul 15 23:38:31.720752 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:31.720643 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a881fb94-9a0b-4c46-b5fc-d841398d576d" containerName="kube-rbac-proxy-web" Jul 15 23:38:31.720752 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:31.720648 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="a881fb94-9a0b-4c46-b5fc-d841398d576d" containerName="kube-rbac-proxy-web" Jul 15 23:38:31.720752 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:31.720655 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a881fb94-9a0b-4c46-b5fc-d841398d576d" containerName="kube-rbac-proxy-thanos" Jul 15 23:38:31.720752 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:31.720660 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="a881fb94-9a0b-4c46-b5fc-d841398d576d" containerName="kube-rbac-proxy-thanos" Jul 15 23:38:31.720752 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:31.720712 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="a881fb94-9a0b-4c46-b5fc-d841398d576d" containerName="kube-rbac-proxy" Jul 15 23:38:31.720752 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:31.720720 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="a881fb94-9a0b-4c46-b5fc-d841398d576d" containerName="thanos-sidecar" Jul 15 23:38:31.720752 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:31.720726 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="a881fb94-9a0b-4c46-b5fc-d841398d576d" containerName="kube-rbac-proxy-thanos" Jul 15 23:38:31.720752 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:31.720732 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="a881fb94-9a0b-4c46-b5fc-d841398d576d" containerName="kube-rbac-proxy-web" Jul 15 23:38:31.720752 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:31.720738 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="a881fb94-9a0b-4c46-b5fc-d841398d576d" containerName="prometheus" Jul 15 23:38:31.720752 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:31.720743 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="a881fb94-9a0b-4c46-b5fc-d841398d576d" containerName="config-reloader" Jul 15 23:38:31.756557 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:31.756533 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 15 23:38:31.756686 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:31.756674 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 15 23:38:31.759189 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:31.759173 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 15 23:38:31.759298 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:31.759226 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-4ao1ufe85b9l1\"" Jul 15 23:38:31.759417 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:31.759396 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 15 23:38:31.759493 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:31.759395 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 15 23:38:31.759722 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:31.759699 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 15 23:38:31.759722 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:31.759716 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 15 23:38:31.759889 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:31.759732 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 15 23:38:31.760040 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:31.760024 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 15 23:38:31.760130 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:31.760037 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 15 23:38:31.760191 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:31.760162 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-z6md5\"" Jul 15 23:38:31.760283 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:31.760267 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 15 23:38:31.760358 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:31.760334 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 15 23:38:31.760422 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:31.760408 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 15 23:38:31.760481 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:31.760389 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 15 23:38:31.762103 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:31.762084 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 15 23:38:31.765518 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:31.765497 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 15 23:38:31.791228 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:31.791205 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/69f21f45-2600-4cef-9230-5c5d37e80c88-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"69f21f45-2600-4cef-9230-5c5d37e80c88\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 23:38:31.791310 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:31.791232 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/69f21f45-2600-4cef-9230-5c5d37e80c88-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"69f21f45-2600-4cef-9230-5c5d37e80c88\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 23:38:31.791310 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:31.791288 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/69f21f45-2600-4cef-9230-5c5d37e80c88-config\") pod \"prometheus-k8s-0\" (UID: \"69f21f45-2600-4cef-9230-5c5d37e80c88\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 23:38:31.791388 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:31.791320 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/69f21f45-2600-4cef-9230-5c5d37e80c88-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"69f21f45-2600-4cef-9230-5c5d37e80c88\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 23:38:31.791388 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:31.791345 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/69f21f45-2600-4cef-9230-5c5d37e80c88-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"69f21f45-2600-4cef-9230-5c5d37e80c88\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 23:38:31.791388 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:31.791375 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/69f21f45-2600-4cef-9230-5c5d37e80c88-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"69f21f45-2600-4cef-9230-5c5d37e80c88\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 23:38:31.791482 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:31.791402 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/69f21f45-2600-4cef-9230-5c5d37e80c88-web-config\") pod \"prometheus-k8s-0\" (UID: \"69f21f45-2600-4cef-9230-5c5d37e80c88\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 23:38:31.791482 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:31.791452 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/69f21f45-2600-4cef-9230-5c5d37e80c88-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"69f21f45-2600-4cef-9230-5c5d37e80c88\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 23:38:31.791540 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:31.791484 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/69f21f45-2600-4cef-9230-5c5d37e80c88-config-out\") pod \"prometheus-k8s-0\" (UID: \"69f21f45-2600-4cef-9230-5c5d37e80c88\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 23:38:31.791540 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:31.791502 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/69f21f45-2600-4cef-9230-5c5d37e80c88-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"69f21f45-2600-4cef-9230-5c5d37e80c88\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 23:38:31.791540 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:31.791521 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/69f21f45-2600-4cef-9230-5c5d37e80c88-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"69f21f45-2600-4cef-9230-5c5d37e80c88\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 23:38:31.791624 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:31.791545 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q287g\" (UniqueName: \"kubernetes.io/projected/69f21f45-2600-4cef-9230-5c5d37e80c88-kube-api-access-q287g\") pod \"prometheus-k8s-0\" (UID: \"69f21f45-2600-4cef-9230-5c5d37e80c88\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 23:38:31.791624 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:31.791575 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/69f21f45-2600-4cef-9230-5c5d37e80c88-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"69f21f45-2600-4cef-9230-5c5d37e80c88\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 23:38:31.791624 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:31.791594 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/69f21f45-2600-4cef-9230-5c5d37e80c88-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"69f21f45-2600-4cef-9230-5c5d37e80c88\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 23:38:31.791624 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:31.791611 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/69f21f45-2600-4cef-9230-5c5d37e80c88-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"69f21f45-2600-4cef-9230-5c5d37e80c88\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 23:38:31.791735 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:31.791643 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/69f21f45-2600-4cef-9230-5c5d37e80c88-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"69f21f45-2600-4cef-9230-5c5d37e80c88\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 23:38:31.791735 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:31.791683 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/69f21f45-2600-4cef-9230-5c5d37e80c88-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"69f21f45-2600-4cef-9230-5c5d37e80c88\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 23:38:31.791735 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:31.791723 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/69f21f45-2600-4cef-9230-5c5d37e80c88-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"69f21f45-2600-4cef-9230-5c5d37e80c88\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 23:38:31.791825 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:31.791761 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/69f21f45-2600-4cef-9230-5c5d37e80c88-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"69f21f45-2600-4cef-9230-5c5d37e80c88\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 23:38:31.791825 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:31.791778 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/69f21f45-2600-4cef-9230-5c5d37e80c88-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"69f21f45-2600-4cef-9230-5c5d37e80c88\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 23:38:31.892427 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:31.892406 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/69f21f45-2600-4cef-9230-5c5d37e80c88-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"69f21f45-2600-4cef-9230-5c5d37e80c88\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 23:38:31.892511 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:31.892430 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q287g\" (UniqueName: \"kubernetes.io/projected/69f21f45-2600-4cef-9230-5c5d37e80c88-kube-api-access-q287g\") pod \"prometheus-k8s-0\" (UID: \"69f21f45-2600-4cef-9230-5c5d37e80c88\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 23:38:31.892511 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:31.892453 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/69f21f45-2600-4cef-9230-5c5d37e80c88-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"69f21f45-2600-4cef-9230-5c5d37e80c88\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 23:38:31.892511 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:31.892471 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/69f21f45-2600-4cef-9230-5c5d37e80c88-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"69f21f45-2600-4cef-9230-5c5d37e80c88\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 23:38:31.892626 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:31.892564 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/69f21f45-2600-4cef-9230-5c5d37e80c88-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"69f21f45-2600-4cef-9230-5c5d37e80c88\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 23:38:31.892626 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:31.892580 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/69f21f45-2600-4cef-9230-5c5d37e80c88-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"69f21f45-2600-4cef-9230-5c5d37e80c88\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 23:38:31.892626 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:31.892606 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/69f21f45-2600-4cef-9230-5c5d37e80c88-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"69f21f45-2600-4cef-9230-5c5d37e80c88\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 23:38:31.892751 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:31.892628 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/69f21f45-2600-4cef-9230-5c5d37e80c88-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"69f21f45-2600-4cef-9230-5c5d37e80c88\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 23:38:31.892886 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:31.892860 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/69f21f45-2600-4cef-9230-5c5d37e80c88-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"69f21f45-2600-4cef-9230-5c5d37e80c88\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 23:38:31.892990 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:31.892905 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/69f21f45-2600-4cef-9230-5c5d37e80c88-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"69f21f45-2600-4cef-9230-5c5d37e80c88\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 23:38:31.892990 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:31.892951 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/69f21f45-2600-4cef-9230-5c5d37e80c88-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"69f21f45-2600-4cef-9230-5c5d37e80c88\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 23:38:31.893106 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:31.893003 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/69f21f45-2600-4cef-9230-5c5d37e80c88-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"69f21f45-2600-4cef-9230-5c5d37e80c88\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 23:38:31.893106 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:31.893023 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/69f21f45-2600-4cef-9230-5c5d37e80c88-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"69f21f45-2600-4cef-9230-5c5d37e80c88\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 23:38:31.893106 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:31.893037 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/69f21f45-2600-4cef-9230-5c5d37e80c88-config\") pod \"prometheus-k8s-0\" (UID: \"69f21f45-2600-4cef-9230-5c5d37e80c88\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 23:38:31.893106 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:31.893078 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/69f21f45-2600-4cef-9230-5c5d37e80c88-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"69f21f45-2600-4cef-9230-5c5d37e80c88\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 23:38:31.893309 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:31.893109 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/69f21f45-2600-4cef-9230-5c5d37e80c88-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"69f21f45-2600-4cef-9230-5c5d37e80c88\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 23:38:31.893309 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:31.893136 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/69f21f45-2600-4cef-9230-5c5d37e80c88-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"69f21f45-2600-4cef-9230-5c5d37e80c88\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 23:38:31.893309 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:31.893161 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/69f21f45-2600-4cef-9230-5c5d37e80c88-web-config\") pod \"prometheus-k8s-0\" (UID: \"69f21f45-2600-4cef-9230-5c5d37e80c88\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 23:38:31.893309 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:31.893205 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/69f21f45-2600-4cef-9230-5c5d37e80c88-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"69f21f45-2600-4cef-9230-5c5d37e80c88\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 23:38:31.893309 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:31.893237 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/69f21f45-2600-4cef-9230-5c5d37e80c88-config-out\") pod \"prometheus-k8s-0\" (UID: \"69f21f45-2600-4cef-9230-5c5d37e80c88\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 23:38:31.893309 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:31.893259 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/69f21f45-2600-4cef-9230-5c5d37e80c88-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"69f21f45-2600-4cef-9230-5c5d37e80c88\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 23:38:31.894342 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:31.893788 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/69f21f45-2600-4cef-9230-5c5d37e80c88-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"69f21f45-2600-4cef-9230-5c5d37e80c88\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 23:38:31.894342 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:31.894232 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/69f21f45-2600-4cef-9230-5c5d37e80c88-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"69f21f45-2600-4cef-9230-5c5d37e80c88\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 23:38:31.894872 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:31.894842 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/69f21f45-2600-4cef-9230-5c5d37e80c88-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"69f21f45-2600-4cef-9230-5c5d37e80c88\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 23:38:31.895147 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:31.895122 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/69f21f45-2600-4cef-9230-5c5d37e80c88-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"69f21f45-2600-4cef-9230-5c5d37e80c88\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 23:38:31.895829 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:31.895519 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/69f21f45-2600-4cef-9230-5c5d37e80c88-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"69f21f45-2600-4cef-9230-5c5d37e80c88\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 23:38:31.895829 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:31.895788 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/69f21f45-2600-4cef-9230-5c5d37e80c88-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"69f21f45-2600-4cef-9230-5c5d37e80c88\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 23:38:31.896230 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:31.896206 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/69f21f45-2600-4cef-9230-5c5d37e80c88-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"69f21f45-2600-4cef-9230-5c5d37e80c88\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 23:38:31.896400 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:31.896378 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/69f21f45-2600-4cef-9230-5c5d37e80c88-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"69f21f45-2600-4cef-9230-5c5d37e80c88\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 23:38:31.896935 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:31.896645 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/69f21f45-2600-4cef-9230-5c5d37e80c88-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"69f21f45-2600-4cef-9230-5c5d37e80c88\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 23:38:31.896935 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:31.896893 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/69f21f45-2600-4cef-9230-5c5d37e80c88-config\") pod \"prometheus-k8s-0\" (UID: \"69f21f45-2600-4cef-9230-5c5d37e80c88\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 23:38:31.897343 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:31.897319 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/69f21f45-2600-4cef-9230-5c5d37e80c88-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"69f21f45-2600-4cef-9230-5c5d37e80c88\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 23:38:31.897453 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:31.897368 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/69f21f45-2600-4cef-9230-5c5d37e80c88-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"69f21f45-2600-4cef-9230-5c5d37e80c88\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 23:38:31.897523 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:31.897503 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/69f21f45-2600-4cef-9230-5c5d37e80c88-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"69f21f45-2600-4cef-9230-5c5d37e80c88\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 23:38:31.897657 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:31.897639 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/69f21f45-2600-4cef-9230-5c5d37e80c88-config-out\") pod \"prometheus-k8s-0\" (UID: \"69f21f45-2600-4cef-9230-5c5d37e80c88\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 23:38:31.898633 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:31.898613 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/69f21f45-2600-4cef-9230-5c5d37e80c88-web-config\") pod \"prometheus-k8s-0\" (UID: \"69f21f45-2600-4cef-9230-5c5d37e80c88\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 23:38:31.898803 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:31.898787 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/69f21f45-2600-4cef-9230-5c5d37e80c88-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"69f21f45-2600-4cef-9230-5c5d37e80c88\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 23:38:31.899145 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:31.899118 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/69f21f45-2600-4cef-9230-5c5d37e80c88-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"69f21f45-2600-4cef-9230-5c5d37e80c88\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 23:38:31.899408 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:31.899392 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/69f21f45-2600-4cef-9230-5c5d37e80c88-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"69f21f45-2600-4cef-9230-5c5d37e80c88\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 23:38:31.900186 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:31.900170 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q287g\" (UniqueName: \"kubernetes.io/projected/69f21f45-2600-4cef-9230-5c5d37e80c88-kube-api-access-q287g\") pod \"prometheus-k8s-0\" (UID: \"69f21f45-2600-4cef-9230-5c5d37e80c88\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 23:38:32.067317 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:32.067292 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 15 23:38:32.235371 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:32.235334 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 15 23:38:32.237619 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:38:32.237589 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69f21f45_2600_4cef_9230_5c5d37e80c88.slice/crio-7f589bc472d145b75a3e860810316e4d3f00f5eb94bfb6fca162765cc85e5ddd WatchSource:0}: Error finding container 7f589bc472d145b75a3e860810316e4d3f00f5eb94bfb6fca162765cc85e5ddd: Status 404 returned error can't find the container with id 7f589bc472d145b75a3e860810316e4d3f00f5eb94bfb6fca162765cc85e5ddd Jul 15 23:38:32.678495 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:32.678413 2563 generic.go:356] "Generic (PLEG): container finished" podID="69f21f45-2600-4cef-9230-5c5d37e80c88" containerID="c146e6d09ce0e86ccbcb6db169b6910c606f3abbc3f8d9f2c20e89ee7541840e" exitCode=0 Jul 15 23:38:32.678495 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:32.678488 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"69f21f45-2600-4cef-9230-5c5d37e80c88","Type":"ContainerDied","Data":"c146e6d09ce0e86ccbcb6db169b6910c606f3abbc3f8d9f2c20e89ee7541840e"} Jul 15 23:38:32.678892 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:32.678542 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"69f21f45-2600-4cef-9230-5c5d37e80c88","Type":"ContainerStarted","Data":"7f589bc472d145b75a3e860810316e4d3f00f5eb94bfb6fca162765cc85e5ddd"} Jul 15 23:38:33.171307 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:33.170572 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a881fb94-9a0b-4c46-b5fc-d841398d576d" path="/var/lib/kubelet/pods/a881fb94-9a0b-4c46-b5fc-d841398d576d/volumes" Jul 15 23:38:33.689018 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:33.688951 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"69f21f45-2600-4cef-9230-5c5d37e80c88","Type":"ContainerStarted","Data":"95620f143dc46f6d4f3ff3aa0e0ba9583e89d6ed3400836719f347ceca152454"} Jul 15 23:38:33.689370 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:33.689092 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"69f21f45-2600-4cef-9230-5c5d37e80c88","Type":"ContainerStarted","Data":"14d67ebd3ac5e10d15bfbbbc4b5943a7385b6d4c2f9556ca27386ed8d5bcff47"} Jul 15 23:38:33.689370 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:33.689132 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"69f21f45-2600-4cef-9230-5c5d37e80c88","Type":"ContainerStarted","Data":"dad18390c974e8fadaeae2298e440ee11ab6674ccdf06fe34a38577ba7a5dc7d"} Jul 15 23:38:33.689370 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:33.689169 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"69f21f45-2600-4cef-9230-5c5d37e80c88","Type":"ContainerStarted","Data":"efeca07a05332729cc97b5f3fb52b55d488bf9be51a7573afc5cfaa15f554965"} Jul 15 23:38:33.689370 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:33.689205 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"69f21f45-2600-4cef-9230-5c5d37e80c88","Type":"ContainerStarted","Data":"892719fda6e640043afd7183ab517f92011335cf06698a11b4ce285fe32b8047"} Jul 15 23:38:33.689370 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:33.689247 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"69f21f45-2600-4cef-9230-5c5d37e80c88","Type":"ContainerStarted","Data":"225a21a7f3ba1e1439370e1be53a22730584c735f69bb7d861a6586b6ac1d020"} Jul 15 23:38:33.712814 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:33.712770 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.689838474 podStartE2EDuration="2.712756321s" totalImagesPullingTime="0s" totalInitContainerRuntime="22.917847ms" isStatefulPod=false podCreationTimestamp="2026-07-15 23:38:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 23:38:33.711687671 +0000 UTC m=+129.085145657" watchObservedRunningTime="2026-07-15 23:38:33.712756321 +0000 UTC m=+129.086214283" Jul 15 23:38:37.068154 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:37.068120 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 15 23:38:39.819801 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:39.819770 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7d6d44479b-g6lw7"] Jul 15 23:38:39.922001 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:39.921950 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:38:40.024702 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:40.024679 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:38:40.125079 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:40.125007 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:38:40.225295 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:38:40.225269 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:39:01.385293 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:39:01.385251 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:39:01.485305 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:39:01.485271 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:39:04.840065 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:39:04.839943 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-7d6d44479b-g6lw7" podUID="4a3b0780-bf04-4522-92df-8b09e2f29bcb" containerName="console" containerID="cri-o://644c17f4386873fb31adbc1e5c7543804a12939ae20f4db42d46c2c843d4815a" gracePeriod=15 Jul 15 23:39:05.095477 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:39:05.095422 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7d6d44479b-g6lw7" Jul 15 23:39:05.188834 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:39:05.188809 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4a3b0780-bf04-4522-92df-8b09e2f29bcb-console-serving-cert\") pod \"4a3b0780-bf04-4522-92df-8b09e2f29bcb\" (UID: \"4a3b0780-bf04-4522-92df-8b09e2f29bcb\") " Jul 15 23:39:05.188987 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:39:05.188851 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4a3b0780-bf04-4522-92df-8b09e2f29bcb-console-oauth-config\") pod \"4a3b0780-bf04-4522-92df-8b09e2f29bcb\" (UID: \"4a3b0780-bf04-4522-92df-8b09e2f29bcb\") " Jul 15 23:39:05.188987 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:39:05.188885 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4a3b0780-bf04-4522-92df-8b09e2f29bcb-oauth-serving-cert\") pod \"4a3b0780-bf04-4522-92df-8b09e2f29bcb\" (UID: \"4a3b0780-bf04-4522-92df-8b09e2f29bcb\") " Jul 15 23:39:05.188987 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:39:05.188928 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4a3b0780-bf04-4522-92df-8b09e2f29bcb-trusted-ca-bundle\") pod \"4a3b0780-bf04-4522-92df-8b09e2f29bcb\" (UID: \"4a3b0780-bf04-4522-92df-8b09e2f29bcb\") " Jul 15 23:39:05.188987 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:39:05.188945 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c5sc9\" (UniqueName: \"kubernetes.io/projected/4a3b0780-bf04-4522-92df-8b09e2f29bcb-kube-api-access-c5sc9\") pod \"4a3b0780-bf04-4522-92df-8b09e2f29bcb\" (UID: \"4a3b0780-bf04-4522-92df-8b09e2f29bcb\") " Jul 15 23:39:05.189198 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:39:05.189004 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4a3b0780-bf04-4522-92df-8b09e2f29bcb-service-ca\") pod \"4a3b0780-bf04-4522-92df-8b09e2f29bcb\" (UID: \"4a3b0780-bf04-4522-92df-8b09e2f29bcb\") " Jul 15 23:39:05.189198 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:39:05.189060 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4a3b0780-bf04-4522-92df-8b09e2f29bcb-console-config\") pod \"4a3b0780-bf04-4522-92df-8b09e2f29bcb\" (UID: \"4a3b0780-bf04-4522-92df-8b09e2f29bcb\") " Jul 15 23:39:05.189396 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:39:05.189349 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a3b0780-bf04-4522-92df-8b09e2f29bcb-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "4a3b0780-bf04-4522-92df-8b09e2f29bcb" (UID: "4a3b0780-bf04-4522-92df-8b09e2f29bcb"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 23:39:05.189587 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:39:05.189389 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a3b0780-bf04-4522-92df-8b09e2f29bcb-service-ca" (OuterVolumeSpecName: "service-ca") pod "4a3b0780-bf04-4522-92df-8b09e2f29bcb" (UID: "4a3b0780-bf04-4522-92df-8b09e2f29bcb"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 23:39:05.189587 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:39:05.189425 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a3b0780-bf04-4522-92df-8b09e2f29bcb-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "4a3b0780-bf04-4522-92df-8b09e2f29bcb" (UID: "4a3b0780-bf04-4522-92df-8b09e2f29bcb"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 23:39:05.189587 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:39:05.189565 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a3b0780-bf04-4522-92df-8b09e2f29bcb-console-config" (OuterVolumeSpecName: "console-config") pod "4a3b0780-bf04-4522-92df-8b09e2f29bcb" (UID: "4a3b0780-bf04-4522-92df-8b09e2f29bcb"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 23:39:05.191096 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:39:05.191061 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a3b0780-bf04-4522-92df-8b09e2f29bcb-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "4a3b0780-bf04-4522-92df-8b09e2f29bcb" (UID: "4a3b0780-bf04-4522-92df-8b09e2f29bcb"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 23:39:05.191243 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:39:05.191221 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a3b0780-bf04-4522-92df-8b09e2f29bcb-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "4a3b0780-bf04-4522-92df-8b09e2f29bcb" (UID: "4a3b0780-bf04-4522-92df-8b09e2f29bcb"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 23:39:05.191285 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:39:05.191250 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a3b0780-bf04-4522-92df-8b09e2f29bcb-kube-api-access-c5sc9" (OuterVolumeSpecName: "kube-api-access-c5sc9") pod "4a3b0780-bf04-4522-92df-8b09e2f29bcb" (UID: "4a3b0780-bf04-4522-92df-8b09e2f29bcb"). InnerVolumeSpecName "kube-api-access-c5sc9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 23:39:05.290355 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:39:05.290325 2563 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4a3b0780-bf04-4522-92df-8b09e2f29bcb-console-config\") on node \"ip-10-0-128-197.ec2.internal\" DevicePath \"\"" Jul 15 23:39:05.290355 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:39:05.290349 2563 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4a3b0780-bf04-4522-92df-8b09e2f29bcb-console-serving-cert\") on node \"ip-10-0-128-197.ec2.internal\" DevicePath \"\"" Jul 15 23:39:05.290355 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:39:05.290359 2563 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4a3b0780-bf04-4522-92df-8b09e2f29bcb-console-oauth-config\") on node \"ip-10-0-128-197.ec2.internal\" DevicePath \"\"" Jul 15 23:39:05.290583 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:39:05.290368 2563 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4a3b0780-bf04-4522-92df-8b09e2f29bcb-oauth-serving-cert\") on node \"ip-10-0-128-197.ec2.internal\" DevicePath \"\"" Jul 15 23:39:05.290583 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:39:05.290377 2563 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4a3b0780-bf04-4522-92df-8b09e2f29bcb-trusted-ca-bundle\") on node \"ip-10-0-128-197.ec2.internal\" DevicePath \"\"" Jul 15 23:39:05.290583 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:39:05.290386 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c5sc9\" (UniqueName: \"kubernetes.io/projected/4a3b0780-bf04-4522-92df-8b09e2f29bcb-kube-api-access-c5sc9\") on node \"ip-10-0-128-197.ec2.internal\" DevicePath \"\"" Jul 15 23:39:05.290583 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:39:05.290395 2563 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4a3b0780-bf04-4522-92df-8b09e2f29bcb-service-ca\") on node \"ip-10-0-128-197.ec2.internal\" DevicePath \"\"" Jul 15 23:39:05.823769 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:39:05.823734 2563 generic.go:356] "Generic (PLEG): container finished" podID="4a3b0780-bf04-4522-92df-8b09e2f29bcb" containerID="644c17f4386873fb31adbc1e5c7543804a12939ae20f4db42d46c2c843d4815a" exitCode=0 Jul 15 23:39:05.823958 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:39:05.823781 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7d6d44479b-g6lw7" Jul 15 23:39:05.823958 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:39:05.823827 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7d6d44479b-g6lw7" event={"ID":"4a3b0780-bf04-4522-92df-8b09e2f29bcb","Type":"ContainerDied","Data":"644c17f4386873fb31adbc1e5c7543804a12939ae20f4db42d46c2c843d4815a"} Jul 15 23:39:05.823958 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:39:05.823882 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7d6d44479b-g6lw7" event={"ID":"4a3b0780-bf04-4522-92df-8b09e2f29bcb","Type":"ContainerDied","Data":"832bf46e1a027bc5db0a3627e5ac0e2812614c5deee619fbc72937988da8703b"} Jul 15 23:39:05.823958 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:39:05.823903 2563 scope.go:117] "RemoveContainer" containerID="644c17f4386873fb31adbc1e5c7543804a12939ae20f4db42d46c2c843d4815a" Jul 15 23:39:05.833105 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:39:05.833078 2563 scope.go:117] "RemoveContainer" containerID="644c17f4386873fb31adbc1e5c7543804a12939ae20f4db42d46c2c843d4815a" Jul 15 23:39:05.833351 ip-10-0-128-197 kubenswrapper[2563]: E0715 23:39:05.833335 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"644c17f4386873fb31adbc1e5c7543804a12939ae20f4db42d46c2c843d4815a\": container with ID starting with 644c17f4386873fb31adbc1e5c7543804a12939ae20f4db42d46c2c843d4815a not found: ID does not exist" containerID="644c17f4386873fb31adbc1e5c7543804a12939ae20f4db42d46c2c843d4815a" Jul 15 23:39:05.833389 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:39:05.833359 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"644c17f4386873fb31adbc1e5c7543804a12939ae20f4db42d46c2c843d4815a"} err="failed to get container status \"644c17f4386873fb31adbc1e5c7543804a12939ae20f4db42d46c2c843d4815a\": rpc error: code = NotFound desc = could not find container \"644c17f4386873fb31adbc1e5c7543804a12939ae20f4db42d46c2c843d4815a\": container with ID starting with 644c17f4386873fb31adbc1e5c7543804a12939ae20f4db42d46c2c843d4815a not found: ID does not exist" Jul 15 23:39:05.843661 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:39:05.843636 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7d6d44479b-g6lw7"] Jul 15 23:39:05.847323 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:39:05.847305 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7d6d44479b-g6lw7"] Jul 15 23:39:07.168716 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:39:07.168686 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4a3b0780-bf04-4522-92df-8b09e2f29bcb" path="/var/lib/kubelet/pods/4a3b0780-bf04-4522-92df-8b09e2f29bcb/volumes" Jul 15 23:39:16.392735 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:39:16.392702 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:39:22.951466 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:39:22.951430 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:39:23.712612 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:39:23.712581 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:39:26.545477 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:39:26.545439 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:39:26.645110 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:39:26.645081 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:39:28.412044 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:39:28.412015 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:39:28.529756 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:39:28.529730 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:39:28.787680 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:39:28.787647 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:39:28.887807 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:39:28.887780 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:39:30.970622 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:39:30.970588 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:39:31.071005 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:39:31.070955 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:39:31.177924 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:39:31.177887 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:39:31.578225 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:39:31.578198 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-8tpqg"] Jul 15 23:39:31.578565 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:39:31.578553 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a3b0780-bf04-4522-92df-8b09e2f29bcb" containerName="console" Jul 15 23:39:31.578610 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:39:31.578567 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a3b0780-bf04-4522-92df-8b09e2f29bcb" containerName="console" Jul 15 23:39:31.578643 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:39:31.578626 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="4a3b0780-bf04-4522-92df-8b09e2f29bcb" containerName="console" Jul 15 23:39:31.582881 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:39:31.582864 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-8tpqg" Jul 15 23:39:31.585336 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:39:31.585312 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 15 23:39:31.585450 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:39:31.585354 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 15 23:39:31.586214 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:39:31.586200 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-hrrfc\"" Jul 15 23:39:31.590623 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:39:31.590599 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-8tpqg"] Jul 15 23:39:31.595031 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:39:31.595009 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:39:31.724208 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:39:31.724176 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b00a1e9e-efb8-42e4-8669-91a16cf3913e-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-8tpqg\" (UID: \"b00a1e9e-efb8-42e4-8669-91a16cf3913e\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-8tpqg" Jul 15 23:39:31.724358 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:39:31.724236 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wp9lq\" (UniqueName: \"kubernetes.io/projected/b00a1e9e-efb8-42e4-8669-91a16cf3913e-kube-api-access-wp9lq\") pod \"cert-manager-cainjector-559cdd8c87-8tpqg\" (UID: \"b00a1e9e-efb8-42e4-8669-91a16cf3913e\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-8tpqg" Jul 15 23:39:31.825317 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:39:31.825289 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wp9lq\" (UniqueName: \"kubernetes.io/projected/b00a1e9e-efb8-42e4-8669-91a16cf3913e-kube-api-access-wp9lq\") pod \"cert-manager-cainjector-559cdd8c87-8tpqg\" (UID: \"b00a1e9e-efb8-42e4-8669-91a16cf3913e\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-8tpqg" Jul 15 23:39:31.825453 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:39:31.825351 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b00a1e9e-efb8-42e4-8669-91a16cf3913e-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-8tpqg\" (UID: \"b00a1e9e-efb8-42e4-8669-91a16cf3913e\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-8tpqg" Jul 15 23:39:31.833045 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:39:31.832960 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b00a1e9e-efb8-42e4-8669-91a16cf3913e-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-8tpqg\" (UID: \"b00a1e9e-efb8-42e4-8669-91a16cf3913e\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-8tpqg" Jul 15 23:39:31.833045 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:39:31.832985 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wp9lq\" (UniqueName: \"kubernetes.io/projected/b00a1e9e-efb8-42e4-8669-91a16cf3913e-kube-api-access-wp9lq\") pod \"cert-manager-cainjector-559cdd8c87-8tpqg\" (UID: \"b00a1e9e-efb8-42e4-8669-91a16cf3913e\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-8tpqg" Jul 15 23:39:31.912763 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:39:31.912741 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-8tpqg" Jul 15 23:39:32.031189 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:39:32.031166 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-8tpqg"] Jul 15 23:39:32.033398 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:39:32.033367 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb00a1e9e_efb8_42e4_8669_91a16cf3913e.slice/crio-49debd2346aae2a3d58d635fc391ed95fad025f626430eb4a739e08aca8d3214 WatchSource:0}: Error finding container 49debd2346aae2a3d58d635fc391ed95fad025f626430eb4a739e08aca8d3214: Status 404 returned error can't find the container with id 49debd2346aae2a3d58d635fc391ed95fad025f626430eb4a739e08aca8d3214 Jul 15 23:39:32.068472 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:39:32.068451 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 15 23:39:32.083337 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:39:32.083280 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 15 23:39:32.935284 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:39:32.935221 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-8tpqg" event={"ID":"b00a1e9e-efb8-42e4-8669-91a16cf3913e","Type":"ContainerStarted","Data":"49debd2346aae2a3d58d635fc391ed95fad025f626430eb4a739e08aca8d3214"} Jul 15 23:39:32.955348 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:39:32.955319 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 15 23:39:33.349457 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:39:33.349417 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:39:33.450518 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:39:33.450481 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:39:35.950891 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:39:35.950857 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-8tpqg" event={"ID":"b00a1e9e-efb8-42e4-8669-91a16cf3913e","Type":"ContainerStarted","Data":"68b8f1c5666b9aecd1c5de3dbda5c8c2ebeb8409ed9de2cc59f72832c10a138e"} Jul 15 23:39:35.964805 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:39:35.964755 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-8tpqg" podStartSLOduration=1.888687457 podStartE2EDuration="4.964739726s" totalImagesPullingTime="3.076052269s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 23:39:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 23:39:35.96315074 +0000 UTC m=+191.336608704" watchObservedRunningTime="2026-07-15 23:39:35.964739726 +0000 UTC m=+191.338197690" Jul 15 23:39:38.390450 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:39:38.390390 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:39:39.764755 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:39:39.764709 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:39:39.861306 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:39:39.861276 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:39:41.773274 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:39:41.773238 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-vf6zb"] Jul 15 23:39:41.776768 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:39:41.776746 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-vf6zb" Jul 15 23:39:41.778913 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:39:41.778885 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 15 23:39:41.779024 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:39:41.778959 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-kpc7t\"" Jul 15 23:39:41.779713 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:39:41.779697 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 15 23:39:41.786135 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:39:41.786116 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-vf6zb"] Jul 15 23:39:41.917584 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:39:41.917541 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-94sv8\" (UniqueName: \"kubernetes.io/projected/8e445fcf-dd96-4267-a442-8d52334e53ec-kube-api-access-94sv8\") pod \"openshift-lws-operator-fd8ccff4c-vf6zb\" (UID: \"8e445fcf-dd96-4267-a442-8d52334e53ec\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-vf6zb" Jul 15 23:39:41.917749 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:39:41.917653 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8e445fcf-dd96-4267-a442-8d52334e53ec-tmp\") pod \"openshift-lws-operator-fd8ccff4c-vf6zb\" (UID: \"8e445fcf-dd96-4267-a442-8d52334e53ec\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-vf6zb" Jul 15 23:39:42.018910 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:39:42.018862 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94sv8\" (UniqueName: \"kubernetes.io/projected/8e445fcf-dd96-4267-a442-8d52334e53ec-kube-api-access-94sv8\") pod \"openshift-lws-operator-fd8ccff4c-vf6zb\" (UID: \"8e445fcf-dd96-4267-a442-8d52334e53ec\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-vf6zb" Jul 15 23:39:42.019146 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:39:42.018944 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8e445fcf-dd96-4267-a442-8d52334e53ec-tmp\") pod \"openshift-lws-operator-fd8ccff4c-vf6zb\" (UID: \"8e445fcf-dd96-4267-a442-8d52334e53ec\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-vf6zb" Jul 15 23:39:42.019327 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:39:42.019307 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8e445fcf-dd96-4267-a442-8d52334e53ec-tmp\") pod \"openshift-lws-operator-fd8ccff4c-vf6zb\" (UID: \"8e445fcf-dd96-4267-a442-8d52334e53ec\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-vf6zb" Jul 15 23:39:42.026643 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:39:42.026581 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-94sv8\" (UniqueName: \"kubernetes.io/projected/8e445fcf-dd96-4267-a442-8d52334e53ec-kube-api-access-94sv8\") pod \"openshift-lws-operator-fd8ccff4c-vf6zb\" (UID: \"8e445fcf-dd96-4267-a442-8d52334e53ec\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-vf6zb" Jul 15 23:39:42.085915 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:39:42.085887 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-vf6zb" Jul 15 23:39:42.227706 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:39:42.227684 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-vf6zb"] Jul 15 23:39:42.229320 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:39:42.229291 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8e445fcf_dd96_4267_a442_8d52334e53ec.slice/crio-8210b5be1d0a7bd914aceae2d0d4ef191149dd9ac6cc049a31eef4a84c35536b WatchSource:0}: Error finding container 8210b5be1d0a7bd914aceae2d0d4ef191149dd9ac6cc049a31eef4a84c35536b: Status 404 returned error can't find the container with id 8210b5be1d0a7bd914aceae2d0d4ef191149dd9ac6cc049a31eef4a84c35536b Jul 15 23:39:42.982461 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:39:42.982420 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-vf6zb" event={"ID":"8e445fcf-dd96-4267-a442-8d52334e53ec","Type":"ContainerStarted","Data":"8210b5be1d0a7bd914aceae2d0d4ef191149dd9ac6cc049a31eef4a84c35536b"} Jul 15 23:39:44.993100 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:39:44.993049 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-vf6zb" event={"ID":"8e445fcf-dd96-4267-a442-8d52334e53ec","Type":"ContainerStarted","Data":"448d89288f35aedc5a02ed2ed98b3af3cf9617dbe2223e714550b9e37ffd0fe8"} Jul 15 23:39:45.008790 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:39:45.008743 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-vf6zb" podStartSLOduration=1.572505966 podStartE2EDuration="4.008726698s" totalImagesPullingTime="2.436220732s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 23:39:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 23:39:45.007302177 +0000 UTC m=+200.380760163" watchObservedRunningTime="2026-07-15 23:39:45.008726698 +0000 UTC m=+200.382184661" Jul 15 23:39:45.692042 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:39:45.692000 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:39:45.793537 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:39:45.793501 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:39:47.816943 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:39:47.816906 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:39:56.395086 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:39:56.395052 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:39:56.448230 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:39:56.448196 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:40:02.072471 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:40:02.072435 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:40:02.206036 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:40:02.206008 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:40:02.622242 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:40:02.622210 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:40:02.774504 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:40:02.774458 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:40:02.876636 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:40:02.876555 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:40:02.943511 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:40:02.943484 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5dcb68dc66-xjnhk"] Jul 15 23:40:02.994987 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:40:02.994940 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5dcb68dc66-xjnhk"] Jul 15 23:40:02.995156 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:40:02.995093 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5dcb68dc66-xjnhk" Jul 15 23:40:02.999899 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:40:02.999869 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 15 23:40:03.001063 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:40:03.001039 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 15 23:40:03.001185 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:40:03.001051 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-6mmdc\"" Jul 15 23:40:03.001185 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:40:03.001152 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 15 23:40:03.001299 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:40:03.001185 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 15 23:40:03.100901 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:40:03.100865 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/fb5125e5-eb7a-40ea-9a0a-27c4997beef5-manifests\") pod \"opendatahub-operator-controller-manager-5dcb68dc66-xjnhk\" (UID: \"fb5125e5-eb7a-40ea-9a0a-27c4997beef5\") " pod="opendatahub/opendatahub-operator-controller-manager-5dcb68dc66-xjnhk" Jul 15 23:40:03.100901 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:40:03.100901 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fb5125e5-eb7a-40ea-9a0a-27c4997beef5-webhook-cert\") pod \"opendatahub-operator-controller-manager-5dcb68dc66-xjnhk\" (UID: \"fb5125e5-eb7a-40ea-9a0a-27c4997beef5\") " pod="opendatahub/opendatahub-operator-controller-manager-5dcb68dc66-xjnhk" Jul 15 23:40:03.101376 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:40:03.101008 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/fb5125e5-eb7a-40ea-9a0a-27c4997beef5-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5dcb68dc66-xjnhk\" (UID: \"fb5125e5-eb7a-40ea-9a0a-27c4997beef5\") " pod="opendatahub/opendatahub-operator-controller-manager-5dcb68dc66-xjnhk" Jul 15 23:40:03.101376 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:40:03.101069 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6rvdm\" (UniqueName: \"kubernetes.io/projected/fb5125e5-eb7a-40ea-9a0a-27c4997beef5-kube-api-access-6rvdm\") pod \"opendatahub-operator-controller-manager-5dcb68dc66-xjnhk\" (UID: \"fb5125e5-eb7a-40ea-9a0a-27c4997beef5\") " pod="opendatahub/opendatahub-operator-controller-manager-5dcb68dc66-xjnhk" Jul 15 23:40:03.101376 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:40:03.101102 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fb5125e5-eb7a-40ea-9a0a-27c4997beef5-tmp\") pod \"opendatahub-operator-controller-manager-5dcb68dc66-xjnhk\" (UID: \"fb5125e5-eb7a-40ea-9a0a-27c4997beef5\") " pod="opendatahub/opendatahub-operator-controller-manager-5dcb68dc66-xjnhk" Jul 15 23:40:03.149633 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:40:03.149553 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:40:03.202190 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:40:03.202159 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rvdm\" (UniqueName: \"kubernetes.io/projected/fb5125e5-eb7a-40ea-9a0a-27c4997beef5-kube-api-access-6rvdm\") pod \"opendatahub-operator-controller-manager-5dcb68dc66-xjnhk\" (UID: \"fb5125e5-eb7a-40ea-9a0a-27c4997beef5\") " pod="opendatahub/opendatahub-operator-controller-manager-5dcb68dc66-xjnhk" Jul 15 23:40:03.202388 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:40:03.202214 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fb5125e5-eb7a-40ea-9a0a-27c4997beef5-tmp\") pod \"opendatahub-operator-controller-manager-5dcb68dc66-xjnhk\" (UID: \"fb5125e5-eb7a-40ea-9a0a-27c4997beef5\") " pod="opendatahub/opendatahub-operator-controller-manager-5dcb68dc66-xjnhk" Jul 15 23:40:03.202388 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:40:03.202265 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/fb5125e5-eb7a-40ea-9a0a-27c4997beef5-manifests\") pod \"opendatahub-operator-controller-manager-5dcb68dc66-xjnhk\" (UID: \"fb5125e5-eb7a-40ea-9a0a-27c4997beef5\") " pod="opendatahub/opendatahub-operator-controller-manager-5dcb68dc66-xjnhk" Jul 15 23:40:03.202388 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:40:03.202298 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fb5125e5-eb7a-40ea-9a0a-27c4997beef5-webhook-cert\") pod \"opendatahub-operator-controller-manager-5dcb68dc66-xjnhk\" (UID: \"fb5125e5-eb7a-40ea-9a0a-27c4997beef5\") " pod="opendatahub/opendatahub-operator-controller-manager-5dcb68dc66-xjnhk" Jul 15 23:40:03.202388 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:40:03.202374 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/fb5125e5-eb7a-40ea-9a0a-27c4997beef5-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5dcb68dc66-xjnhk\" (UID: \"fb5125e5-eb7a-40ea-9a0a-27c4997beef5\") " pod="opendatahub/opendatahub-operator-controller-manager-5dcb68dc66-xjnhk" Jul 15 23:40:03.202747 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:40:03.202709 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/fb5125e5-eb7a-40ea-9a0a-27c4997beef5-manifests\") pod \"opendatahub-operator-controller-manager-5dcb68dc66-xjnhk\" (UID: \"fb5125e5-eb7a-40ea-9a0a-27c4997beef5\") " pod="opendatahub/opendatahub-operator-controller-manager-5dcb68dc66-xjnhk" Jul 15 23:40:03.204479 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:40:03.204453 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fb5125e5-eb7a-40ea-9a0a-27c4997beef5-tmp\") pod \"opendatahub-operator-controller-manager-5dcb68dc66-xjnhk\" (UID: \"fb5125e5-eb7a-40ea-9a0a-27c4997beef5\") " pod="opendatahub/opendatahub-operator-controller-manager-5dcb68dc66-xjnhk" Jul 15 23:40:03.204751 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:40:03.204734 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fb5125e5-eb7a-40ea-9a0a-27c4997beef5-webhook-cert\") pod \"opendatahub-operator-controller-manager-5dcb68dc66-xjnhk\" (UID: \"fb5125e5-eb7a-40ea-9a0a-27c4997beef5\") " pod="opendatahub/opendatahub-operator-controller-manager-5dcb68dc66-xjnhk" Jul 15 23:40:03.204816 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:40:03.204797 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/fb5125e5-eb7a-40ea-9a0a-27c4997beef5-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5dcb68dc66-xjnhk\" (UID: \"fb5125e5-eb7a-40ea-9a0a-27c4997beef5\") " pod="opendatahub/opendatahub-operator-controller-manager-5dcb68dc66-xjnhk" Jul 15 23:40:03.211327 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:40:03.211307 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rvdm\" (UniqueName: \"kubernetes.io/projected/fb5125e5-eb7a-40ea-9a0a-27c4997beef5-kube-api-access-6rvdm\") pod \"opendatahub-operator-controller-manager-5dcb68dc66-xjnhk\" (UID: \"fb5125e5-eb7a-40ea-9a0a-27c4997beef5\") " pod="opendatahub/opendatahub-operator-controller-manager-5dcb68dc66-xjnhk" Jul 15 23:40:03.249264 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:40:03.249228 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:40:03.307561 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:40:03.307528 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5dcb68dc66-xjnhk" Jul 15 23:40:03.456881 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:40:03.456857 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5dcb68dc66-xjnhk"] Jul 15 23:40:03.459171 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:40:03.459136 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb5125e5_eb7a_40ea_9a0a_27c4997beef5.slice/crio-d487152d7ac6036ccaadfc1357b99973356904cc25d3b221812800c708f1ec8a WatchSource:0}: Error finding container d487152d7ac6036ccaadfc1357b99973356904cc25d3b221812800c708f1ec8a: Status 404 returned error can't find the container with id d487152d7ac6036ccaadfc1357b99973356904cc25d3b221812800c708f1ec8a Jul 15 23:40:03.485327 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:40:03.485286 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:40:03.652914 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:40:03.652886 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:40:03.753186 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:40:03.753155 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:40:04.073624 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:40:04.073544 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5dcb68dc66-xjnhk" event={"ID":"fb5125e5-eb7a-40ea-9a0a-27c4997beef5","Type":"ContainerStarted","Data":"d487152d7ac6036ccaadfc1357b99973356904cc25d3b221812800c708f1ec8a"} Jul 15 23:40:07.091338 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:40:07.091300 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5dcb68dc66-xjnhk" event={"ID":"fb5125e5-eb7a-40ea-9a0a-27c4997beef5","Type":"ContainerStarted","Data":"f9e267972530ecbb54aee713f12d6b82eea6732c18ec694c251529b7fd5d1c35"} Jul 15 23:40:08.099635 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:40:08.099601 2563 generic.go:356] "Generic (PLEG): container finished" podID="fb5125e5-eb7a-40ea-9a0a-27c4997beef5" containerID="f9e267972530ecbb54aee713f12d6b82eea6732c18ec694c251529b7fd5d1c35" exitCode=0 Jul 15 23:40:08.100027 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:40:08.099642 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5dcb68dc66-xjnhk" event={"ID":"fb5125e5-eb7a-40ea-9a0a-27c4997beef5","Type":"ContainerDied","Data":"f9e267972530ecbb54aee713f12d6b82eea6732c18ec694c251529b7fd5d1c35"} Jul 15 23:40:09.106126 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:40:09.106086 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5dcb68dc66-xjnhk" event={"ID":"fb5125e5-eb7a-40ea-9a0a-27c4997beef5","Type":"ContainerStarted","Data":"7bee03942ac77b71d755568c623b41b6785b92747e9cdafcae82497a6a52c374"} Jul 15 23:40:09.106518 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:40:09.106198 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-5dcb68dc66-xjnhk" Jul 15 23:40:09.126194 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:40:09.126151 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-5dcb68dc66-xjnhk" podStartSLOduration=2.755643835 podStartE2EDuration="7.126135613s" totalImagesPullingTime="3.33617874s" totalInitContainerRuntime="1.034313038s" isStatefulPod=false podCreationTimestamp="2026-07-15 23:40:02 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 23:40:09.124832902 +0000 UTC m=+224.498290892" watchObservedRunningTime="2026-07-15 23:40:09.126135613 +0000 UTC m=+224.499593576" Jul 15 23:40:18.519278 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:40:18.519229 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:40:18.614839 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:40:18.614809 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:40:20.113289 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:40:20.113261 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-5dcb68dc66-xjnhk" Jul 15 23:40:20.450821 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:40:20.450731 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-787bd49fc4-pmzdt"] Jul 15 23:40:20.454247 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:40:20.454227 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-787bd49fc4-pmzdt" Jul 15 23:40:20.456600 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:40:20.456579 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 15 23:40:20.457092 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:40:20.456802 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-pt8zq\"" Jul 15 23:40:20.457092 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:40:20.456936 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 15 23:40:20.457092 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:40:20.456999 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 15 23:40:20.465955 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:40:20.465936 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-787bd49fc4-pmzdt"] Jul 15 23:40:20.555232 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:40:20.555206 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nlhn5\" (UniqueName: \"kubernetes.io/projected/2fc4c9e2-ffb6-4dae-9041-e2a5a5ac820b-kube-api-access-nlhn5\") pod \"lws-controller-manager-787bd49fc4-pmzdt\" (UID: \"2fc4c9e2-ffb6-4dae-9041-e2a5a5ac820b\") " pod="openshift-lws-operator/lws-controller-manager-787bd49fc4-pmzdt" Jul 15 23:40:20.555400 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:40:20.555245 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/2fc4c9e2-ffb6-4dae-9041-e2a5a5ac820b-metrics-cert\") pod \"lws-controller-manager-787bd49fc4-pmzdt\" (UID: \"2fc4c9e2-ffb6-4dae-9041-e2a5a5ac820b\") " pod="openshift-lws-operator/lws-controller-manager-787bd49fc4-pmzdt" Jul 15 23:40:20.555400 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:40:20.555275 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2fc4c9e2-ffb6-4dae-9041-e2a5a5ac820b-cert\") pod \"lws-controller-manager-787bd49fc4-pmzdt\" (UID: \"2fc4c9e2-ffb6-4dae-9041-e2a5a5ac820b\") " pod="openshift-lws-operator/lws-controller-manager-787bd49fc4-pmzdt" Jul 15 23:40:20.555400 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:40:20.555353 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/2fc4c9e2-ffb6-4dae-9041-e2a5a5ac820b-manager-config\") pod \"lws-controller-manager-787bd49fc4-pmzdt\" (UID: \"2fc4c9e2-ffb6-4dae-9041-e2a5a5ac820b\") " pod="openshift-lws-operator/lws-controller-manager-787bd49fc4-pmzdt" Jul 15 23:40:20.656003 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:40:20.655957 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nlhn5\" (UniqueName: \"kubernetes.io/projected/2fc4c9e2-ffb6-4dae-9041-e2a5a5ac820b-kube-api-access-nlhn5\") pod \"lws-controller-manager-787bd49fc4-pmzdt\" (UID: \"2fc4c9e2-ffb6-4dae-9041-e2a5a5ac820b\") " pod="openshift-lws-operator/lws-controller-manager-787bd49fc4-pmzdt" Jul 15 23:40:20.656149 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:40:20.656014 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/2fc4c9e2-ffb6-4dae-9041-e2a5a5ac820b-metrics-cert\") pod \"lws-controller-manager-787bd49fc4-pmzdt\" (UID: \"2fc4c9e2-ffb6-4dae-9041-e2a5a5ac820b\") " pod="openshift-lws-operator/lws-controller-manager-787bd49fc4-pmzdt" Jul 15 23:40:20.656149 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:40:20.656050 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2fc4c9e2-ffb6-4dae-9041-e2a5a5ac820b-cert\") pod \"lws-controller-manager-787bd49fc4-pmzdt\" (UID: \"2fc4c9e2-ffb6-4dae-9041-e2a5a5ac820b\") " pod="openshift-lws-operator/lws-controller-manager-787bd49fc4-pmzdt" Jul 15 23:40:20.656149 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:40:20.656067 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/2fc4c9e2-ffb6-4dae-9041-e2a5a5ac820b-manager-config\") pod \"lws-controller-manager-787bd49fc4-pmzdt\" (UID: \"2fc4c9e2-ffb6-4dae-9041-e2a5a5ac820b\") " pod="openshift-lws-operator/lws-controller-manager-787bd49fc4-pmzdt" Jul 15 23:40:20.656747 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:40:20.656725 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/2fc4c9e2-ffb6-4dae-9041-e2a5a5ac820b-manager-config\") pod \"lws-controller-manager-787bd49fc4-pmzdt\" (UID: \"2fc4c9e2-ffb6-4dae-9041-e2a5a5ac820b\") " pod="openshift-lws-operator/lws-controller-manager-787bd49fc4-pmzdt" Jul 15 23:40:20.658465 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:40:20.658447 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/2fc4c9e2-ffb6-4dae-9041-e2a5a5ac820b-metrics-cert\") pod \"lws-controller-manager-787bd49fc4-pmzdt\" (UID: \"2fc4c9e2-ffb6-4dae-9041-e2a5a5ac820b\") " pod="openshift-lws-operator/lws-controller-manager-787bd49fc4-pmzdt" Jul 15 23:40:20.658572 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:40:20.658511 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2fc4c9e2-ffb6-4dae-9041-e2a5a5ac820b-cert\") pod \"lws-controller-manager-787bd49fc4-pmzdt\" (UID: \"2fc4c9e2-ffb6-4dae-9041-e2a5a5ac820b\") " pod="openshift-lws-operator/lws-controller-manager-787bd49fc4-pmzdt" Jul 15 23:40:20.663659 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:40:20.663640 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nlhn5\" (UniqueName: \"kubernetes.io/projected/2fc4c9e2-ffb6-4dae-9041-e2a5a5ac820b-kube-api-access-nlhn5\") pod \"lws-controller-manager-787bd49fc4-pmzdt\" (UID: \"2fc4c9e2-ffb6-4dae-9041-e2a5a5ac820b\") " pod="openshift-lws-operator/lws-controller-manager-787bd49fc4-pmzdt" Jul 15 23:40:20.764003 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:40:20.763958 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-787bd49fc4-pmzdt" Jul 15 23:40:20.894582 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:40:20.894221 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-787bd49fc4-pmzdt"] Jul 15 23:40:21.159984 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:40:21.159877 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-787bd49fc4-pmzdt" event={"ID":"2fc4c9e2-ffb6-4dae-9041-e2a5a5ac820b","Type":"ContainerStarted","Data":"6f1931192f18509a8e55e8b9cbc34d185712932f812c69067b14a8d00d9b124c"} Jul 15 23:40:22.090593 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:40:22.090552 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:40:22.224585 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:40:22.224556 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:40:22.919050 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:40:22.919020 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:40:23.019370 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:40:23.019338 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:40:23.119590 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:40:23.119552 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:40:23.170749 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:40:23.170667 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-787bd49fc4-pmzdt" event={"ID":"2fc4c9e2-ffb6-4dae-9041-e2a5a5ac820b","Type":"ContainerStarted","Data":"53b44efe74fe3230315d1f55cc62dbe7ce9ee3d001f7980824bc53d93aef9aa2"} Jul 15 23:40:23.170749 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:40:23.170731 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-787bd49fc4-pmzdt" Jul 15 23:40:23.195222 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:40:23.195176 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-787bd49fc4-pmzdt" podStartSLOduration=1.951161411 podStartE2EDuration="3.195160773s" totalImagesPullingTime="1.243999362s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 23:40:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 23:40:23.192484478 +0000 UTC m=+238.565942567" watchObservedRunningTime="2026-07-15 23:40:23.195160773 +0000 UTC m=+238.568618735" Jul 15 23:40:23.526817 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:40:23.526785 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:40:23.793920 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:40:23.793836 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:40:28.039166 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:40:28.039118 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:40:28.103632 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:40:28.103601 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-7d6684b565-gd7qb"] Jul 15 23:40:28.108535 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:40:28.108509 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7d6684b565-gd7qb" Jul 15 23:40:28.110832 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:40:28.110814 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 15 23:40:28.110944 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:40:28.110829 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 15 23:40:28.111211 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:40:28.111195 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 15 23:40:28.111816 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:40:28.111800 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-w9kgv\"" Jul 15 23:40:28.111865 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:40:28.111834 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 15 23:40:28.121161 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:40:28.121139 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7d6684b565-gd7qb"] Jul 15 23:40:28.213982 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:40:28.213916 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:40:28.228151 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:40:28.228109 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qpwh7\" (UniqueName: \"kubernetes.io/projected/73313ae6-33c4-4994-b37e-31324804d8f2-kube-api-access-qpwh7\") pod \"kube-auth-proxy-7d6684b565-gd7qb\" (UID: \"73313ae6-33c4-4994-b37e-31324804d8f2\") " pod="openshift-ingress/kube-auth-proxy-7d6684b565-gd7qb" Jul 15 23:40:28.228226 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:40:28.228218 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/73313ae6-33c4-4994-b37e-31324804d8f2-tls-certs\") pod \"kube-auth-proxy-7d6684b565-gd7qb\" (UID: \"73313ae6-33c4-4994-b37e-31324804d8f2\") " pod="openshift-ingress/kube-auth-proxy-7d6684b565-gd7qb" Jul 15 23:40:28.228271 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:40:28.228236 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/73313ae6-33c4-4994-b37e-31324804d8f2-tmp\") pod \"kube-auth-proxy-7d6684b565-gd7qb\" (UID: \"73313ae6-33c4-4994-b37e-31324804d8f2\") " pod="openshift-ingress/kube-auth-proxy-7d6684b565-gd7qb" Jul 15 23:40:28.329285 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:40:28.329208 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qpwh7\" (UniqueName: \"kubernetes.io/projected/73313ae6-33c4-4994-b37e-31324804d8f2-kube-api-access-qpwh7\") pod \"kube-auth-proxy-7d6684b565-gd7qb\" (UID: \"73313ae6-33c4-4994-b37e-31324804d8f2\") " pod="openshift-ingress/kube-auth-proxy-7d6684b565-gd7qb" Jul 15 23:40:28.329429 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:40:28.329317 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/73313ae6-33c4-4994-b37e-31324804d8f2-tls-certs\") pod \"kube-auth-proxy-7d6684b565-gd7qb\" (UID: \"73313ae6-33c4-4994-b37e-31324804d8f2\") " pod="openshift-ingress/kube-auth-proxy-7d6684b565-gd7qb" Jul 15 23:40:28.329429 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:40:28.329333 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/73313ae6-33c4-4994-b37e-31324804d8f2-tmp\") pod \"kube-auth-proxy-7d6684b565-gd7qb\" (UID: \"73313ae6-33c4-4994-b37e-31324804d8f2\") " pod="openshift-ingress/kube-auth-proxy-7d6684b565-gd7qb" Jul 15 23:40:28.331637 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:40:28.331615 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/73313ae6-33c4-4994-b37e-31324804d8f2-tmp\") pod \"kube-auth-proxy-7d6684b565-gd7qb\" (UID: \"73313ae6-33c4-4994-b37e-31324804d8f2\") " pod="openshift-ingress/kube-auth-proxy-7d6684b565-gd7qb" Jul 15 23:40:28.331934 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:40:28.331911 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/73313ae6-33c4-4994-b37e-31324804d8f2-tls-certs\") pod \"kube-auth-proxy-7d6684b565-gd7qb\" (UID: \"73313ae6-33c4-4994-b37e-31324804d8f2\") " pod="openshift-ingress/kube-auth-proxy-7d6684b565-gd7qb" Jul 15 23:40:28.337049 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:40:28.337029 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qpwh7\" (UniqueName: \"kubernetes.io/projected/73313ae6-33c4-4994-b37e-31324804d8f2-kube-api-access-qpwh7\") pod \"kube-auth-proxy-7d6684b565-gd7qb\" (UID: \"73313ae6-33c4-4994-b37e-31324804d8f2\") " pod="openshift-ingress/kube-auth-proxy-7d6684b565-gd7qb" Jul 15 23:40:28.420303 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:40:28.420282 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7d6684b565-gd7qb" Jul 15 23:40:28.547209 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:40:28.547177 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7d6684b565-gd7qb"] Jul 15 23:40:28.549435 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:40:28.549409 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73313ae6_33c4_4994_b37e_31324804d8f2.slice/crio-189480abf98b3d88fb645b48c73c784d1719ac86b0662f0561ace66ceeb53d05 WatchSource:0}: Error finding container 189480abf98b3d88fb645b48c73c784d1719ac86b0662f0561ace66ceeb53d05: Status 404 returned error can't find the container with id 189480abf98b3d88fb645b48c73c784d1719ac86b0662f0561ace66ceeb53d05 Jul 15 23:40:29.198468 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:40:29.198432 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7d6684b565-gd7qb" event={"ID":"73313ae6-33c4-4994-b37e-31324804d8f2","Type":"ContainerStarted","Data":"189480abf98b3d88fb645b48c73c784d1719ac86b0662f0561ace66ceeb53d05"} Jul 15 23:40:29.442463 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:40:29.442421 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:40:29.553672 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:40:29.553641 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:40:30.880193 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:40:30.880158 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:40:31.209808 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:40:31.209731 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7d6684b565-gd7qb" event={"ID":"73313ae6-33c4-4994-b37e-31324804d8f2","Type":"ContainerStarted","Data":"458a1482d19ff411367b6747707eceb7da2edc623ffc534f64dcf7ce462aeab1"} Jul 15 23:40:31.230541 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:40:31.230491 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-7d6684b565-gd7qb" podStartSLOduration=0.743740352 podStartE2EDuration="3.230475481s" totalImagesPullingTime="2.486735129s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 23:40:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 23:40:31.228631117 +0000 UTC m=+246.602089075" watchObservedRunningTime="2026-07-15 23:40:31.230475481 +0000 UTC m=+246.603933443" Jul 15 23:40:33.802285 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:40:33.802250 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:40:33.943496 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:40:33.943465 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:40:34.116038 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:40:34.115938 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:40:34.177497 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:40:34.177469 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-787bd49fc4-pmzdt" Jul 15 23:40:34.270822 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:40:34.270794 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:40:35.368843 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:40:35.368806 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4ldjtc"] Jul 15 23:40:35.375619 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:40:35.375589 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4ldjtc" Jul 15 23:40:35.378777 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:40:35.378753 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 15 23:40:35.378887 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:40:35.378753 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-45m47\"" Jul 15 23:40:35.400729 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:40:35.400707 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4ldjtc"] Jul 15 23:40:35.491353 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:40:35.491315 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:40:35.499376 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:40:35.499344 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2e4216a-a127-4cfd-8510-c1352e97aa97-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4ldjtc\" (UID: \"a2e4216a-a127-4cfd-8510-c1352e97aa97\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4ldjtc" Jul 15 23:40:35.499513 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:40:35.499381 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t57vn\" (UniqueName: \"kubernetes.io/projected/a2e4216a-a127-4cfd-8510-c1352e97aa97-kube-api-access-t57vn\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4ldjtc\" (UID: \"a2e4216a-a127-4cfd-8510-c1352e97aa97\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4ldjtc" Jul 15 23:40:35.499513 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:40:35.499427 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2e4216a-a127-4cfd-8510-c1352e97aa97-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4ldjtc\" (UID: \"a2e4216a-a127-4cfd-8510-c1352e97aa97\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4ldjtc" Jul 15 23:40:35.499513 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:40:35.499449 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2e4216a-a127-4cfd-8510-c1352e97aa97-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4ldjtc\" (UID: \"a2e4216a-a127-4cfd-8510-c1352e97aa97\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4ldjtc" Jul 15 23:40:35.499513 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:40:35.499486 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2e4216a-a127-4cfd-8510-c1352e97aa97-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4ldjtc\" (UID: \"a2e4216a-a127-4cfd-8510-c1352e97aa97\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4ldjtc" Jul 15 23:40:35.499645 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:40:35.499553 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2e4216a-a127-4cfd-8510-c1352e97aa97-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4ldjtc\" (UID: \"a2e4216a-a127-4cfd-8510-c1352e97aa97\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4ldjtc" Jul 15 23:40:35.499645 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:40:35.499621 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2e4216a-a127-4cfd-8510-c1352e97aa97-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4ldjtc\" (UID: \"a2e4216a-a127-4cfd-8510-c1352e97aa97\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4ldjtc" Jul 15 23:40:35.499710 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:40:35.499646 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2e4216a-a127-4cfd-8510-c1352e97aa97-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4ldjtc\" (UID: \"a2e4216a-a127-4cfd-8510-c1352e97aa97\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4ldjtc" Jul 15 23:40:35.499710 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:40:35.499675 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2e4216a-a127-4cfd-8510-c1352e97aa97-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4ldjtc\" (UID: \"a2e4216a-a127-4cfd-8510-c1352e97aa97\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4ldjtc" Jul 15 23:40:35.600260 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:40:35.600224 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2e4216a-a127-4cfd-8510-c1352e97aa97-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4ldjtc\" (UID: \"a2e4216a-a127-4cfd-8510-c1352e97aa97\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4ldjtc" Jul 15 23:40:35.600440 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:40:35.600273 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2e4216a-a127-4cfd-8510-c1352e97aa97-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4ldjtc\" (UID: \"a2e4216a-a127-4cfd-8510-c1352e97aa97\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4ldjtc" Jul 15 23:40:35.600440 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:40:35.600297 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2e4216a-a127-4cfd-8510-c1352e97aa97-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4ldjtc\" (UID: \"a2e4216a-a127-4cfd-8510-c1352e97aa97\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4ldjtc" Jul 15 23:40:35.600440 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:40:35.600319 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2e4216a-a127-4cfd-8510-c1352e97aa97-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4ldjtc\" (UID: \"a2e4216a-a127-4cfd-8510-c1352e97aa97\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4ldjtc" Jul 15 23:40:35.600440 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:40:35.600345 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2e4216a-a127-4cfd-8510-c1352e97aa97-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4ldjtc\" (UID: \"a2e4216a-a127-4cfd-8510-c1352e97aa97\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4ldjtc" Jul 15 23:40:35.600440 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:40:35.600375 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2e4216a-a127-4cfd-8510-c1352e97aa97-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4ldjtc\" (UID: \"a2e4216a-a127-4cfd-8510-c1352e97aa97\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4ldjtc" Jul 15 23:40:35.600440 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:40:35.600420 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2e4216a-a127-4cfd-8510-c1352e97aa97-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4ldjtc\" (UID: \"a2e4216a-a127-4cfd-8510-c1352e97aa97\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4ldjtc" Jul 15 23:40:35.600796 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:40:35.600452 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t57vn\" (UniqueName: \"kubernetes.io/projected/a2e4216a-a127-4cfd-8510-c1352e97aa97-kube-api-access-t57vn\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4ldjtc\" (UID: \"a2e4216a-a127-4cfd-8510-c1352e97aa97\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4ldjtc" Jul 15 23:40:35.600796 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:40:35.600533 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2e4216a-a127-4cfd-8510-c1352e97aa97-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4ldjtc\" (UID: \"a2e4216a-a127-4cfd-8510-c1352e97aa97\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4ldjtc" Jul 15 23:40:35.600796 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:40:35.600727 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2e4216a-a127-4cfd-8510-c1352e97aa97-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4ldjtc\" (UID: \"a2e4216a-a127-4cfd-8510-c1352e97aa97\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4ldjtc" Jul 15 23:40:35.601087 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:40:35.601067 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2e4216a-a127-4cfd-8510-c1352e97aa97-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4ldjtc\" (UID: \"a2e4216a-a127-4cfd-8510-c1352e97aa97\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4ldjtc" Jul 15 23:40:35.601192 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:40:35.601122 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2e4216a-a127-4cfd-8510-c1352e97aa97-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4ldjtc\" (UID: \"a2e4216a-a127-4cfd-8510-c1352e97aa97\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4ldjtc" Jul 15 23:40:35.601192 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:40:35.601168 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2e4216a-a127-4cfd-8510-c1352e97aa97-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4ldjtc\" (UID: \"a2e4216a-a127-4cfd-8510-c1352e97aa97\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4ldjtc" Jul 15 23:40:35.601296 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:40:35.601203 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2e4216a-a127-4cfd-8510-c1352e97aa97-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4ldjtc\" (UID: \"a2e4216a-a127-4cfd-8510-c1352e97aa97\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4ldjtc" Jul 15 23:40:35.602801 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:40:35.602781 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2e4216a-a127-4cfd-8510-c1352e97aa97-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4ldjtc\" (UID: \"a2e4216a-a127-4cfd-8510-c1352e97aa97\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4ldjtc" Jul 15 23:40:35.602926 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:40:35.602910 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2e4216a-a127-4cfd-8510-c1352e97aa97-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4ldjtc\" (UID: \"a2e4216a-a127-4cfd-8510-c1352e97aa97\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4ldjtc" Jul 15 23:40:35.617755 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:40:35.617734 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2e4216a-a127-4cfd-8510-c1352e97aa97-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4ldjtc\" (UID: \"a2e4216a-a127-4cfd-8510-c1352e97aa97\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4ldjtc" Jul 15 23:40:35.617874 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:40:35.617752 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t57vn\" (UniqueName: \"kubernetes.io/projected/a2e4216a-a127-4cfd-8510-c1352e97aa97-kube-api-access-t57vn\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4ldjtc\" (UID: \"a2e4216a-a127-4cfd-8510-c1352e97aa97\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4ldjtc" Jul 15 23:40:35.652049 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:40:35.651950 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:40:35.685178 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:40:35.685150 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4ldjtc" Jul 15 23:40:35.790280 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:40:35.790242 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:40:35.861061 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:40:35.861034 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4ldjtc"] Jul 15 23:40:35.864072 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:40:35.864050 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2e4216a_a127_4cfd_8510_c1352e97aa97.slice/crio-9b9d8cf8095a9b2ccc3e7cfaa3389688384f7b3ee6e61b3d7ea09781bae633f1 WatchSource:0}: Error finding container 9b9d8cf8095a9b2ccc3e7cfaa3389688384f7b3ee6e61b3d7ea09781bae633f1: Status 404 returned error can't find the container with id 9b9d8cf8095a9b2ccc3e7cfaa3389688384f7b3ee6e61b3d7ea09781bae633f1 Jul 15 23:40:36.232658 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:40:36.232624 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4ldjtc" event={"ID":"a2e4216a-a127-4cfd-8510-c1352e97aa97","Type":"ContainerStarted","Data":"9b9d8cf8095a9b2ccc3e7cfaa3389688384f7b3ee6e61b3d7ea09781bae633f1"} Jul 15 23:40:36.421498 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:40:36.421458 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:40:36.610353 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:40:36.610178 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:40:36.739874 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:40:36.739835 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:40:38.288932 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:40:38.288891 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 15 23:40:38.289217 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:40:38.288999 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 15 23:40:38.289217 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:40:38.289045 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 15 23:40:39.247057 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:40:39.247025 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4ldjtc" event={"ID":"a2e4216a-a127-4cfd-8510-c1352e97aa97","Type":"ContainerStarted","Data":"9536b452c46c1f16a7b4b13083068a3b5f26842d90515f5771cfeec4be00a049"} Jul 15 23:40:39.685835 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:40:39.685741 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4ldjtc" Jul 15 23:40:39.690115 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:40:39.690091 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4ldjtc" Jul 15 23:40:39.726694 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:40:39.726634 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4ldjtc" podStartSLOduration=2.304243834 podStartE2EDuration="4.726616765s" totalImagesPullingTime="2.422372931s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 23:40:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 23:40:39.287292529 +0000 UTC m=+254.660750492" watchObservedRunningTime="2026-07-15 23:40:39.726616765 +0000 UTC m=+255.100074724" Jul 15 23:40:40.252203 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:40:40.252174 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4ldjtc" Jul 15 23:40:40.253026 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:40:40.253005 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4ldjtc" Jul 15 23:40:46.006043 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:40:46.006006 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:40:47.138547 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:40:47.138452 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:41:23.537625 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:41:23.537591 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:41:23.633550 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:41:23.633517 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:41:23.713797 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:41:23.713756 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:41:24.452537 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:41:24.452503 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:41:24.660091 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:41:24.660063 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:41:24.977146 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:41:24.977121 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d8f6k_2f64cab3-104a-44f6-b125-ba458f54f058/ovn-acl-logging/0.log" Jul 15 23:41:24.981110 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:41:24.981087 2563 kubelet.go:1648] "Image garbage collection succeeded" Jul 15 23:41:24.981736 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:41:24.981716 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d8f6k_2f64cab3-104a-44f6-b125-ba458f54f058/ovn-acl-logging/0.log" Jul 15 23:41:40.825665 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:41:40.825632 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:41:40.891590 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:41:40.891552 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:41:42.784744 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:41:42.784715 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:41:42.880804 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:41:42.880778 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:41:52.813772 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:41:52.813736 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:41:57.020597 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:41:57.020562 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:41:59.392589 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:41:59.392560 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:41:59.666955 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:41:59.666867 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:42:00.004996 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:42:00.004954 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:42:03.407738 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:42:03.407697 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:42:04.444926 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:42:04.444893 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:42:04.580857 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:42:04.580818 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:42:06.803477 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:42:06.803431 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:42:08.296786 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:42:08.296753 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:42:09.620574 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:42:09.620537 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:42:09.721921 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:42:09.721891 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:42:11.431096 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:42:11.431050 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:42:13.413084 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:42:13.413053 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:42:13.514930 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:42:13.514876 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:42:15.903681 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:42:15.903633 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:42:16.176808 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:42:16.176713 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:42:17.616027 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:42:17.615962 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:42:17.715387 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:42:17.715348 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:42:18.018377 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:42:18.018343 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:42:18.120398 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:42:18.120368 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:42:21.776297 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:42:21.776263 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:42:21.877545 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:42:21.877490 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:42:28.344693 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:42:28.344661 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:42:28.443980 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:42:28.443943 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:42:31.159842 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:42:31.159808 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:42:31.265127 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:42:31.265095 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:42:35.968024 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:42:35.967980 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:42:36.069655 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:42:36.069609 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:42:39.493875 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:42:39.493830 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:42:39.596944 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:42:39.596914 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:42:45.599984 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:42:45.599941 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7c98766766-59kcb"] Jul 15 23:42:45.603295 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:42:45.603274 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7c98766766-59kcb" Jul 15 23:42:45.606142 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:42:45.606124 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 15 23:42:45.606244 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:42:45.606124 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 15 23:42:45.607005 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:42:45.606990 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 15 23:42:45.607102 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:42:45.607011 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 15 23:42:45.607151 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:42:45.607112 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-89vtl\"" Jul 15 23:42:45.607517 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:42:45.607499 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 15 23:42:45.611384 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:42:45.611366 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 15 23:42:45.616930 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:42:45.616911 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7c98766766-59kcb"] Jul 15 23:42:45.632779 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:42:45.632751 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ff49s\" (UniqueName: \"kubernetes.io/projected/32a80dce-f1af-4b10-ad64-3a783d915e63-kube-api-access-ff49s\") pod \"console-7c98766766-59kcb\" (UID: \"32a80dce-f1af-4b10-ad64-3a783d915e63\") " pod="openshift-console/console-7c98766766-59kcb" Jul 15 23:42:45.632870 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:42:45.632812 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/32a80dce-f1af-4b10-ad64-3a783d915e63-trusted-ca-bundle\") pod \"console-7c98766766-59kcb\" (UID: \"32a80dce-f1af-4b10-ad64-3a783d915e63\") " pod="openshift-console/console-7c98766766-59kcb" Jul 15 23:42:45.632922 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:42:45.632904 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/32a80dce-f1af-4b10-ad64-3a783d915e63-console-config\") pod \"console-7c98766766-59kcb\" (UID: \"32a80dce-f1af-4b10-ad64-3a783d915e63\") " pod="openshift-console/console-7c98766766-59kcb" Jul 15 23:42:45.632988 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:42:45.632951 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/32a80dce-f1af-4b10-ad64-3a783d915e63-console-oauth-config\") pod \"console-7c98766766-59kcb\" (UID: \"32a80dce-f1af-4b10-ad64-3a783d915e63\") " pod="openshift-console/console-7c98766766-59kcb" Jul 15 23:42:45.633084 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:42:45.633065 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/32a80dce-f1af-4b10-ad64-3a783d915e63-console-serving-cert\") pod \"console-7c98766766-59kcb\" (UID: \"32a80dce-f1af-4b10-ad64-3a783d915e63\") " pod="openshift-console/console-7c98766766-59kcb" Jul 15 23:42:45.633142 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:42:45.633107 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/32a80dce-f1af-4b10-ad64-3a783d915e63-service-ca\") pod \"console-7c98766766-59kcb\" (UID: \"32a80dce-f1af-4b10-ad64-3a783d915e63\") " pod="openshift-console/console-7c98766766-59kcb" Jul 15 23:42:45.633196 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:42:45.633153 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/32a80dce-f1af-4b10-ad64-3a783d915e63-oauth-serving-cert\") pod \"console-7c98766766-59kcb\" (UID: \"32a80dce-f1af-4b10-ad64-3a783d915e63\") " pod="openshift-console/console-7c98766766-59kcb" Jul 15 23:42:45.656339 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:42:45.656315 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:42:45.721465 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:42:45.721415 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:42:45.733908 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:42:45.733883 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/32a80dce-f1af-4b10-ad64-3a783d915e63-console-oauth-config\") pod \"console-7c98766766-59kcb\" (UID: \"32a80dce-f1af-4b10-ad64-3a783d915e63\") " pod="openshift-console/console-7c98766766-59kcb" Jul 15 23:42:45.734034 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:42:45.733940 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/32a80dce-f1af-4b10-ad64-3a783d915e63-console-serving-cert\") pod \"console-7c98766766-59kcb\" (UID: \"32a80dce-f1af-4b10-ad64-3a783d915e63\") " pod="openshift-console/console-7c98766766-59kcb" Jul 15 23:42:45.734034 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:42:45.734002 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/32a80dce-f1af-4b10-ad64-3a783d915e63-service-ca\") pod \"console-7c98766766-59kcb\" (UID: \"32a80dce-f1af-4b10-ad64-3a783d915e63\") " pod="openshift-console/console-7c98766766-59kcb" Jul 15 23:42:45.734160 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:42:45.734035 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/32a80dce-f1af-4b10-ad64-3a783d915e63-oauth-serving-cert\") pod \"console-7c98766766-59kcb\" (UID: \"32a80dce-f1af-4b10-ad64-3a783d915e63\") " pod="openshift-console/console-7c98766766-59kcb" Jul 15 23:42:45.734160 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:42:45.734085 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ff49s\" (UniqueName: \"kubernetes.io/projected/32a80dce-f1af-4b10-ad64-3a783d915e63-kube-api-access-ff49s\") pod \"console-7c98766766-59kcb\" (UID: \"32a80dce-f1af-4b10-ad64-3a783d915e63\") " pod="openshift-console/console-7c98766766-59kcb" Jul 15 23:42:45.734160 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:42:45.734140 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/32a80dce-f1af-4b10-ad64-3a783d915e63-trusted-ca-bundle\") pod \"console-7c98766766-59kcb\" (UID: \"32a80dce-f1af-4b10-ad64-3a783d915e63\") " pod="openshift-console/console-7c98766766-59kcb" Jul 15 23:42:45.734535 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:42:45.734180 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/32a80dce-f1af-4b10-ad64-3a783d915e63-console-config\") pod \"console-7c98766766-59kcb\" (UID: \"32a80dce-f1af-4b10-ad64-3a783d915e63\") " pod="openshift-console/console-7c98766766-59kcb" Jul 15 23:42:45.734813 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:42:45.734791 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/32a80dce-f1af-4b10-ad64-3a783d915e63-service-ca\") pod \"console-7c98766766-59kcb\" (UID: \"32a80dce-f1af-4b10-ad64-3a783d915e63\") " pod="openshift-console/console-7c98766766-59kcb" Jul 15 23:42:45.734897 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:42:45.734791 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/32a80dce-f1af-4b10-ad64-3a783d915e63-oauth-serving-cert\") pod \"console-7c98766766-59kcb\" (UID: \"32a80dce-f1af-4b10-ad64-3a783d915e63\") " pod="openshift-console/console-7c98766766-59kcb" Jul 15 23:42:45.734897 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:42:45.734846 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/32a80dce-f1af-4b10-ad64-3a783d915e63-console-config\") pod \"console-7c98766766-59kcb\" (UID: \"32a80dce-f1af-4b10-ad64-3a783d915e63\") " pod="openshift-console/console-7c98766766-59kcb" Jul 15 23:42:45.735342 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:42:45.735321 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/32a80dce-f1af-4b10-ad64-3a783d915e63-trusted-ca-bundle\") pod \"console-7c98766766-59kcb\" (UID: \"32a80dce-f1af-4b10-ad64-3a783d915e63\") " pod="openshift-console/console-7c98766766-59kcb" Jul 15 23:42:45.736361 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:42:45.736339 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/32a80dce-f1af-4b10-ad64-3a783d915e63-console-serving-cert\") pod \"console-7c98766766-59kcb\" (UID: \"32a80dce-f1af-4b10-ad64-3a783d915e63\") " pod="openshift-console/console-7c98766766-59kcb" Jul 15 23:42:45.736537 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:42:45.736518 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/32a80dce-f1af-4b10-ad64-3a783d915e63-console-oauth-config\") pod \"console-7c98766766-59kcb\" (UID: \"32a80dce-f1af-4b10-ad64-3a783d915e63\") " pod="openshift-console/console-7c98766766-59kcb" Jul 15 23:42:45.741916 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:42:45.741894 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ff49s\" (UniqueName: \"kubernetes.io/projected/32a80dce-f1af-4b10-ad64-3a783d915e63-kube-api-access-ff49s\") pod \"console-7c98766766-59kcb\" (UID: \"32a80dce-f1af-4b10-ad64-3a783d915e63\") " pod="openshift-console/console-7c98766766-59kcb" Jul 15 23:42:45.913594 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:42:45.913529 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7c98766766-59kcb" Jul 15 23:42:46.046129 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:42:46.046101 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7c98766766-59kcb"] Jul 15 23:42:46.048724 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:42:46.048695 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32a80dce_f1af_4b10_ad64_3a783d915e63.slice/crio-b068fb0a7fd2460dab6bd3395f54c3e650b73878233eaa9097d7733a182674f7 WatchSource:0}: Error finding container b068fb0a7fd2460dab6bd3395f54c3e650b73878233eaa9097d7733a182674f7: Status 404 returned error can't find the container with id b068fb0a7fd2460dab6bd3395f54c3e650b73878233eaa9097d7733a182674f7 Jul 15 23:42:46.131710 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:42:46.131683 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:42:46.232654 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:42:46.232620 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:42:46.827983 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:42:46.827939 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7c98766766-59kcb" event={"ID":"32a80dce-f1af-4b10-ad64-3a783d915e63","Type":"ContainerStarted","Data":"a702a75e38038bcf731805061c588bb41c262954381b651afaea6532b415dddc"} Jul 15 23:42:46.828367 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:42:46.827996 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7c98766766-59kcb" event={"ID":"32a80dce-f1af-4b10-ad64-3a783d915e63","Type":"ContainerStarted","Data":"b068fb0a7fd2460dab6bd3395f54c3e650b73878233eaa9097d7733a182674f7"} Jul 15 23:42:46.845627 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:42:46.845587 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7c98766766-59kcb" podStartSLOduration=1.8455736200000001 podStartE2EDuration="1.84557362s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 23:42:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 23:42:46.84317361 +0000 UTC m=+382.216631574" watchObservedRunningTime="2026-07-15 23:42:46.84557362 +0000 UTC m=+382.219031583" Jul 15 23:42:49.332231 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:42:49.332193 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:42:49.437985 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:42:49.437934 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:42:53.762223 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:42:53.762186 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:42:53.825324 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:42:53.825292 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:42:54.000153 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:42:54.000125 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:42:54.090289 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:42:54.090210 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:42:55.914552 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:42:55.914521 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7c98766766-59kcb" Jul 15 23:42:55.914926 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:42:55.914572 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7c98766766-59kcb" Jul 15 23:42:55.919310 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:42:55.919288 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7c98766766-59kcb" Jul 15 23:42:56.876835 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:42:56.876808 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7c98766766-59kcb" Jul 15 23:42:57.055625 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:42:57.055577 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:42:57.173508 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:42:57.173419 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:42:57.229325 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:42:57.229294 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:42:57.676322 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:42:57.676277 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:42:57.769887 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:42:57.769857 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:42:58.241902 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:42:58.241871 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-sdscx"] Jul 15 23:42:58.245339 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:42:58.245318 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-sdscx" Jul 15 23:42:58.247474 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:42:58.247457 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-xqpdm\"" Jul 15 23:42:58.255378 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:42:58.255355 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-sdscx"] Jul 15 23:42:58.269605 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:42:58.269575 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:42:58.343276 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:42:58.343251 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c655a6b-6d89-459f-b331-3f8b08d0eae0-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-sdscx\" (UID: \"5c655a6b-6d89-459f-b331-3f8b08d0eae0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-sdscx" Jul 15 23:42:58.343405 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:42:58.343279 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c655a6b-6d89-459f-b331-3f8b08d0eae0-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-sdscx\" (UID: \"5c655a6b-6d89-459f-b331-3f8b08d0eae0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-sdscx" Jul 15 23:42:58.343405 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:42:58.343317 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c655a6b-6d89-459f-b331-3f8b08d0eae0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-sdscx\" (UID: \"5c655a6b-6d89-459f-b331-3f8b08d0eae0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-sdscx" Jul 15 23:42:58.343518 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:42:58.343414 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c655a6b-6d89-459f-b331-3f8b08d0eae0-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-sdscx\" (UID: \"5c655a6b-6d89-459f-b331-3f8b08d0eae0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-sdscx" Jul 15 23:42:58.343518 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:42:58.343483 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c655a6b-6d89-459f-b331-3f8b08d0eae0-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-sdscx\" (UID: \"5c655a6b-6d89-459f-b331-3f8b08d0eae0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-sdscx" Jul 15 23:42:58.343627 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:42:58.343541 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c655a6b-6d89-459f-b331-3f8b08d0eae0-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-sdscx\" (UID: \"5c655a6b-6d89-459f-b331-3f8b08d0eae0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-sdscx" Jul 15 23:42:58.343627 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:42:58.343594 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c655a6b-6d89-459f-b331-3f8b08d0eae0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-sdscx\" (UID: \"5c655a6b-6d89-459f-b331-3f8b08d0eae0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-sdscx" Jul 15 23:42:58.343718 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:42:58.343623 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c655a6b-6d89-459f-b331-3f8b08d0eae0-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-sdscx\" (UID: \"5c655a6b-6d89-459f-b331-3f8b08d0eae0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-sdscx" Jul 15 23:42:58.343718 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:42:58.343648 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vc98r\" (UniqueName: \"kubernetes.io/projected/5c655a6b-6d89-459f-b331-3f8b08d0eae0-kube-api-access-vc98r\") pod \"maas-default-gateway-openshift-default-56654dcf7-sdscx\" (UID: \"5c655a6b-6d89-459f-b331-3f8b08d0eae0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-sdscx" Jul 15 23:42:58.389004 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:42:58.388977 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:42:58.445012 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:42:58.444958 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c655a6b-6d89-459f-b331-3f8b08d0eae0-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-sdscx\" (UID: \"5c655a6b-6d89-459f-b331-3f8b08d0eae0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-sdscx" Jul 15 23:42:58.445142 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:42:58.445044 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c655a6b-6d89-459f-b331-3f8b08d0eae0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-sdscx\" (UID: \"5c655a6b-6d89-459f-b331-3f8b08d0eae0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-sdscx" Jul 15 23:42:58.445142 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:42:58.445070 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c655a6b-6d89-459f-b331-3f8b08d0eae0-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-sdscx\" (UID: \"5c655a6b-6d89-459f-b331-3f8b08d0eae0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-sdscx" Jul 15 23:42:58.445142 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:42:58.445086 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vc98r\" (UniqueName: \"kubernetes.io/projected/5c655a6b-6d89-459f-b331-3f8b08d0eae0-kube-api-access-vc98r\") pod \"maas-default-gateway-openshift-default-56654dcf7-sdscx\" (UID: \"5c655a6b-6d89-459f-b331-3f8b08d0eae0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-sdscx" Jul 15 23:42:58.445142 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:42:58.445115 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c655a6b-6d89-459f-b331-3f8b08d0eae0-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-sdscx\" (UID: \"5c655a6b-6d89-459f-b331-3f8b08d0eae0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-sdscx" Jul 15 23:42:58.445142 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:42:58.445140 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c655a6b-6d89-459f-b331-3f8b08d0eae0-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-sdscx\" (UID: \"5c655a6b-6d89-459f-b331-3f8b08d0eae0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-sdscx" Jul 15 23:42:58.445420 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:42:58.445189 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c655a6b-6d89-459f-b331-3f8b08d0eae0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-sdscx\" (UID: \"5c655a6b-6d89-459f-b331-3f8b08d0eae0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-sdscx" Jul 15 23:42:58.445420 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:42:58.445251 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c655a6b-6d89-459f-b331-3f8b08d0eae0-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-sdscx\" (UID: \"5c655a6b-6d89-459f-b331-3f8b08d0eae0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-sdscx" Jul 15 23:42:58.445420 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:42:58.445309 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c655a6b-6d89-459f-b331-3f8b08d0eae0-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-sdscx\" (UID: \"5c655a6b-6d89-459f-b331-3f8b08d0eae0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-sdscx" Jul 15 23:42:58.445420 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:42:58.445350 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c655a6b-6d89-459f-b331-3f8b08d0eae0-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-sdscx\" (UID: \"5c655a6b-6d89-459f-b331-3f8b08d0eae0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-sdscx" Jul 15 23:42:58.445622 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:42:58.445432 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c655a6b-6d89-459f-b331-3f8b08d0eae0-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-sdscx\" (UID: \"5c655a6b-6d89-459f-b331-3f8b08d0eae0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-sdscx" Jul 15 23:42:58.445622 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:42:58.445446 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c655a6b-6d89-459f-b331-3f8b08d0eae0-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-sdscx\" (UID: \"5c655a6b-6d89-459f-b331-3f8b08d0eae0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-sdscx" Jul 15 23:42:58.445622 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:42:58.445582 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c655a6b-6d89-459f-b331-3f8b08d0eae0-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-sdscx\" (UID: \"5c655a6b-6d89-459f-b331-3f8b08d0eae0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-sdscx" Jul 15 23:42:58.445990 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:42:58.445952 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c655a6b-6d89-459f-b331-3f8b08d0eae0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-sdscx\" (UID: \"5c655a6b-6d89-459f-b331-3f8b08d0eae0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-sdscx" Jul 15 23:42:58.447554 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:42:58.447527 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c655a6b-6d89-459f-b331-3f8b08d0eae0-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-sdscx\" (UID: \"5c655a6b-6d89-459f-b331-3f8b08d0eae0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-sdscx" Jul 15 23:42:58.447661 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:42:58.447625 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c655a6b-6d89-459f-b331-3f8b08d0eae0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-sdscx\" (UID: \"5c655a6b-6d89-459f-b331-3f8b08d0eae0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-sdscx" Jul 15 23:42:58.456021 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:42:58.456000 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c655a6b-6d89-459f-b331-3f8b08d0eae0-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-sdscx\" (UID: \"5c655a6b-6d89-459f-b331-3f8b08d0eae0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-sdscx" Jul 15 23:42:58.456110 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:42:58.456035 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vc98r\" (UniqueName: \"kubernetes.io/projected/5c655a6b-6d89-459f-b331-3f8b08d0eae0-kube-api-access-vc98r\") pod \"maas-default-gateway-openshift-default-56654dcf7-sdscx\" (UID: \"5c655a6b-6d89-459f-b331-3f8b08d0eae0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-sdscx" Jul 15 23:42:58.557456 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:42:58.557367 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-sdscx" Jul 15 23:42:58.691308 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:42:58.691281 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-sdscx"] Jul 15 23:42:58.694248 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:42:58.694218 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c655a6b_6d89_459f_b331_3f8b08d0eae0.slice/crio-84e0fd9916e27c6f5db872176ab852e20e5f0d0047384aa34d87597e66c54676 WatchSource:0}: Error finding container 84e0fd9916e27c6f5db872176ab852e20e5f0d0047384aa34d87597e66c54676: Status 404 returned error can't find the container with id 84e0fd9916e27c6f5db872176ab852e20e5f0d0047384aa34d87597e66c54676 Jul 15 23:42:58.696343 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:42:58.696305 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 15 23:42:58.696419 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:42:58.696392 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 15 23:42:58.696462 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:42:58.696440 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 15 23:42:58.886131 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:42:58.886042 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-sdscx" event={"ID":"5c655a6b-6d89-459f-b331-3f8b08d0eae0","Type":"ContainerStarted","Data":"0bdc693117d6e560dfc6619a7870aca9a9890adb9144d44aaaa070b0d086ae9e"} Jul 15 23:42:58.886131 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:42:58.886104 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-sdscx" event={"ID":"5c655a6b-6d89-459f-b331-3f8b08d0eae0","Type":"ContainerStarted","Data":"84e0fd9916e27c6f5db872176ab852e20e5f0d0047384aa34d87597e66c54676"} Jul 15 23:42:58.903418 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:42:58.903339 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-sdscx" podStartSLOduration=0.903320926 podStartE2EDuration="903.320926ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 23:42:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 23:42:58.903217216 +0000 UTC m=+394.276675198" watchObservedRunningTime="2026-07-15 23:42:58.903320926 +0000 UTC m=+394.276778892" Jul 15 23:42:59.557920 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:42:59.557888 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-sdscx" Jul 15 23:42:59.562292 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:42:59.562263 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-sdscx" Jul 15 23:42:59.890553 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:42:59.890470 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-sdscx" Jul 15 23:42:59.891463 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:42:59.891444 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-sdscx" Jul 15 23:43:14.686283 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:43:14.686243 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:43:14.790326 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:43:14.790294 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:43:14.895482 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:43:14.895449 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:43:15.477302 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:43:15.477274 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-58cd955675-897kk"] Jul 15 23:43:15.480742 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:43:15.480725 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-897kk" Jul 15 23:43:15.483231 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:43:15.483202 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 15 23:43:15.483231 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:43:15.483206 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 15 23:43:15.483387 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:43:15.483232 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-2mr4t\"" Jul 15 23:43:15.488825 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:43:15.488805 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-897kk"] Jul 15 23:43:15.519933 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:43:15.519908 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:43:15.587887 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:43:15.587859 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:43:15.604193 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:43:15.604168 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ps42f\" (UniqueName: \"kubernetes.io/projected/02d35b22-882e-4374-85a0-3176211c3850-kube-api-access-ps42f\") pod \"authorino-58cd955675-897kk\" (UID: \"02d35b22-882e-4374-85a0-3176211c3850\") " pod="kuadrant-system/authorino-58cd955675-897kk" Jul 15 23:43:15.698945 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:43:15.698922 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-75d556bd77-8qpbr"] Jul 15 23:43:15.702162 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:43:15.702147 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-8qpbr" Jul 15 23:43:15.704892 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:43:15.704877 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ps42f\" (UniqueName: \"kubernetes.io/projected/02d35b22-882e-4374-85a0-3176211c3850-kube-api-access-ps42f\") pod \"authorino-58cd955675-897kk\" (UID: \"02d35b22-882e-4374-85a0-3176211c3850\") " pod="kuadrant-system/authorino-58cd955675-897kk" Jul 15 23:43:15.711687 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:43:15.711663 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-8qpbr"] Jul 15 23:43:15.714862 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:43:15.714841 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ps42f\" (UniqueName: \"kubernetes.io/projected/02d35b22-882e-4374-85a0-3176211c3850-kube-api-access-ps42f\") pod \"authorino-58cd955675-897kk\" (UID: \"02d35b22-882e-4374-85a0-3176211c3850\") " pod="kuadrant-system/authorino-58cd955675-897kk" Jul 15 23:43:15.791731 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:43:15.791706 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-897kk" Jul 15 23:43:15.805903 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:43:15.805878 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-csql9\" (UniqueName: \"kubernetes.io/projected/6cfb5a18-2ecd-4925-9873-d93f4b6b7b90-kube-api-access-csql9\") pod \"authorino-75d556bd77-8qpbr\" (UID: \"6cfb5a18-2ecd-4925-9873-d93f4b6b7b90\") " pod="kuadrant-system/authorino-75d556bd77-8qpbr" Jul 15 23:43:15.906434 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:43:15.906405 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-csql9\" (UniqueName: \"kubernetes.io/projected/6cfb5a18-2ecd-4925-9873-d93f4b6b7b90-kube-api-access-csql9\") pod \"authorino-75d556bd77-8qpbr\" (UID: \"6cfb5a18-2ecd-4925-9873-d93f4b6b7b90\") " pod="kuadrant-system/authorino-75d556bd77-8qpbr" Jul 15 23:43:15.912812 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:43:15.912776 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-897kk"] Jul 15 23:43:15.915100 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:43:15.915076 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod02d35b22_882e_4374_85a0_3176211c3850.slice/crio-35f69d9d91c472d3d3872a3cdb735b85ad3f4c1bcbaab8c137dc9c35b1b5178f WatchSource:0}: Error finding container 35f69d9d91c472d3d3872a3cdb735b85ad3f4c1bcbaab8c137dc9c35b1b5178f: Status 404 returned error can't find the container with id 35f69d9d91c472d3d3872a3cdb735b85ad3f4c1bcbaab8c137dc9c35b1b5178f Jul 15 23:43:15.915279 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:43:15.915259 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-csql9\" (UniqueName: \"kubernetes.io/projected/6cfb5a18-2ecd-4925-9873-d93f4b6b7b90-kube-api-access-csql9\") pod \"authorino-75d556bd77-8qpbr\" (UID: \"6cfb5a18-2ecd-4925-9873-d93f4b6b7b90\") " pod="kuadrant-system/authorino-75d556bd77-8qpbr" Jul 15 23:43:15.916673 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:43:15.916648 2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 15 23:43:15.963708 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:43:15.963683 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-897kk" event={"ID":"02d35b22-882e-4374-85a0-3176211c3850","Type":"ContainerStarted","Data":"35f69d9d91c472d3d3872a3cdb735b85ad3f4c1bcbaab8c137dc9c35b1b5178f"} Jul 15 23:43:16.011056 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:43:16.011034 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-8qpbr" Jul 15 23:43:16.142142 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:43:16.142120 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-8qpbr"] Jul 15 23:43:16.144119 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:43:16.144091 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6cfb5a18_2ecd_4925_9873_d93f4b6b7b90.slice/crio-f90f8e6db749fa54d51a09009d64f580f7332a48eec99e108167467a876bbd84 WatchSource:0}: Error finding container f90f8e6db749fa54d51a09009d64f580f7332a48eec99e108167467a876bbd84: Status 404 returned error can't find the container with id f90f8e6db749fa54d51a09009d64f580f7332a48eec99e108167467a876bbd84 Jul 15 23:43:16.983535 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:43:16.983491 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-8qpbr" event={"ID":"6cfb5a18-2ecd-4925-9873-d93f4b6b7b90","Type":"ContainerStarted","Data":"f90f8e6db749fa54d51a09009d64f580f7332a48eec99e108167467a876bbd84"} Jul 15 23:43:20.000995 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:43:20.000941 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-8qpbr" event={"ID":"6cfb5a18-2ecd-4925-9873-d93f4b6b7b90","Type":"ContainerStarted","Data":"8659c190409360e7c2dbc78ca00a80193885c5602c71a81674638d34b6842a03"} Jul 15 23:43:20.002298 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:43:20.002274 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-897kk" event={"ID":"02d35b22-882e-4374-85a0-3176211c3850","Type":"ContainerStarted","Data":"453b35e361b7726431af0b051b6a04fee1ce3f461338dd6d0b202f19787775fd"} Jul 15 23:43:20.020586 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:43:20.020545 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-75d556bd77-8qpbr" podStartSLOduration=1.6790166389999999 podStartE2EDuration="5.020532565s" totalImagesPullingTime="3.341515926s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 23:43:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 23:43:20.019423892 +0000 UTC m=+415.392881886" watchObservedRunningTime="2026-07-15 23:43:20.020532565 +0000 UTC m=+415.393990528" Jul 15 23:43:20.033134 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:43:20.033080 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-58cd955675-897kk" podStartSLOduration=1.474883821 podStartE2EDuration="5.033060333s" totalImagesPullingTime="3.558176512s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 23:43:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 23:43:20.031854523 +0000 UTC m=+415.405312487" watchObservedRunningTime="2026-07-15 23:43:20.033060333 +0000 UTC m=+415.406518297" Jul 15 23:43:20.056148 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:43:20.056121 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-897kk"] Jul 15 23:43:22.011626 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:43:22.011586 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-58cd955675-897kk" podUID="02d35b22-882e-4374-85a0-3176211c3850" containerName="authorino" containerID="cri-o://453b35e361b7726431af0b051b6a04fee1ce3f461338dd6d0b202f19787775fd" gracePeriod=30 Jul 15 23:43:22.261468 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:43:22.261449 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-897kk" Jul 15 23:43:22.370143 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:43:22.370068 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ps42f\" (UniqueName: \"kubernetes.io/projected/02d35b22-882e-4374-85a0-3176211c3850-kube-api-access-ps42f\") pod \"02d35b22-882e-4374-85a0-3176211c3850\" (UID: \"02d35b22-882e-4374-85a0-3176211c3850\") " Jul 15 23:43:22.372133 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:43:22.372107 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02d35b22-882e-4374-85a0-3176211c3850-kube-api-access-ps42f" (OuterVolumeSpecName: "kube-api-access-ps42f") pod "02d35b22-882e-4374-85a0-3176211c3850" (UID: "02d35b22-882e-4374-85a0-3176211c3850"). InnerVolumeSpecName "kube-api-access-ps42f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 23:43:22.471177 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:43:22.471145 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ps42f\" (UniqueName: \"kubernetes.io/projected/02d35b22-882e-4374-85a0-3176211c3850-kube-api-access-ps42f\") on node \"ip-10-0-128-197.ec2.internal\" DevicePath \"\"" Jul 15 23:43:23.017225 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:43:23.017191 2563 generic.go:356] "Generic (PLEG): container finished" podID="02d35b22-882e-4374-85a0-3176211c3850" containerID="453b35e361b7726431af0b051b6a04fee1ce3f461338dd6d0b202f19787775fd" exitCode=0 Jul 15 23:43:23.017609 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:43:23.017233 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-897kk" Jul 15 23:43:23.017609 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:43:23.017241 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-897kk" event={"ID":"02d35b22-882e-4374-85a0-3176211c3850","Type":"ContainerDied","Data":"453b35e361b7726431af0b051b6a04fee1ce3f461338dd6d0b202f19787775fd"} Jul 15 23:43:23.017609 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:43:23.017268 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-897kk" event={"ID":"02d35b22-882e-4374-85a0-3176211c3850","Type":"ContainerDied","Data":"35f69d9d91c472d3d3872a3cdb735b85ad3f4c1bcbaab8c137dc9c35b1b5178f"} Jul 15 23:43:23.017609 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:43:23.017283 2563 scope.go:117] "RemoveContainer" containerID="453b35e361b7726431af0b051b6a04fee1ce3f461338dd6d0b202f19787775fd" Jul 15 23:43:23.026119 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:43:23.026104 2563 scope.go:117] "RemoveContainer" containerID="453b35e361b7726431af0b051b6a04fee1ce3f461338dd6d0b202f19787775fd" Jul 15 23:43:23.026348 ip-10-0-128-197 kubenswrapper[2563]: E0715 23:43:23.026332 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"453b35e361b7726431af0b051b6a04fee1ce3f461338dd6d0b202f19787775fd\": container with ID starting with 453b35e361b7726431af0b051b6a04fee1ce3f461338dd6d0b202f19787775fd not found: ID does not exist" containerID="453b35e361b7726431af0b051b6a04fee1ce3f461338dd6d0b202f19787775fd" Jul 15 23:43:23.026395 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:43:23.026356 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"453b35e361b7726431af0b051b6a04fee1ce3f461338dd6d0b202f19787775fd"} err="failed to get container status \"453b35e361b7726431af0b051b6a04fee1ce3f461338dd6d0b202f19787775fd\": rpc error: code = NotFound desc = could not find container \"453b35e361b7726431af0b051b6a04fee1ce3f461338dd6d0b202f19787775fd\": container with ID starting with 453b35e361b7726431af0b051b6a04fee1ce3f461338dd6d0b202f19787775fd not found: ID does not exist" Jul 15 23:43:23.038033 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:43:23.038009 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-897kk"] Jul 15 23:43:23.041358 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:43:23.041338 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-58cd955675-897kk"] Jul 15 23:43:23.169354 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:43:23.169323 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="02d35b22-882e-4374-85a0-3176211c3850" path="/var/lib/kubelet/pods/02d35b22-882e-4374-85a0-3176211c3850/volumes" Jul 15 23:43:31.320669 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:43:31.320635 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:43:31.417826 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:43:31.417799 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:43:31.780929 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:43:31.780895 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:43:43.605183 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:43:43.605145 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:43:43.830001 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:43:43.829960 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5d4b67c446-ljs5v"] Jul 15 23:43:43.830349 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:43:43.830338 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="02d35b22-882e-4374-85a0-3176211c3850" containerName="authorino" Jul 15 23:43:43.830394 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:43:43.830351 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="02d35b22-882e-4374-85a0-3176211c3850" containerName="authorino" Jul 15 23:43:43.830429 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:43:43.830413 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="02d35b22-882e-4374-85a0-3176211c3850" containerName="authorino" Jul 15 23:43:43.834876 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:43:43.834860 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-ljs5v" Jul 15 23:43:43.839226 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:43:43.839201 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-ljs5v"] Jul 15 23:43:43.953362 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:43:43.953299 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l64z9\" (UniqueName: \"kubernetes.io/projected/5428e11b-8582-4862-a79e-0582435b5e7a-kube-api-access-l64z9\") pod \"authorino-5d4b67c446-ljs5v\" (UID: \"5428e11b-8582-4862-a79e-0582435b5e7a\") " pod="kuadrant-system/authorino-5d4b67c446-ljs5v" Jul 15 23:43:44.054393 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:43:44.054368 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l64z9\" (UniqueName: \"kubernetes.io/projected/5428e11b-8582-4862-a79e-0582435b5e7a-kube-api-access-l64z9\") pod \"authorino-5d4b67c446-ljs5v\" (UID: \"5428e11b-8582-4862-a79e-0582435b5e7a\") " pod="kuadrant-system/authorino-5d4b67c446-ljs5v" Jul 15 23:43:44.063476 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:43:44.063456 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-ljs5v"] Jul 15 23:43:44.063707 ip-10-0-128-197 kubenswrapper[2563]: E0715 23:43:44.063677 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-l64z9], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-5d4b67c446-ljs5v" podUID="5428e11b-8582-4862-a79e-0582435b5e7a" Jul 15 23:43:44.064749 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:43:44.064729 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l64z9\" (UniqueName: \"kubernetes.io/projected/5428e11b-8582-4862-a79e-0582435b5e7a-kube-api-access-l64z9\") pod \"authorino-5d4b67c446-ljs5v\" (UID: \"5428e11b-8582-4862-a79e-0582435b5e7a\") " pod="kuadrant-system/authorino-5d4b67c446-ljs5v" Jul 15 23:43:44.090522 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:43:44.090497 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-766ccd6b46-76ktl"] Jul 15 23:43:44.093774 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:43:44.093754 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-766ccd6b46-76ktl" Jul 15 23:43:44.099840 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:43:44.099819 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-766ccd6b46-76ktl"] Jul 15 23:43:44.119042 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:43:44.119018 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-ljs5v" Jul 15 23:43:44.124997 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:43:44.124961 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-ljs5v" Jul 15 23:43:44.155362 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:43:44.155342 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s27rn\" (UniqueName: \"kubernetes.io/projected/2de84e5c-5322-44ee-8de2-83edbb1a5371-kube-api-access-s27rn\") pod \"authorino-766ccd6b46-76ktl\" (UID: \"2de84e5c-5322-44ee-8de2-83edbb1a5371\") " pod="kuadrant-system/authorino-766ccd6b46-76ktl" Jul 15 23:43:44.256684 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:43:44.256661 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l64z9\" (UniqueName: \"kubernetes.io/projected/5428e11b-8582-4862-a79e-0582435b5e7a-kube-api-access-l64z9\") pod \"5428e11b-8582-4862-a79e-0582435b5e7a\" (UID: \"5428e11b-8582-4862-a79e-0582435b5e7a\") " Jul 15 23:43:44.256839 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:43:44.256826 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s27rn\" (UniqueName: \"kubernetes.io/projected/2de84e5c-5322-44ee-8de2-83edbb1a5371-kube-api-access-s27rn\") pod \"authorino-766ccd6b46-76ktl\" (UID: \"2de84e5c-5322-44ee-8de2-83edbb1a5371\") " pod="kuadrant-system/authorino-766ccd6b46-76ktl" Jul 15 23:43:44.258765 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:43:44.258733 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5428e11b-8582-4862-a79e-0582435b5e7a-kube-api-access-l64z9" (OuterVolumeSpecName: "kube-api-access-l64z9") pod "5428e11b-8582-4862-a79e-0582435b5e7a" (UID: "5428e11b-8582-4862-a79e-0582435b5e7a"). InnerVolumeSpecName "kube-api-access-l64z9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 23:43:44.264929 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:43:44.264903 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s27rn\" (UniqueName: \"kubernetes.io/projected/2de84e5c-5322-44ee-8de2-83edbb1a5371-kube-api-access-s27rn\") pod \"authorino-766ccd6b46-76ktl\" (UID: \"2de84e5c-5322-44ee-8de2-83edbb1a5371\") " pod="kuadrant-system/authorino-766ccd6b46-76ktl" Jul 15 23:43:44.287047 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:43:44.287027 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:43:44.358188 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:43:44.358157 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l64z9\" (UniqueName: \"kubernetes.io/projected/5428e11b-8582-4862-a79e-0582435b5e7a-kube-api-access-l64z9\") on node \"ip-10-0-128-197.ec2.internal\" DevicePath \"\"" Jul 15 23:43:44.403900 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:43:44.403874 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-766ccd6b46-76ktl" Jul 15 23:43:44.533423 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:43:44.533368 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-766ccd6b46-76ktl"] Jul 15 23:43:44.535637 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:43:44.535614 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2de84e5c_5322_44ee_8de2_83edbb1a5371.slice/crio-e068efef60e2fbf8847b6027d8813ca4f0cf22b414a2be402a3488fa5e44aa51 WatchSource:0}: Error finding container e068efef60e2fbf8847b6027d8813ca4f0cf22b414a2be402a3488fa5e44aa51: Status 404 returned error can't find the container with id e068efef60e2fbf8847b6027d8813ca4f0cf22b414a2be402a3488fa5e44aa51 Jul 15 23:43:45.125237 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:43:45.125205 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-766ccd6b46-76ktl" event={"ID":"2de84e5c-5322-44ee-8de2-83edbb1a5371","Type":"ContainerStarted","Data":"d29400fa57e2fee5ca48404d77878fad755c124721af7a83918ca007be6710c5"} Jul 15 23:43:45.125605 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:43:45.125246 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-766ccd6b46-76ktl" event={"ID":"2de84e5c-5322-44ee-8de2-83edbb1a5371","Type":"ContainerStarted","Data":"e068efef60e2fbf8847b6027d8813ca4f0cf22b414a2be402a3488fa5e44aa51"} Jul 15 23:43:45.125605 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:43:45.125280 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-ljs5v" Jul 15 23:43:45.140663 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:43:45.140614 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-766ccd6b46-76ktl" podStartSLOduration=0.796966583 podStartE2EDuration="1.140601134s" totalImagesPullingTime="343.634551ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 23:43:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 23:43:45.139251213 +0000 UTC m=+440.512709195" watchObservedRunningTime="2026-07-15 23:43:45.140601134 +0000 UTC m=+440.514059098" Jul 15 23:43:45.161486 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:43:45.161459 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-ljs5v"] Jul 15 23:43:45.171992 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:43:45.171951 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-8qpbr"] Jul 15 23:43:45.172189 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:43:45.172143 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-75d556bd77-8qpbr" podUID="6cfb5a18-2ecd-4925-9873-d93f4b6b7b90" containerName="authorino" containerID="cri-o://8659c190409360e7c2dbc78ca00a80193885c5602c71a81674638d34b6842a03" gracePeriod=30 Jul 15 23:43:45.172502 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:43:45.172481 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-ljs5v"] Jul 15 23:43:45.377894 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:43:45.377861 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-555cfbc94-5x458"] Jul 15 23:43:45.381884 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:43:45.381830 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-555cfbc94-5x458" Jul 15 23:43:45.384147 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:43:45.384130 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Jul 15 23:43:45.389695 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:43:45.389675 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-555cfbc94-5x458"] Jul 15 23:43:45.413066 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:43:45.413048 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-8qpbr" Jul 15 23:43:45.468521 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:43:45.468493 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-csql9\" (UniqueName: \"kubernetes.io/projected/6cfb5a18-2ecd-4925-9873-d93f4b6b7b90-kube-api-access-csql9\") pod \"6cfb5a18-2ecd-4925-9873-d93f4b6b7b90\" (UID: \"6cfb5a18-2ecd-4925-9873-d93f4b6b7b90\") " Jul 15 23:43:45.468758 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:43:45.468743 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/8be49e59-a30d-4230-a602-6964668430bd-tls-cert\") pod \"authorino-555cfbc94-5x458\" (UID: \"8be49e59-a30d-4230-a602-6964668430bd\") " pod="kuadrant-system/authorino-555cfbc94-5x458" Jul 15 23:43:45.468895 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:43:45.468871 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ztcpj\" (UniqueName: \"kubernetes.io/projected/8be49e59-a30d-4230-a602-6964668430bd-kube-api-access-ztcpj\") pod \"authorino-555cfbc94-5x458\" (UID: \"8be49e59-a30d-4230-a602-6964668430bd\") " pod="kuadrant-system/authorino-555cfbc94-5x458" Jul 15 23:43:45.470515 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:43:45.470494 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6cfb5a18-2ecd-4925-9873-d93f4b6b7b90-kube-api-access-csql9" (OuterVolumeSpecName: "kube-api-access-csql9") pod "6cfb5a18-2ecd-4925-9873-d93f4b6b7b90" (UID: "6cfb5a18-2ecd-4925-9873-d93f4b6b7b90"). InnerVolumeSpecName "kube-api-access-csql9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 23:43:45.569993 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:43:45.569953 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/8be49e59-a30d-4230-a602-6964668430bd-tls-cert\") pod \"authorino-555cfbc94-5x458\" (UID: \"8be49e59-a30d-4230-a602-6964668430bd\") " pod="kuadrant-system/authorino-555cfbc94-5x458" Jul 15 23:43:45.570145 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:43:45.570042 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztcpj\" (UniqueName: \"kubernetes.io/projected/8be49e59-a30d-4230-a602-6964668430bd-kube-api-access-ztcpj\") pod \"authorino-555cfbc94-5x458\" (UID: \"8be49e59-a30d-4230-a602-6964668430bd\") " pod="kuadrant-system/authorino-555cfbc94-5x458" Jul 15 23:43:45.570222 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:43:45.570128 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-csql9\" (UniqueName: \"kubernetes.io/projected/6cfb5a18-2ecd-4925-9873-d93f4b6b7b90-kube-api-access-csql9\") on node \"ip-10-0-128-197.ec2.internal\" DevicePath \"\"" Jul 15 23:43:45.572369 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:43:45.572349 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/8be49e59-a30d-4230-a602-6964668430bd-tls-cert\") pod \"authorino-555cfbc94-5x458\" (UID: \"8be49e59-a30d-4230-a602-6964668430bd\") " pod="kuadrant-system/authorino-555cfbc94-5x458" Jul 15 23:43:45.577998 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:43:45.577959 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ztcpj\" (UniqueName: \"kubernetes.io/projected/8be49e59-a30d-4230-a602-6964668430bd-kube-api-access-ztcpj\") pod \"authorino-555cfbc94-5x458\" (UID: \"8be49e59-a30d-4230-a602-6964668430bd\") " pod="kuadrant-system/authorino-555cfbc94-5x458" Jul 15 23:43:45.707390 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:43:45.707324 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-555cfbc94-5x458" Jul 15 23:43:45.837921 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:43:45.837896 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-555cfbc94-5x458"] Jul 15 23:43:45.839675 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:43:45.839645 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8be49e59_a30d_4230_a602_6964668430bd.slice/crio-3d74e5ab73d9dbc748ff7379a6c4187c48271a3bd8caf5762b5584aa81173237 WatchSource:0}: Error finding container 3d74e5ab73d9dbc748ff7379a6c4187c48271a3bd8caf5762b5584aa81173237: Status 404 returned error can't find the container with id 3d74e5ab73d9dbc748ff7379a6c4187c48271a3bd8caf5762b5584aa81173237 Jul 15 23:43:46.131382 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:43:46.131347 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-555cfbc94-5x458" event={"ID":"8be49e59-a30d-4230-a602-6964668430bd","Type":"ContainerStarted","Data":"3d74e5ab73d9dbc748ff7379a6c4187c48271a3bd8caf5762b5584aa81173237"} Jul 15 23:43:46.132635 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:43:46.132606 2563 generic.go:356] "Generic (PLEG): container finished" podID="6cfb5a18-2ecd-4925-9873-d93f4b6b7b90" containerID="8659c190409360e7c2dbc78ca00a80193885c5602c71a81674638d34b6842a03" exitCode=0 Jul 15 23:43:46.132769 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:43:46.132647 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-8qpbr" Jul 15 23:43:46.132769 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:43:46.132688 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-8qpbr" event={"ID":"6cfb5a18-2ecd-4925-9873-d93f4b6b7b90","Type":"ContainerDied","Data":"8659c190409360e7c2dbc78ca00a80193885c5602c71a81674638d34b6842a03"} Jul 15 23:43:46.132769 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:43:46.132721 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-8qpbr" event={"ID":"6cfb5a18-2ecd-4925-9873-d93f4b6b7b90","Type":"ContainerDied","Data":"f90f8e6db749fa54d51a09009d64f580f7332a48eec99e108167467a876bbd84"} Jul 15 23:43:46.132769 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:43:46.132744 2563 scope.go:117] "RemoveContainer" containerID="8659c190409360e7c2dbc78ca00a80193885c5602c71a81674638d34b6842a03" Jul 15 23:43:46.146408 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:43:46.146391 2563 scope.go:117] "RemoveContainer" containerID="8659c190409360e7c2dbc78ca00a80193885c5602c71a81674638d34b6842a03" Jul 15 23:43:46.146674 ip-10-0-128-197 kubenswrapper[2563]: E0715 23:43:46.146650 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8659c190409360e7c2dbc78ca00a80193885c5602c71a81674638d34b6842a03\": container with ID starting with 8659c190409360e7c2dbc78ca00a80193885c5602c71a81674638d34b6842a03 not found: ID does not exist" containerID="8659c190409360e7c2dbc78ca00a80193885c5602c71a81674638d34b6842a03" Jul 15 23:43:46.146745 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:43:46.146686 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8659c190409360e7c2dbc78ca00a80193885c5602c71a81674638d34b6842a03"} err="failed to get container status \"8659c190409360e7c2dbc78ca00a80193885c5602c71a81674638d34b6842a03\": rpc error: code = NotFound desc = could not find container \"8659c190409360e7c2dbc78ca00a80193885c5602c71a81674638d34b6842a03\": container with ID starting with 8659c190409360e7c2dbc78ca00a80193885c5602c71a81674638d34b6842a03 not found: ID does not exist" Jul 15 23:43:46.157809 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:43:46.157787 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-8qpbr"] Jul 15 23:43:46.162290 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:43:46.162270 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-75d556bd77-8qpbr"] Jul 15 23:43:46.457541 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:43:46.457464 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:43:47.140014 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:43:47.139938 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-555cfbc94-5x458" event={"ID":"8be49e59-a30d-4230-a602-6964668430bd","Type":"ContainerStarted","Data":"7040316559b4089bee7a4a667a8ef901ed2cf9b1dc7e441b129cc1e1ecc365de"} Jul 15 23:43:47.154532 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:43:47.154483 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-555cfbc94-5x458" podStartSLOduration=1.77944031 podStartE2EDuration="2.154466566s" totalImagesPullingTime="375.026256ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 23:43:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 23:43:47.153228513 +0000 UTC m=+442.526686477" watchObservedRunningTime="2026-07-15 23:43:47.154466566 +0000 UTC m=+442.527924568" Jul 15 23:43:47.179460 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:43:47.172596 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5428e11b-8582-4862-a79e-0582435b5e7a" path="/var/lib/kubelet/pods/5428e11b-8582-4862-a79e-0582435b5e7a/volumes" Jul 15 23:43:47.179460 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:43:47.172919 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6cfb5a18-2ecd-4925-9873-d93f4b6b7b90" path="/var/lib/kubelet/pods/6cfb5a18-2ecd-4925-9873-d93f4b6b7b90/volumes" Jul 15 23:43:47.184715 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:43:47.182795 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-766ccd6b46-76ktl"] Jul 15 23:43:47.184715 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:43:47.182986 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-766ccd6b46-76ktl" podUID="2de84e5c-5322-44ee-8de2-83edbb1a5371" containerName="authorino" containerID="cri-o://d29400fa57e2fee5ca48404d77878fad755c124721af7a83918ca007be6710c5" gracePeriod=30 Jul 15 23:43:47.437806 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:43:47.437777 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-766ccd6b46-76ktl" Jul 15 23:43:47.487888 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:43:47.487854 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s27rn\" (UniqueName: \"kubernetes.io/projected/2de84e5c-5322-44ee-8de2-83edbb1a5371-kube-api-access-s27rn\") pod \"2de84e5c-5322-44ee-8de2-83edbb1a5371\" (UID: \"2de84e5c-5322-44ee-8de2-83edbb1a5371\") " Jul 15 23:43:47.489839 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:43:47.489810 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2de84e5c-5322-44ee-8de2-83edbb1a5371-kube-api-access-s27rn" (OuterVolumeSpecName: "kube-api-access-s27rn") pod "2de84e5c-5322-44ee-8de2-83edbb1a5371" (UID: "2de84e5c-5322-44ee-8de2-83edbb1a5371"). InnerVolumeSpecName "kube-api-access-s27rn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 23:43:47.588593 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:43:47.588564 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s27rn\" (UniqueName: \"kubernetes.io/projected/2de84e5c-5322-44ee-8de2-83edbb1a5371-kube-api-access-s27rn\") on node \"ip-10-0-128-197.ec2.internal\" DevicePath \"\"" Jul 15 23:43:48.145624 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:43:48.145581 2563 generic.go:356] "Generic (PLEG): container finished" podID="2de84e5c-5322-44ee-8de2-83edbb1a5371" containerID="d29400fa57e2fee5ca48404d77878fad755c124721af7a83918ca007be6710c5" exitCode=0 Jul 15 23:43:48.146153 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:43:48.145637 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-766ccd6b46-76ktl" Jul 15 23:43:48.146153 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:43:48.145666 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-766ccd6b46-76ktl" event={"ID":"2de84e5c-5322-44ee-8de2-83edbb1a5371","Type":"ContainerDied","Data":"d29400fa57e2fee5ca48404d77878fad755c124721af7a83918ca007be6710c5"} Jul 15 23:43:48.146153 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:43:48.145715 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-766ccd6b46-76ktl" event={"ID":"2de84e5c-5322-44ee-8de2-83edbb1a5371","Type":"ContainerDied","Data":"e068efef60e2fbf8847b6027d8813ca4f0cf22b414a2be402a3488fa5e44aa51"} Jul 15 23:43:48.146153 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:43:48.145738 2563 scope.go:117] "RemoveContainer" containerID="d29400fa57e2fee5ca48404d77878fad755c124721af7a83918ca007be6710c5" Jul 15 23:43:48.154675 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:43:48.154654 2563 scope.go:117] "RemoveContainer" containerID="d29400fa57e2fee5ca48404d77878fad755c124721af7a83918ca007be6710c5" Jul 15 23:43:48.154944 ip-10-0-128-197 kubenswrapper[2563]: E0715 23:43:48.154925 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d29400fa57e2fee5ca48404d77878fad755c124721af7a83918ca007be6710c5\": container with ID starting with d29400fa57e2fee5ca48404d77878fad755c124721af7a83918ca007be6710c5 not found: ID does not exist" containerID="d29400fa57e2fee5ca48404d77878fad755c124721af7a83918ca007be6710c5" Jul 15 23:43:48.155057 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:43:48.154955 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d29400fa57e2fee5ca48404d77878fad755c124721af7a83918ca007be6710c5"} err="failed to get container status \"d29400fa57e2fee5ca48404d77878fad755c124721af7a83918ca007be6710c5\": rpc error: code = NotFound desc = could not find container \"d29400fa57e2fee5ca48404d77878fad755c124721af7a83918ca007be6710c5\": container with ID starting with d29400fa57e2fee5ca48404d77878fad755c124721af7a83918ca007be6710c5 not found: ID does not exist" Jul 15 23:43:48.165506 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:43:48.165486 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-766ccd6b46-76ktl"] Jul 15 23:43:48.167334 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:43:48.167310 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-766ccd6b46-76ktl"] Jul 15 23:43:49.169157 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:43:49.169125 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2de84e5c-5322-44ee-8de2-83edbb1a5371" path="/var/lib/kubelet/pods/2de84e5c-5322-44ee-8de2-83edbb1a5371/volumes" Jul 15 23:43:49.694589 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:43:49.694555 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:43:49.778141 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:43:49.778102 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:43:49.829460 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:43:49.829429 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:43:49.916813 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:43:49.916785 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:43:53.037622 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:43:53.037455 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:43:53.139172 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:43:53.139114 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:44:24.772395 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:44:24.772362 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:44:24.869419 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:44:24.869391 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:44:25.283140 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:44:25.283067 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:44:25.650758 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:44:25.650668 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:44:25.767977 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:44:25.767921 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:44:25.862159 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:44:25.862132 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:44:34.367549 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:44:34.367508 2563 generic.go:356] "Generic (PLEG): container finished" podID="5c655a6b-6d89-459f-b331-3f8b08d0eae0" containerID="0bdc693117d6e560dfc6619a7870aca9a9890adb9144d44aaaa070b0d086ae9e" exitCode=0 Jul 15 23:44:34.367957 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:44:34.367581 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-sdscx" event={"ID":"5c655a6b-6d89-459f-b331-3f8b08d0eae0","Type":"ContainerDied","Data":"0bdc693117d6e560dfc6619a7870aca9a9890adb9144d44aaaa070b0d086ae9e"} Jul 15 23:44:34.367957 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:44:34.367947 2563 scope.go:117] "RemoveContainer" containerID="0bdc693117d6e560dfc6619a7870aca9a9890adb9144d44aaaa070b0d086ae9e" Jul 15 23:44:34.368902 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:44:34.368876 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 15 23:44:34.368991 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:44:34.368954 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 15 23:44:34.369041 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:44:34.369011 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 15 23:44:35.373900 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:44:35.373867 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-sdscx" event={"ID":"5c655a6b-6d89-459f-b331-3f8b08d0eae0","Type":"ContainerStarted","Data":"b418e00139f77a147fe394ce42aa756c3bffb7b4381aa5195342cd3cac271343"} Jul 15 23:44:35.558145 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:44:35.558115 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-sdscx" Jul 15 23:44:35.558348 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:44:35.558329 2563 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-sdscx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" start-of-body= Jul 15 23:44:35.558391 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:44:35.558374 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-sdscx" podUID="5c655a6b-6d89-459f-b331-3f8b08d0eae0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" Jul 15 23:44:36.135415 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:44:36.135373 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:44:36.230442 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:44:36.230410 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:44:36.558313 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:44:36.558279 2563 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-sdscx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" start-of-body= Jul 15 23:44:36.558787 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:44:36.558335 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-sdscx" podUID="5c655a6b-6d89-459f-b331-3f8b08d0eae0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" Jul 15 23:44:37.558976 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:44:37.558941 2563 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-sdscx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" start-of-body= Jul 15 23:44:37.559370 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:44:37.559008 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-sdscx" podUID="5c655a6b-6d89-459f-b331-3f8b08d0eae0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" Jul 15 23:44:38.558945 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:44:38.558901 2563 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-sdscx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" start-of-body= Jul 15 23:44:38.559154 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:44:38.558989 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-sdscx" podUID="5c655a6b-6d89-459f-b331-3f8b08d0eae0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" Jul 15 23:44:39.561554 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:44:39.561528 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-sdscx" Jul 15 23:44:39.561906 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:44:39.561777 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-sdscx" Jul 15 23:44:39.562366 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:44:39.562350 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-sdscx" Jul 15 23:44:40.178923 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:44:40.178890 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-7cx2p"] Jul 15 23:44:40.179507 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:44:40.179478 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6cfb5a18-2ecd-4925-9873-d93f4b6b7b90" containerName="authorino" Jul 15 23:44:40.179507 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:44:40.179504 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="6cfb5a18-2ecd-4925-9873-d93f4b6b7b90" containerName="authorino" Jul 15 23:44:40.179661 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:44:40.179532 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2de84e5c-5322-44ee-8de2-83edbb1a5371" containerName="authorino" Jul 15 23:44:40.179661 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:44:40.179542 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="2de84e5c-5322-44ee-8de2-83edbb1a5371" containerName="authorino" Jul 15 23:44:40.179661 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:44:40.179651 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="6cfb5a18-2ecd-4925-9873-d93f4b6b7b90" containerName="authorino" Jul 15 23:44:40.179758 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:44:40.179665 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="2de84e5c-5322-44ee-8de2-83edbb1a5371" containerName="authorino" Jul 15 23:44:40.182284 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:44:40.182267 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-7cx2p" Jul 15 23:44:40.185247 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:44:40.185225 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-j2wfm\"" Jul 15 23:44:40.185364 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:44:40.185227 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 15 23:44:40.185364 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:44:40.185225 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 15 23:44:40.185485 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:44:40.185233 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Jul 15 23:44:40.190189 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:44:40.190169 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-7cx2p"] Jul 15 23:44:40.212924 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:44:40.212897 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:44:40.364309 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:44:40.364274 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f5105a71-3f0e-4641-ad18-0b9c6c552cdc-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-7cx2p\" (UID: \"f5105a71-3f0e-4641-ad18-0b9c6c552cdc\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-7cx2p" Jul 15 23:44:40.364516 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:44:40.364322 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f5105a71-3f0e-4641-ad18-0b9c6c552cdc-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-7cx2p\" (UID: \"f5105a71-3f0e-4641-ad18-0b9c6c552cdc\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-7cx2p" Jul 15 23:44:40.364516 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:44:40.364425 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8sft\" (UniqueName: \"kubernetes.io/projected/f5105a71-3f0e-4641-ad18-0b9c6c552cdc-kube-api-access-f8sft\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-7cx2p\" (UID: \"f5105a71-3f0e-4641-ad18-0b9c6c552cdc\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-7cx2p" Jul 15 23:44:40.364516 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:44:40.364467 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f5105a71-3f0e-4641-ad18-0b9c6c552cdc-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-7cx2p\" (UID: \"f5105a71-3f0e-4641-ad18-0b9c6c552cdc\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-7cx2p" Jul 15 23:44:40.364643 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:44:40.364540 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f5105a71-3f0e-4641-ad18-0b9c6c552cdc-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-7cx2p\" (UID: \"f5105a71-3f0e-4641-ad18-0b9c6c552cdc\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-7cx2p" Jul 15 23:44:40.364643 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:44:40.364564 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5105a71-3f0e-4641-ad18-0b9c6c552cdc-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-7cx2p\" (UID: \"f5105a71-3f0e-4641-ad18-0b9c6c552cdc\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-7cx2p" Jul 15 23:44:40.466070 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:44:40.465955 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8sft\" (UniqueName: \"kubernetes.io/projected/f5105a71-3f0e-4641-ad18-0b9c6c552cdc-kube-api-access-f8sft\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-7cx2p\" (UID: \"f5105a71-3f0e-4641-ad18-0b9c6c552cdc\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-7cx2p" Jul 15 23:44:40.466070 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:44:40.466020 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f5105a71-3f0e-4641-ad18-0b9c6c552cdc-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-7cx2p\" (UID: \"f5105a71-3f0e-4641-ad18-0b9c6c552cdc\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-7cx2p" Jul 15 23:44:40.466070 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:44:40.466058 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f5105a71-3f0e-4641-ad18-0b9c6c552cdc-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-7cx2p\" (UID: \"f5105a71-3f0e-4641-ad18-0b9c6c552cdc\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-7cx2p" Jul 15 23:44:40.466353 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:44:40.466080 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5105a71-3f0e-4641-ad18-0b9c6c552cdc-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-7cx2p\" (UID: \"f5105a71-3f0e-4641-ad18-0b9c6c552cdc\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-7cx2p" Jul 15 23:44:40.466353 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:44:40.466127 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f5105a71-3f0e-4641-ad18-0b9c6c552cdc-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-7cx2p\" (UID: \"f5105a71-3f0e-4641-ad18-0b9c6c552cdc\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-7cx2p" Jul 15 23:44:40.466353 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:44:40.466171 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f5105a71-3f0e-4641-ad18-0b9c6c552cdc-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-7cx2p\" (UID: \"f5105a71-3f0e-4641-ad18-0b9c6c552cdc\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-7cx2p" Jul 15 23:44:40.466516 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:44:40.466493 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f5105a71-3f0e-4641-ad18-0b9c6c552cdc-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-7cx2p\" (UID: \"f5105a71-3f0e-4641-ad18-0b9c6c552cdc\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-7cx2p" Jul 15 23:44:40.466561 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:44:40.466516 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f5105a71-3f0e-4641-ad18-0b9c6c552cdc-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-7cx2p\" (UID: \"f5105a71-3f0e-4641-ad18-0b9c6c552cdc\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-7cx2p" Jul 15 23:44:40.466597 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:44:40.466558 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f5105a71-3f0e-4641-ad18-0b9c6c552cdc-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-7cx2p\" (UID: \"f5105a71-3f0e-4641-ad18-0b9c6c552cdc\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-7cx2p" Jul 15 23:44:40.468421 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:44:40.468396 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f5105a71-3f0e-4641-ad18-0b9c6c552cdc-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-7cx2p\" (UID: \"f5105a71-3f0e-4641-ad18-0b9c6c552cdc\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-7cx2p" Jul 15 23:44:40.468603 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:44:40.468585 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5105a71-3f0e-4641-ad18-0b9c6c552cdc-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-7cx2p\" (UID: \"f5105a71-3f0e-4641-ad18-0b9c6c552cdc\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-7cx2p" Jul 15 23:44:40.473762 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:44:40.473736 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8sft\" (UniqueName: \"kubernetes.io/projected/f5105a71-3f0e-4641-ad18-0b9c6c552cdc-kube-api-access-f8sft\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-7cx2p\" (UID: \"f5105a71-3f0e-4641-ad18-0b9c6c552cdc\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-7cx2p" Jul 15 23:44:40.495835 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:44:40.495812 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-7cx2p" Jul 15 23:44:40.631600 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:44:40.631576 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-7cx2p"] Jul 15 23:44:40.633795 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:44:40.633770 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5105a71_3f0e_4641_ad18_0b9c6c552cdc.slice/crio-6a324e419665a7376037e53b573aa30d7d3570887130c18729f9d5fe5ec2a174 WatchSource:0}: Error finding container 6a324e419665a7376037e53b573aa30d7d3570887130c18729f9d5fe5ec2a174: Status 404 returned error can't find the container with id 6a324e419665a7376037e53b573aa30d7d3570887130c18729f9d5fe5ec2a174 Jul 15 23:44:41.405415 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:44:41.405373 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-7cx2p" event={"ID":"f5105a71-3f0e-4641-ad18-0b9c6c552cdc","Type":"ContainerStarted","Data":"6a324e419665a7376037e53b573aa30d7d3570887130c18729f9d5fe5ec2a174"} Jul 15 23:44:42.412182 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:44:42.412145 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-7cx2p" event={"ID":"f5105a71-3f0e-4641-ad18-0b9c6c552cdc","Type":"ContainerStarted","Data":"de47c3ac5897c22676be5b36e09d6f5c2720f9f6061ab2902254f1f7abe38828"} Jul 15 23:44:42.429326 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:44:42.429276 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-7cx2p" podStartSLOduration=1.379592143 podStartE2EDuration="2.429260527s" totalImagesPullingTime="1.049668384s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 23:44:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 23:44:42.427083324 +0000 UTC m=+497.800541287" watchObservedRunningTime="2026-07-15 23:44:42.429260527 +0000 UTC m=+497.802718490" Jul 15 23:44:47.983528 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:44:47.983496 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:44:50.496348 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:44:50.496315 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-7cx2p" Jul 15 23:44:50.496799 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:44:50.496397 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-7cx2p" Jul 15 23:44:50.508500 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:44:50.508476 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-7cx2p" Jul 15 23:44:51.467192 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:44:51.467166 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-7cx2p" Jul 15 23:45:00.139007 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:45:00.138980 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29735985-4ns9m"] Jul 15 23:45:00.142381 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:45:00.142361 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29735985-4ns9m" Jul 15 23:45:00.144339 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:45:00.144320 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vnx5t\"" Jul 15 23:45:00.148329 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:45:00.148306 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29735985-4ns9m"] Jul 15 23:45:00.153136 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:45:00.153116 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29735985-4ns9m" Jul 15 23:45:00.298767 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:45:00.298747 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29735985-4ns9m"] Jul 15 23:45:00.300808 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:45:00.300770 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48afe7ab_7a09_4d75_816f_b46baf75b974.slice/crio-74e364a414b438c2fca8048ca4a379a2e7424cad1d6cee0018b616936660ffbd WatchSource:0}: Error finding container 74e364a414b438c2fca8048ca4a379a2e7424cad1d6cee0018b616936660ffbd: Status 404 returned error can't find the container with id 74e364a414b438c2fca8048ca4a379a2e7424cad1d6cee0018b616936660ffbd Jul 15 23:45:00.499983 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:45:00.499933 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29735985-4ns9m" event={"ID":"48afe7ab-7a09-4d75-816f-b46baf75b974","Type":"ContainerStarted","Data":"74e364a414b438c2fca8048ca4a379a2e7424cad1d6cee0018b616936660ffbd"} Jul 15 23:45:01.737939 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:45:01.737904 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:45:03.519270 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:45:03.519227 2563 generic.go:356] "Generic (PLEG): container finished" podID="48afe7ab-7a09-4d75-816f-b46baf75b974" containerID="679590c98ead848b531eaa6dc3d2db51a11fbe91ac40918e5bb6a4daea6cdce4" exitCode=0 Jul 15 23:45:03.519715 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:45:03.519286 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29735985-4ns9m" event={"ID":"48afe7ab-7a09-4d75-816f-b46baf75b974","Type":"ContainerDied","Data":"679590c98ead848b531eaa6dc3d2db51a11fbe91ac40918e5bb6a4daea6cdce4"} Jul 15 23:45:04.288279 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:45:04.288249 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:45:04.650501 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:45:04.650478 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29735985-4ns9m" Jul 15 23:45:05.542430 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:45:05.541864 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29735985-4ns9m" Jul 15 23:45:05.542872 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:45:05.542799 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29735985-4ns9m" event={"ID":"48afe7ab-7a09-4d75-816f-b46baf75b974","Type":"ContainerDied","Data":"74e364a414b438c2fca8048ca4a379a2e7424cad1d6cee0018b616936660ffbd"} Jul 15 23:45:05.542872 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:45:05.542853 2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="74e364a414b438c2fca8048ca4a379a2e7424cad1d6cee0018b616936660ffbd" Jul 15 23:45:41.593568 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:45:41.593533 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbws9xh"] Jul 15 23:45:41.594058 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:45:41.593911 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="48afe7ab-7a09-4d75-816f-b46baf75b974" containerName="cleanup" Jul 15 23:45:41.594058 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:45:41.593922 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="48afe7ab-7a09-4d75-816f-b46baf75b974" containerName="cleanup" Jul 15 23:45:41.594058 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:45:41.594022 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="48afe7ab-7a09-4d75-816f-b46baf75b974" containerName="cleanup" Jul 15 23:45:41.600387 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:45:41.600367 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbws9xh" Jul 15 23:45:41.608886 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:45:41.603265 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Jul 15 23:45:41.616021 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:45:41.615978 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbws9xh"] Jul 15 23:45:41.630376 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:45:41.630343 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:45:41.729057 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:45:41.729029 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce6485a1-9190-4c1a-953b-0076bced45d4-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbws9xh\" (UID: \"ce6485a1-9190-4c1a-953b-0076bced45d4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbws9xh" Jul 15 23:45:41.729186 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:45:41.729063 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce6485a1-9190-4c1a-953b-0076bced45d4-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbws9xh\" (UID: \"ce6485a1-9190-4c1a-953b-0076bced45d4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbws9xh" Jul 15 23:45:41.729228 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:45:41.729186 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6twpd\" (UniqueName: \"kubernetes.io/projected/ce6485a1-9190-4c1a-953b-0076bced45d4-kube-api-access-6twpd\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbws9xh\" (UID: \"ce6485a1-9190-4c1a-953b-0076bced45d4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbws9xh" Jul 15 23:45:41.729228 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:45:41.729218 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ce6485a1-9190-4c1a-953b-0076bced45d4-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbws9xh\" (UID: \"ce6485a1-9190-4c1a-953b-0076bced45d4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbws9xh" Jul 15 23:45:41.729298 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:45:41.729272 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce6485a1-9190-4c1a-953b-0076bced45d4-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbws9xh\" (UID: \"ce6485a1-9190-4c1a-953b-0076bced45d4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbws9xh" Jul 15 23:45:41.729298 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:45:41.729288 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ce6485a1-9190-4c1a-953b-0076bced45d4-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbws9xh\" (UID: \"ce6485a1-9190-4c1a-953b-0076bced45d4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbws9xh" Jul 15 23:45:41.829888 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:45:41.829860 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce6485a1-9190-4c1a-953b-0076bced45d4-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbws9xh\" (UID: \"ce6485a1-9190-4c1a-953b-0076bced45d4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbws9xh" Jul 15 23:45:41.829888 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:45:41.829891 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ce6485a1-9190-4c1a-953b-0076bced45d4-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbws9xh\" (UID: \"ce6485a1-9190-4c1a-953b-0076bced45d4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbws9xh" Jul 15 23:45:41.830150 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:45:41.829941 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce6485a1-9190-4c1a-953b-0076bced45d4-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbws9xh\" (UID: \"ce6485a1-9190-4c1a-953b-0076bced45d4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbws9xh" Jul 15 23:45:41.830150 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:45:41.829962 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce6485a1-9190-4c1a-953b-0076bced45d4-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbws9xh\" (UID: \"ce6485a1-9190-4c1a-953b-0076bced45d4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbws9xh" Jul 15 23:45:41.830150 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:45:41.830042 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6twpd\" (UniqueName: \"kubernetes.io/projected/ce6485a1-9190-4c1a-953b-0076bced45d4-kube-api-access-6twpd\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbws9xh\" (UID: \"ce6485a1-9190-4c1a-953b-0076bced45d4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbws9xh" Jul 15 23:45:41.830150 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:45:41.830060 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ce6485a1-9190-4c1a-953b-0076bced45d4-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbws9xh\" (UID: \"ce6485a1-9190-4c1a-953b-0076bced45d4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbws9xh" Jul 15 23:45:41.830382 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:45:41.830357 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ce6485a1-9190-4c1a-953b-0076bced45d4-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbws9xh\" (UID: \"ce6485a1-9190-4c1a-953b-0076bced45d4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbws9xh" Jul 15 23:45:41.830480 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:45:41.830460 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce6485a1-9190-4c1a-953b-0076bced45d4-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbws9xh\" (UID: \"ce6485a1-9190-4c1a-953b-0076bced45d4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbws9xh" Jul 15 23:45:41.830534 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:45:41.830507 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce6485a1-9190-4c1a-953b-0076bced45d4-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbws9xh\" (UID: \"ce6485a1-9190-4c1a-953b-0076bced45d4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbws9xh" Jul 15 23:45:41.832267 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:45:41.832249 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ce6485a1-9190-4c1a-953b-0076bced45d4-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbws9xh\" (UID: \"ce6485a1-9190-4c1a-953b-0076bced45d4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbws9xh" Jul 15 23:45:41.832371 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:45:41.832356 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce6485a1-9190-4c1a-953b-0076bced45d4-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbws9xh\" (UID: \"ce6485a1-9190-4c1a-953b-0076bced45d4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbws9xh" Jul 15 23:45:41.837563 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:45:41.837541 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6twpd\" (UniqueName: \"kubernetes.io/projected/ce6485a1-9190-4c1a-953b-0076bced45d4-kube-api-access-6twpd\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbws9xh\" (UID: \"ce6485a1-9190-4c1a-953b-0076bced45d4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbws9xh" Jul 15 23:45:41.920655 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:45:41.920558 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbws9xh" Jul 15 23:45:42.048080 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:45:42.048056 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbws9xh"] Jul 15 23:45:42.049918 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:45:42.049895 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce6485a1_9190_4c1a_953b_0076bced45d4.slice/crio-b46bfd6e8114be9d483215b3a89bce96f1ad52964c8a3e2167943dc0e739f5f3 WatchSource:0}: Error finding container b46bfd6e8114be9d483215b3a89bce96f1ad52964c8a3e2167943dc0e739f5f3: Status 404 returned error can't find the container with id b46bfd6e8114be9d483215b3a89bce96f1ad52964c8a3e2167943dc0e739f5f3 Jul 15 23:45:42.720945 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:45:42.720910 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbws9xh" event={"ID":"ce6485a1-9190-4c1a-953b-0076bced45d4","Type":"ContainerStarted","Data":"d2deea52c058a335f9be5095fc86f68cb400744d2108d76fc00aac65647a66ce"} Jul 15 23:45:42.721327 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:45:42.720977 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbws9xh" event={"ID":"ce6485a1-9190-4c1a-953b-0076bced45d4","Type":"ContainerStarted","Data":"b46bfd6e8114be9d483215b3a89bce96f1ad52964c8a3e2167943dc0e739f5f3"} Jul 15 23:45:42.737211 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:45:42.737162 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbws9xh" podStartSLOduration=1.565459714 podStartE2EDuration="1.737148055s" totalImagesPullingTime="171.688341ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 23:45:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 23:45:42.735899325 +0000 UTC m=+558.109357287" watchObservedRunningTime="2026-07-15 23:45:42.737148055 +0000 UTC m=+558.110606018" Jul 15 23:45:51.921346 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:45:51.921318 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbws9xh" Jul 15 23:45:51.921801 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:45:51.921376 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbws9xh" Jul 15 23:45:51.933769 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:45:51.933749 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbws9xh" Jul 15 23:45:52.781019 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:45:52.780992 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbws9xh" Jul 15 23:45:53.386856 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:45:53.386813 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:46:06.220941 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:46:06.220898 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:46:20.224220 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:46:20.224189 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:46:25.014239 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:46:25.014214 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d8f6k_2f64cab3-104a-44f6-b125-ba458f54f058/ovn-acl-logging/0.log" Jul 15 23:46:25.016287 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:46:25.016266 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d8f6k_2f64cab3-104a-44f6-b125-ba458f54f058/ovn-acl-logging/0.log" Jul 15 23:47:09.427006 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:47:09.426959 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-584bd4579f-v6wzp"] Jul 15 23:47:09.430400 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:47:09.430383 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-584bd4579f-v6wzp" Jul 15 23:47:09.436818 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:47:09.436791 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-584bd4579f-v6wzp"] Jul 15 23:47:09.542613 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:47:09.542585 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hmwkj\" (UniqueName: \"kubernetes.io/projected/2038c4d3-a937-4f11-99ee-a43b6705e985-kube-api-access-hmwkj\") pod \"authorino-584bd4579f-v6wzp\" (UID: \"2038c4d3-a937-4f11-99ee-a43b6705e985\") " pod="kuadrant-system/authorino-584bd4579f-v6wzp" Jul 15 23:47:09.542613 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:47:09.542620 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/2038c4d3-a937-4f11-99ee-a43b6705e985-tls-cert\") pod \"authorino-584bd4579f-v6wzp\" (UID: \"2038c4d3-a937-4f11-99ee-a43b6705e985\") " pod="kuadrant-system/authorino-584bd4579f-v6wzp" Jul 15 23:47:09.643584 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:47:09.643551 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmwkj\" (UniqueName: \"kubernetes.io/projected/2038c4d3-a937-4f11-99ee-a43b6705e985-kube-api-access-hmwkj\") pod \"authorino-584bd4579f-v6wzp\" (UID: \"2038c4d3-a937-4f11-99ee-a43b6705e985\") " pod="kuadrant-system/authorino-584bd4579f-v6wzp" Jul 15 23:47:09.643762 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:47:09.643593 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/2038c4d3-a937-4f11-99ee-a43b6705e985-tls-cert\") pod \"authorino-584bd4579f-v6wzp\" (UID: \"2038c4d3-a937-4f11-99ee-a43b6705e985\") " pod="kuadrant-system/authorino-584bd4579f-v6wzp" Jul 15 23:47:09.646032 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:47:09.646008 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/2038c4d3-a937-4f11-99ee-a43b6705e985-tls-cert\") pod \"authorino-584bd4579f-v6wzp\" (UID: \"2038c4d3-a937-4f11-99ee-a43b6705e985\") " pod="kuadrant-system/authorino-584bd4579f-v6wzp" Jul 15 23:47:09.650983 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:47:09.650949 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmwkj\" (UniqueName: \"kubernetes.io/projected/2038c4d3-a937-4f11-99ee-a43b6705e985-kube-api-access-hmwkj\") pod \"authorino-584bd4579f-v6wzp\" (UID: \"2038c4d3-a937-4f11-99ee-a43b6705e985\") " pod="kuadrant-system/authorino-584bd4579f-v6wzp" Jul 15 23:47:09.740145 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:47:09.740118 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-584bd4579f-v6wzp" Jul 15 23:47:09.767660 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:47:09.767632 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:47:09.858825 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:47:09.858796 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:47:09.873164 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:47:09.873140 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-584bd4579f-v6wzp"] Jul 15 23:47:09.875380 ip-10-0-128-197 kubenswrapper[2563]: W0715 23:47:09.875352 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2038c4d3_a937_4f11_99ee_a43b6705e985.slice/crio-fb0d62f9b45036bf05283e568533cf6cf0f6715f97678f7fdeb69a7a7fb6e271 WatchSource:0}: Error finding container fb0d62f9b45036bf05283e568533cf6cf0f6715f97678f7fdeb69a7a7fb6e271: Status 404 returned error can't find the container with id fb0d62f9b45036bf05283e568533cf6cf0f6715f97678f7fdeb69a7a7fb6e271 Jul 15 23:47:10.150883 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:47:10.150800 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-584bd4579f-v6wzp" event={"ID":"2038c4d3-a937-4f11-99ee-a43b6705e985","Type":"ContainerStarted","Data":"fb0d62f9b45036bf05283e568533cf6cf0f6715f97678f7fdeb69a7a7fb6e271"} Jul 15 23:47:10.871103 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:47:10.871062 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:47:10.977540 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:47:10.977506 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:47:11.160520 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:47:11.160428 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-584bd4579f-v6wzp" event={"ID":"2038c4d3-a937-4f11-99ee-a43b6705e985","Type":"ContainerStarted","Data":"7c83437c579db11b85d089a66318879867b860e794604f637bf7b4aa0a159c7c"} Jul 15 23:47:11.184407 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:47:11.184334 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-584bd4579f-v6wzp" podStartSLOduration=1.6820793699999999 podStartE2EDuration="2.184311072s" totalImagesPullingTime="502.231702ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 23:47:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 23:47:11.182872208 +0000 UTC m=+646.556330173" watchObservedRunningTime="2026-07-15 23:47:11.184311072 +0000 UTC m=+646.557769111" Jul 15 23:47:11.227613 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:47:11.227582 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-555cfbc94-5x458"] Jul 15 23:47:11.227857 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:47:11.227803 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-555cfbc94-5x458" podUID="8be49e59-a30d-4230-a602-6964668430bd" containerName="authorino" containerID="cri-o://7040316559b4089bee7a4a667a8ef901ed2cf9b1dc7e441b129cc1e1ecc365de" gracePeriod=30 Jul 15 23:47:11.488287 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:47:11.488259 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-555cfbc94-5x458" Jul 15 23:47:11.562925 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:47:11.562889 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ztcpj\" (UniqueName: \"kubernetes.io/projected/8be49e59-a30d-4230-a602-6964668430bd-kube-api-access-ztcpj\") pod \"8be49e59-a30d-4230-a602-6964668430bd\" (UID: \"8be49e59-a30d-4230-a602-6964668430bd\") " Jul 15 23:47:11.563107 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:47:11.562995 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/8be49e59-a30d-4230-a602-6964668430bd-tls-cert\") pod \"8be49e59-a30d-4230-a602-6964668430bd\" (UID: \"8be49e59-a30d-4230-a602-6964668430bd\") " Jul 15 23:47:11.565016 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:47:11.564961 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8be49e59-a30d-4230-a602-6964668430bd-kube-api-access-ztcpj" (OuterVolumeSpecName: "kube-api-access-ztcpj") pod "8be49e59-a30d-4230-a602-6964668430bd" (UID: "8be49e59-a30d-4230-a602-6964668430bd"). InnerVolumeSpecName "kube-api-access-ztcpj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 23:47:11.573842 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:47:11.573818 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8be49e59-a30d-4230-a602-6964668430bd-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "8be49e59-a30d-4230-a602-6964668430bd" (UID: "8be49e59-a30d-4230-a602-6964668430bd"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 23:47:11.664372 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:47:11.664344 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ztcpj\" (UniqueName: \"kubernetes.io/projected/8be49e59-a30d-4230-a602-6964668430bd-kube-api-access-ztcpj\") on node \"ip-10-0-128-197.ec2.internal\" DevicePath \"\"" Jul 15 23:47:11.664372 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:47:11.664372 2563 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/8be49e59-a30d-4230-a602-6964668430bd-tls-cert\") on node \"ip-10-0-128-197.ec2.internal\" DevicePath \"\"" Jul 15 23:47:12.167235 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:47:12.167203 2563 generic.go:356] "Generic (PLEG): container finished" podID="8be49e59-a30d-4230-a602-6964668430bd" containerID="7040316559b4089bee7a4a667a8ef901ed2cf9b1dc7e441b129cc1e1ecc365de" exitCode=0 Jul 15 23:47:12.167720 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:47:12.167250 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-555cfbc94-5x458" Jul 15 23:47:12.167720 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:47:12.167286 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-555cfbc94-5x458" event={"ID":"8be49e59-a30d-4230-a602-6964668430bd","Type":"ContainerDied","Data":"7040316559b4089bee7a4a667a8ef901ed2cf9b1dc7e441b129cc1e1ecc365de"} Jul 15 23:47:12.167720 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:47:12.167343 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-555cfbc94-5x458" event={"ID":"8be49e59-a30d-4230-a602-6964668430bd","Type":"ContainerDied","Data":"3d74e5ab73d9dbc748ff7379a6c4187c48271a3bd8caf5762b5584aa81173237"} Jul 15 23:47:12.167720 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:47:12.167368 2563 scope.go:117] "RemoveContainer" containerID="7040316559b4089bee7a4a667a8ef901ed2cf9b1dc7e441b129cc1e1ecc365de" Jul 15 23:47:12.176487 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:47:12.176471 2563 scope.go:117] "RemoveContainer" containerID="7040316559b4089bee7a4a667a8ef901ed2cf9b1dc7e441b129cc1e1ecc365de" Jul 15 23:47:12.176756 ip-10-0-128-197 kubenswrapper[2563]: E0715 23:47:12.176737 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7040316559b4089bee7a4a667a8ef901ed2cf9b1dc7e441b129cc1e1ecc365de\": container with ID starting with 7040316559b4089bee7a4a667a8ef901ed2cf9b1dc7e441b129cc1e1ecc365de not found: ID does not exist" containerID="7040316559b4089bee7a4a667a8ef901ed2cf9b1dc7e441b129cc1e1ecc365de" Jul 15 23:47:12.176798 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:47:12.176765 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7040316559b4089bee7a4a667a8ef901ed2cf9b1dc7e441b129cc1e1ecc365de"} err="failed to get container status \"7040316559b4089bee7a4a667a8ef901ed2cf9b1dc7e441b129cc1e1ecc365de\": rpc error: code = NotFound desc = could not find container \"7040316559b4089bee7a4a667a8ef901ed2cf9b1dc7e441b129cc1e1ecc365de\": container with ID starting with 7040316559b4089bee7a4a667a8ef901ed2cf9b1dc7e441b129cc1e1ecc365de not found: ID does not exist" Jul 15 23:47:12.187077 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:47:12.187048 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-555cfbc94-5x458"] Jul 15 23:47:12.188950 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:47:12.188930 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-555cfbc94-5x458"] Jul 15 23:47:13.174990 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:47:13.171898 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8be49e59-a30d-4230-a602-6964668430bd" path="/var/lib/kubelet/pods/8be49e59-a30d-4230-a602-6964668430bd/volumes" Jul 15 23:47:15.640797 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:47:15.640752 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:47:55.571645 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:47:55.571612 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:47:55.626725 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:47:55.626690 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:48:20.610444 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:48:20.610411 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:48:22.501489 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:48:22.501449 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:49:07.163297 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:49:07.163253 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:49:07.261496 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:49:07.261445 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:49:07.385023 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:49:07.384990 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:49:09.505687 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:49:09.505654 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:49:09.553162 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:49:09.553131 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:49:13.187748 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:49:13.187699 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:49:13.256134 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:49:13.256105 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:49:13.325196 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:49:13.325167 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:49:13.512104 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:49:13.512077 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:49:19.162587 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:49:19.162558 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:49:19.222635 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:49:19.222606 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:50:04.648360 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:50:04.648284 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:51:17.038468 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:51:17.038394 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:51:25.048080 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:51:25.048040 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d8f6k_2f64cab3-104a-44f6-b125-ba458f54f058/ovn-acl-logging/0.log" Jul 15 23:51:25.048080 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:51:25.048071 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d8f6k_2f64cab3-104a-44f6-b125-ba458f54f058/ovn-acl-logging/0.log" Jul 15 23:52:20.515354 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:52:20.515319 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:53:15.694726 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:53:15.694641 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:56:25.081019 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:56:25.080993 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d8f6k_2f64cab3-104a-44f6-b125-ba458f54f058/ovn-acl-logging/0.log" Jul 15 23:56:25.082154 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:56:25.082133 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d8f6k_2f64cab3-104a-44f6-b125-ba458f54f058/ovn-acl-logging/0.log" Jul 15 23:57:59.852307 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:57:59.852275 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 15 23:58:01.087826 ip-10-0-128-197 kubenswrapper[2563]: I0715 23:58:01.087794 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:00:05.101422 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:00:05.101383 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:00:12.900854 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:00:12.900786 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:00:21.150536 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:00:21.150502 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:00:21.345607 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:00:21.345570 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:00:26.223219 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:00:26.223179 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:00:29.105382 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:00:29.105353 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:00:29.205539 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:00:29.205430 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:00:29.552738 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:00:29.552701 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:00:35.203659 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:00:35.203626 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:00:35.261332 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:00:35.261267 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:00:35.358302 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:00:35.358269 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-65d40585-openshift-default-847f9cdf69-4vvnr"] Jul 16 00:00:35.359096 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:00:35.359069 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8be49e59-a30d-4230-a602-6964668430bd" containerName="authorino" Jul 16 00:00:35.359205 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:00:35.359098 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="8be49e59-a30d-4230-a602-6964668430bd" containerName="authorino" Jul 16 00:00:35.359266 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:00:35.359246 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="8be49e59-a30d-4230-a602-6964668430bd" containerName="authorino" Jul 16 00:00:35.362658 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:00:35.362638 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-65d40585-openshift-default-847f9cdf69-4vvnr" Jul 16 00:00:35.364856 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:00:35.364831 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-65d40585-openshift-default-dockercfg-9ctzm\"" Jul 16 00:00:35.373456 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:00:35.373434 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-65d40585-openshift-default-847f9cdf69-4vvnr"] Jul 16 00:00:35.392343 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:00:35.392320 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:00:35.426395 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:00:35.426365 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/228b51b2-2b58-4c65-a338-022863017e85-istio-data\") pod \"e2e-ait-65d40585-openshift-default-847f9cdf69-4vvnr\" (UID: \"228b51b2-2b58-4c65-a338-022863017e85\") " pod="openshift-ingress/e2e-ait-65d40585-openshift-default-847f9cdf69-4vvnr" Jul 16 00:00:35.426551 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:00:35.426450 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2btts\" (UniqueName: \"kubernetes.io/projected/228b51b2-2b58-4c65-a338-022863017e85-kube-api-access-2btts\") pod \"e2e-ait-65d40585-openshift-default-847f9cdf69-4vvnr\" (UID: \"228b51b2-2b58-4c65-a338-022863017e85\") " pod="openshift-ingress/e2e-ait-65d40585-openshift-default-847f9cdf69-4vvnr" Jul 16 00:00:35.426551 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:00:35.426486 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/228b51b2-2b58-4c65-a338-022863017e85-istio-envoy\") pod \"e2e-ait-65d40585-openshift-default-847f9cdf69-4vvnr\" (UID: \"228b51b2-2b58-4c65-a338-022863017e85\") " pod="openshift-ingress/e2e-ait-65d40585-openshift-default-847f9cdf69-4vvnr" Jul 16 00:00:35.427227 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:00:35.427197 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/228b51b2-2b58-4c65-a338-022863017e85-credential-socket\") pod \"e2e-ait-65d40585-openshift-default-847f9cdf69-4vvnr\" (UID: \"228b51b2-2b58-4c65-a338-022863017e85\") " pod="openshift-ingress/e2e-ait-65d40585-openshift-default-847f9cdf69-4vvnr" Jul 16 00:00:35.427348 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:00:35.427265 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/228b51b2-2b58-4c65-a338-022863017e85-istio-podinfo\") pod \"e2e-ait-65d40585-openshift-default-847f9cdf69-4vvnr\" (UID: \"228b51b2-2b58-4c65-a338-022863017e85\") " pod="openshift-ingress/e2e-ait-65d40585-openshift-default-847f9cdf69-4vvnr" Jul 16 00:00:35.427348 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:00:35.427289 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/228b51b2-2b58-4c65-a338-022863017e85-istiod-ca-cert\") pod \"e2e-ait-65d40585-openshift-default-847f9cdf69-4vvnr\" (UID: \"228b51b2-2b58-4c65-a338-022863017e85\") " pod="openshift-ingress/e2e-ait-65d40585-openshift-default-847f9cdf69-4vvnr" Jul 16 00:00:35.427348 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:00:35.427308 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/228b51b2-2b58-4c65-a338-022863017e85-workload-certs\") pod \"e2e-ait-65d40585-openshift-default-847f9cdf69-4vvnr\" (UID: \"228b51b2-2b58-4c65-a338-022863017e85\") " pod="openshift-ingress/e2e-ait-65d40585-openshift-default-847f9cdf69-4vvnr" Jul 16 00:00:35.427488 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:00:35.427415 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/228b51b2-2b58-4c65-a338-022863017e85-workload-socket\") pod \"e2e-ait-65d40585-openshift-default-847f9cdf69-4vvnr\" (UID: \"228b51b2-2b58-4c65-a338-022863017e85\") " pod="openshift-ingress/e2e-ait-65d40585-openshift-default-847f9cdf69-4vvnr" Jul 16 00:00:35.427488 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:00:35.427450 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/228b51b2-2b58-4c65-a338-022863017e85-istio-token\") pod \"e2e-ait-65d40585-openshift-default-847f9cdf69-4vvnr\" (UID: \"228b51b2-2b58-4c65-a338-022863017e85\") " pod="openshift-ingress/e2e-ait-65d40585-openshift-default-847f9cdf69-4vvnr" Jul 16 00:00:35.494601 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:00:35.494572 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:00:35.528759 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:00:35.528732 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/228b51b2-2b58-4c65-a338-022863017e85-workload-socket\") pod \"e2e-ait-65d40585-openshift-default-847f9cdf69-4vvnr\" (UID: \"228b51b2-2b58-4c65-a338-022863017e85\") " pod="openshift-ingress/e2e-ait-65d40585-openshift-default-847f9cdf69-4vvnr" Jul 16 00:00:35.528893 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:00:35.528772 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/228b51b2-2b58-4c65-a338-022863017e85-istio-token\") pod \"e2e-ait-65d40585-openshift-default-847f9cdf69-4vvnr\" (UID: \"228b51b2-2b58-4c65-a338-022863017e85\") " pod="openshift-ingress/e2e-ait-65d40585-openshift-default-847f9cdf69-4vvnr" Jul 16 00:00:35.528893 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:00:35.528835 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/228b51b2-2b58-4c65-a338-022863017e85-istio-data\") pod \"e2e-ait-65d40585-openshift-default-847f9cdf69-4vvnr\" (UID: \"228b51b2-2b58-4c65-a338-022863017e85\") " pod="openshift-ingress/e2e-ait-65d40585-openshift-default-847f9cdf69-4vvnr" Jul 16 00:00:35.528893 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:00:35.528864 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2btts\" (UniqueName: \"kubernetes.io/projected/228b51b2-2b58-4c65-a338-022863017e85-kube-api-access-2btts\") pod \"e2e-ait-65d40585-openshift-default-847f9cdf69-4vvnr\" (UID: \"228b51b2-2b58-4c65-a338-022863017e85\") " pod="openshift-ingress/e2e-ait-65d40585-openshift-default-847f9cdf69-4vvnr" Jul 16 00:00:35.529087 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:00:35.528896 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/228b51b2-2b58-4c65-a338-022863017e85-istio-envoy\") pod \"e2e-ait-65d40585-openshift-default-847f9cdf69-4vvnr\" (UID: \"228b51b2-2b58-4c65-a338-022863017e85\") " pod="openshift-ingress/e2e-ait-65d40585-openshift-default-847f9cdf69-4vvnr" Jul 16 00:00:35.529150 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:00:35.529080 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/228b51b2-2b58-4c65-a338-022863017e85-credential-socket\") pod \"e2e-ait-65d40585-openshift-default-847f9cdf69-4vvnr\" (UID: \"228b51b2-2b58-4c65-a338-022863017e85\") " pod="openshift-ingress/e2e-ait-65d40585-openshift-default-847f9cdf69-4vvnr" Jul 16 00:00:35.529150 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:00:35.529140 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/228b51b2-2b58-4c65-a338-022863017e85-istio-podinfo\") pod \"e2e-ait-65d40585-openshift-default-847f9cdf69-4vvnr\" (UID: \"228b51b2-2b58-4c65-a338-022863017e85\") " pod="openshift-ingress/e2e-ait-65d40585-openshift-default-847f9cdf69-4vvnr" Jul 16 00:00:35.529253 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:00:35.529166 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/228b51b2-2b58-4c65-a338-022863017e85-istiod-ca-cert\") pod \"e2e-ait-65d40585-openshift-default-847f9cdf69-4vvnr\" (UID: \"228b51b2-2b58-4c65-a338-022863017e85\") " pod="openshift-ingress/e2e-ait-65d40585-openshift-default-847f9cdf69-4vvnr" Jul 16 00:00:35.529253 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:00:35.529172 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/228b51b2-2b58-4c65-a338-022863017e85-workload-socket\") pod \"e2e-ait-65d40585-openshift-default-847f9cdf69-4vvnr\" (UID: \"228b51b2-2b58-4c65-a338-022863017e85\") " pod="openshift-ingress/e2e-ait-65d40585-openshift-default-847f9cdf69-4vvnr" Jul 16 00:00:35.529253 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:00:35.529196 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/228b51b2-2b58-4c65-a338-022863017e85-workload-certs\") pod \"e2e-ait-65d40585-openshift-default-847f9cdf69-4vvnr\" (UID: \"228b51b2-2b58-4c65-a338-022863017e85\") " pod="openshift-ingress/e2e-ait-65d40585-openshift-default-847f9cdf69-4vvnr" Jul 16 00:00:35.529253 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:00:35.529211 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/228b51b2-2b58-4c65-a338-022863017e85-istio-data\") pod \"e2e-ait-65d40585-openshift-default-847f9cdf69-4vvnr\" (UID: \"228b51b2-2b58-4c65-a338-022863017e85\") " pod="openshift-ingress/e2e-ait-65d40585-openshift-default-847f9cdf69-4vvnr" Jul 16 00:00:35.529459 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:00:35.529421 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/228b51b2-2b58-4c65-a338-022863017e85-workload-certs\") pod \"e2e-ait-65d40585-openshift-default-847f9cdf69-4vvnr\" (UID: \"228b51b2-2b58-4c65-a338-022863017e85\") " pod="openshift-ingress/e2e-ait-65d40585-openshift-default-847f9cdf69-4vvnr" Jul 16 00:00:35.529459 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:00:35.529419 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/228b51b2-2b58-4c65-a338-022863017e85-credential-socket\") pod \"e2e-ait-65d40585-openshift-default-847f9cdf69-4vvnr\" (UID: \"228b51b2-2b58-4c65-a338-022863017e85\") " pod="openshift-ingress/e2e-ait-65d40585-openshift-default-847f9cdf69-4vvnr" Jul 16 00:00:35.529695 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:00:35.529676 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/228b51b2-2b58-4c65-a338-022863017e85-istiod-ca-cert\") pod \"e2e-ait-65d40585-openshift-default-847f9cdf69-4vvnr\" (UID: \"228b51b2-2b58-4c65-a338-022863017e85\") " pod="openshift-ingress/e2e-ait-65d40585-openshift-default-847f9cdf69-4vvnr" Jul 16 00:00:35.531210 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:00:35.531188 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/228b51b2-2b58-4c65-a338-022863017e85-istio-envoy\") pod \"e2e-ait-65d40585-openshift-default-847f9cdf69-4vvnr\" (UID: \"228b51b2-2b58-4c65-a338-022863017e85\") " pod="openshift-ingress/e2e-ait-65d40585-openshift-default-847f9cdf69-4vvnr" Jul 16 00:00:35.531420 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:00:35.531401 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/228b51b2-2b58-4c65-a338-022863017e85-istio-podinfo\") pod \"e2e-ait-65d40585-openshift-default-847f9cdf69-4vvnr\" (UID: \"228b51b2-2b58-4c65-a338-022863017e85\") " pod="openshift-ingress/e2e-ait-65d40585-openshift-default-847f9cdf69-4vvnr" Jul 16 00:00:35.536777 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:00:35.536750 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2btts\" (UniqueName: \"kubernetes.io/projected/228b51b2-2b58-4c65-a338-022863017e85-kube-api-access-2btts\") pod \"e2e-ait-65d40585-openshift-default-847f9cdf69-4vvnr\" (UID: \"228b51b2-2b58-4c65-a338-022863017e85\") " pod="openshift-ingress/e2e-ait-65d40585-openshift-default-847f9cdf69-4vvnr" Jul 16 00:00:35.536855 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:00:35.536757 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/228b51b2-2b58-4c65-a338-022863017e85-istio-token\") pod \"e2e-ait-65d40585-openshift-default-847f9cdf69-4vvnr\" (UID: \"228b51b2-2b58-4c65-a338-022863017e85\") " pod="openshift-ingress/e2e-ait-65d40585-openshift-default-847f9cdf69-4vvnr" Jul 16 00:00:35.598736 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:00:35.598709 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:00:35.675750 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:00:35.675719 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-65d40585-openshift-default-847f9cdf69-4vvnr" Jul 16 00:00:35.771891 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:00:35.771860 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:00:35.820810 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:00:35.820763 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-65d40585-openshift-default-847f9cdf69-4vvnr"] Jul 16 00:00:35.823945 ip-10-0-128-197 kubenswrapper[2563]: W0716 00:00:35.823917 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod228b51b2_2b58_4c65_a338_022863017e85.slice/crio-6c0e224f1a39edba3f1c98b182e464ff1eb4142dfef5adb2b0de354b329d7866 WatchSource:0}: Error finding container 6c0e224f1a39edba3f1c98b182e464ff1eb4142dfef5adb2b0de354b329d7866: Status 404 returned error can't find the container with id 6c0e224f1a39edba3f1c98b182e464ff1eb4142dfef5adb2b0de354b329d7866 Jul 16 00:00:35.827028 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:00:35.826997 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 16 00:00:35.827109 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:00:35.827066 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 16 00:00:35.827109 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:00:35.827097 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 16 00:00:36.155514 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:00:36.155424 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-65d40585-openshift-default-847f9cdf69-4vvnr" event={"ID":"228b51b2-2b58-4c65-a338-022863017e85","Type":"ContainerStarted","Data":"d43072e2f2c27b13122dabb9db5b684ea7d907256d2320857db194659d8ccbb3"} Jul 16 00:00:36.155514 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:00:36.155481 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-65d40585-openshift-default-847f9cdf69-4vvnr" event={"ID":"228b51b2-2b58-4c65-a338-022863017e85","Type":"ContainerStarted","Data":"6c0e224f1a39edba3f1c98b182e464ff1eb4142dfef5adb2b0de354b329d7866"} Jul 16 00:00:36.173339 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:00:36.173295 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-65d40585-openshift-default-847f9cdf69-4vvnr" podStartSLOduration=1.173280089 podStartE2EDuration="1.173280089s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 00:00:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 00:00:36.171870989 +0000 UTC m=+1451.545328953" watchObservedRunningTime="2026-07-16 00:00:36.173280089 +0000 UTC m=+1451.546738052" Jul 16 00:00:36.675861 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:00:36.675830 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-65d40585-openshift-default-847f9cdf69-4vvnr" Jul 16 00:00:36.680604 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:00:36.680578 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-65d40585-openshift-default-847f9cdf69-4vvnr" Jul 16 00:00:36.749033 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:00:36.748949 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:00:36.850179 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:00:36.850135 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:00:37.160745 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:00:37.160717 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-65d40585-openshift-default-847f9cdf69-4vvnr" Jul 16 00:00:37.161559 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:00:37.161541 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-65d40585-openshift-default-847f9cdf69-4vvnr" Jul 16 00:00:39.164935 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:00:39.164904 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:00:41.679279 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:00:41.679239 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-65d40585-openshift-default-847f9cdf69-4vvnr"] Jul 16 00:00:41.679680 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:00:41.679435 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-65d40585-openshift-default-847f9cdf69-4vvnr" podUID="228b51b2-2b58-4c65-a338-022863017e85" containerName="istio-proxy" containerID="cri-o://d43072e2f2c27b13122dabb9db5b684ea7d907256d2320857db194659d8ccbb3" gracePeriod=30 Jul 16 00:00:41.760099 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:00:41.760066 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:00:41.849202 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:00:41.849169 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:00:43.850498 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:00:43.850472 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:00:46.931404 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:00:46.931379 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-65d40585-openshift-default-847f9cdf69-4vvnr" Jul 16 00:00:47.040473 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:00:47.040296 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/228b51b2-2b58-4c65-a338-022863017e85-credential-socket\") pod \"228b51b2-2b58-4c65-a338-022863017e85\" (UID: \"228b51b2-2b58-4c65-a338-022863017e85\") " Jul 16 00:00:47.040473 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:00:47.040359 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/228b51b2-2b58-4c65-a338-022863017e85-workload-certs\") pod \"228b51b2-2b58-4c65-a338-022863017e85\" (UID: \"228b51b2-2b58-4c65-a338-022863017e85\") " Jul 16 00:00:47.040473 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:00:47.040471 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/228b51b2-2b58-4c65-a338-022863017e85-workload-socket\") pod \"228b51b2-2b58-4c65-a338-022863017e85\" (UID: \"228b51b2-2b58-4c65-a338-022863017e85\") " Jul 16 00:00:47.040755 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:00:47.040538 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/228b51b2-2b58-4c65-a338-022863017e85-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "228b51b2-2b58-4c65-a338-022863017e85" (UID: "228b51b2-2b58-4c65-a338-022863017e85"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 00:00:47.040755 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:00:47.040563 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/228b51b2-2b58-4c65-a338-022863017e85-istio-podinfo\") pod \"228b51b2-2b58-4c65-a338-022863017e85\" (UID: \"228b51b2-2b58-4c65-a338-022863017e85\") " Jul 16 00:00:47.040755 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:00:47.040621 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/228b51b2-2b58-4c65-a338-022863017e85-istio-data\") pod \"228b51b2-2b58-4c65-a338-022863017e85\" (UID: \"228b51b2-2b58-4c65-a338-022863017e85\") " Jul 16 00:00:47.040755 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:00:47.040647 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/228b51b2-2b58-4c65-a338-022863017e85-istio-token\") pod \"228b51b2-2b58-4c65-a338-022863017e85\" (UID: \"228b51b2-2b58-4c65-a338-022863017e85\") " Jul 16 00:00:47.040755 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:00:47.040689 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/228b51b2-2b58-4c65-a338-022863017e85-istiod-ca-cert\") pod \"228b51b2-2b58-4c65-a338-022863017e85\" (UID: \"228b51b2-2b58-4c65-a338-022863017e85\") " Jul 16 00:00:47.040755 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:00:47.040742 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/228b51b2-2b58-4c65-a338-022863017e85-istio-envoy\") pod \"228b51b2-2b58-4c65-a338-022863017e85\" (UID: \"228b51b2-2b58-4c65-a338-022863017e85\") " Jul 16 00:00:47.041088 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:00:47.040845 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2btts\" (UniqueName: \"kubernetes.io/projected/228b51b2-2b58-4c65-a338-022863017e85-kube-api-access-2btts\") pod \"228b51b2-2b58-4c65-a338-022863017e85\" (UID: \"228b51b2-2b58-4c65-a338-022863017e85\") " Jul 16 00:00:47.041427 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:00:47.041399 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/228b51b2-2b58-4c65-a338-022863017e85-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "228b51b2-2b58-4c65-a338-022863017e85" (UID: "228b51b2-2b58-4c65-a338-022863017e85"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 00:00:47.041566 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:00:47.041512 2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/228b51b2-2b58-4c65-a338-022863017e85-credential-socket\") on node \"ip-10-0-128-197.ec2.internal\" DevicePath \"\"" Jul 16 00:00:47.041891 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:00:47.041867 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/228b51b2-2b58-4c65-a338-022863017e85-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "228b51b2-2b58-4c65-a338-022863017e85" (UID: "228b51b2-2b58-4c65-a338-022863017e85"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 00:00:47.042428 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:00:47.042402 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/228b51b2-2b58-4c65-a338-022863017e85-istio-data" (OuterVolumeSpecName: "istio-data") pod "228b51b2-2b58-4c65-a338-022863017e85" (UID: "228b51b2-2b58-4c65-a338-022863017e85"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 00:00:47.044541 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:00:47.044511 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/228b51b2-2b58-4c65-a338-022863017e85-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "228b51b2-2b58-4c65-a338-022863017e85" (UID: "228b51b2-2b58-4c65-a338-022863017e85"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 00:00:47.045398 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:00:47.041809 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/228b51b2-2b58-4c65-a338-022863017e85-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "228b51b2-2b58-4c65-a338-022863017e85" (UID: "228b51b2-2b58-4c65-a338-022863017e85"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 00:00:47.045398 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:00:47.044807 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/228b51b2-2b58-4c65-a338-022863017e85-istio-token" (OuterVolumeSpecName: "istio-token") pod "228b51b2-2b58-4c65-a338-022863017e85" (UID: "228b51b2-2b58-4c65-a338-022863017e85"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 00:00:47.045398 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:00:47.044863 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/228b51b2-2b58-4c65-a338-022863017e85-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "228b51b2-2b58-4c65-a338-022863017e85" (UID: "228b51b2-2b58-4c65-a338-022863017e85"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 16 00:00:47.047398 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:00:47.045754 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/228b51b2-2b58-4c65-a338-022863017e85-kube-api-access-2btts" (OuterVolumeSpecName: "kube-api-access-2btts") pod "228b51b2-2b58-4c65-a338-022863017e85" (UID: "228b51b2-2b58-4c65-a338-022863017e85"). InnerVolumeSpecName "kube-api-access-2btts". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 00:00:47.142838 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:00:47.142806 2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/228b51b2-2b58-4c65-a338-022863017e85-istio-envoy\") on node \"ip-10-0-128-197.ec2.internal\" DevicePath \"\"" Jul 16 00:00:47.142838 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:00:47.142833 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2btts\" (UniqueName: \"kubernetes.io/projected/228b51b2-2b58-4c65-a338-022863017e85-kube-api-access-2btts\") on node \"ip-10-0-128-197.ec2.internal\" DevicePath \"\"" Jul 16 00:00:47.142838 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:00:47.142841 2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/228b51b2-2b58-4c65-a338-022863017e85-workload-certs\") on node \"ip-10-0-128-197.ec2.internal\" DevicePath \"\"" Jul 16 00:00:47.143075 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:00:47.142850 2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/228b51b2-2b58-4c65-a338-022863017e85-workload-socket\") on node \"ip-10-0-128-197.ec2.internal\" DevicePath \"\"" Jul 16 00:00:47.143075 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:00:47.142858 2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/228b51b2-2b58-4c65-a338-022863017e85-istio-podinfo\") on node \"ip-10-0-128-197.ec2.internal\" DevicePath \"\"" Jul 16 00:00:47.143075 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:00:47.142866 2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/228b51b2-2b58-4c65-a338-022863017e85-istio-data\") on node \"ip-10-0-128-197.ec2.internal\" DevicePath \"\"" Jul 16 00:00:47.143075 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:00:47.142874 2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/228b51b2-2b58-4c65-a338-022863017e85-istio-token\") on node \"ip-10-0-128-197.ec2.internal\" DevicePath \"\"" Jul 16 00:00:47.143075 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:00:47.142881 2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/228b51b2-2b58-4c65-a338-022863017e85-istiod-ca-cert\") on node \"ip-10-0-128-197.ec2.internal\" DevicePath \"\"" Jul 16 00:00:47.207746 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:00:47.207719 2563 generic.go:356] "Generic (PLEG): container finished" podID="228b51b2-2b58-4c65-a338-022863017e85" containerID="d43072e2f2c27b13122dabb9db5b684ea7d907256d2320857db194659d8ccbb3" exitCode=0 Jul 16 00:00:47.207901 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:00:47.207775 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-65d40585-openshift-default-847f9cdf69-4vvnr" Jul 16 00:00:47.207901 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:00:47.207798 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-65d40585-openshift-default-847f9cdf69-4vvnr" event={"ID":"228b51b2-2b58-4c65-a338-022863017e85","Type":"ContainerDied","Data":"d43072e2f2c27b13122dabb9db5b684ea7d907256d2320857db194659d8ccbb3"} Jul 16 00:00:47.207901 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:00:47.207859 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-65d40585-openshift-default-847f9cdf69-4vvnr" event={"ID":"228b51b2-2b58-4c65-a338-022863017e85","Type":"ContainerDied","Data":"6c0e224f1a39edba3f1c98b182e464ff1eb4142dfef5adb2b0de354b329d7866"} Jul 16 00:00:47.207901 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:00:47.207888 2563 scope.go:117] "RemoveContainer" containerID="d43072e2f2c27b13122dabb9db5b684ea7d907256d2320857db194659d8ccbb3" Jul 16 00:00:47.217155 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:00:47.217107 2563 scope.go:117] "RemoveContainer" containerID="d43072e2f2c27b13122dabb9db5b684ea7d907256d2320857db194659d8ccbb3" Jul 16 00:00:47.217387 ip-10-0-128-197 kubenswrapper[2563]: E0716 00:00:47.217369 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d43072e2f2c27b13122dabb9db5b684ea7d907256d2320857db194659d8ccbb3\": container with ID starting with d43072e2f2c27b13122dabb9db5b684ea7d907256d2320857db194659d8ccbb3 not found: ID does not exist" containerID="d43072e2f2c27b13122dabb9db5b684ea7d907256d2320857db194659d8ccbb3" Jul 16 00:00:47.217451 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:00:47.217395 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d43072e2f2c27b13122dabb9db5b684ea7d907256d2320857db194659d8ccbb3"} err="failed to get container status \"d43072e2f2c27b13122dabb9db5b684ea7d907256d2320857db194659d8ccbb3\": rpc error: code = NotFound desc = could not find container \"d43072e2f2c27b13122dabb9db5b684ea7d907256d2320857db194659d8ccbb3\": container with ID starting with d43072e2f2c27b13122dabb9db5b684ea7d907256d2320857db194659d8ccbb3 not found: ID does not exist" Jul 16 00:00:47.224441 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:00:47.224416 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-65d40585-openshift-default-847f9cdf69-4vvnr"] Jul 16 00:00:47.229112 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:00:47.229089 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-65d40585-openshift-default-847f9cdf69-4vvnr"] Jul 16 00:00:47.792243 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:00:47.792210 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:00:47.891723 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:00:47.891688 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:00:47.993548 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:00:47.993523 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:00:48.041859 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:00:48.041825 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-55cbdb16-gw-openshift-default-d46ff9dd7-2qbpc"] Jul 16 00:00:48.042452 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:00:48.042397 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="228b51b2-2b58-4c65-a338-022863017e85" containerName="istio-proxy" Jul 16 00:00:48.042452 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:00:48.042418 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="228b51b2-2b58-4c65-a338-022863017e85" containerName="istio-proxy" Jul 16 00:00:48.042595 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:00:48.042507 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="228b51b2-2b58-4c65-a338-022863017e85" containerName="istio-proxy" Jul 16 00:00:48.045651 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:00:48.045628 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-55cbdb16-gw-openshift-default-d46ff9dd7-2qbpc" Jul 16 00:00:48.047980 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:00:48.047946 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-55cbdb16-gw-openshift-default-dockercfg-ztx9k\"" Jul 16 00:00:48.059391 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:00:48.059366 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-55cbdb16-gw-openshift-default-d46ff9dd7-2qbpc"] Jul 16 00:00:48.081213 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:00:48.081184 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:00:48.151911 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:00:48.151875 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb7ce561-2cfa-4232-b6cf-1786997379a0-istio-token\") pod \"e2e-migrate-55cbdb16-gw-openshift-default-d46ff9dd7-2qbpc\" (UID: \"fb7ce561-2cfa-4232-b6cf-1786997379a0\") " pod="openshift-ingress/e2e-migrate-55cbdb16-gw-openshift-default-d46ff9dd7-2qbpc" Jul 16 00:00:48.152090 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:00:48.151960 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb7ce561-2cfa-4232-b6cf-1786997379a0-istio-data\") pod \"e2e-migrate-55cbdb16-gw-openshift-default-d46ff9dd7-2qbpc\" (UID: \"fb7ce561-2cfa-4232-b6cf-1786997379a0\") " pod="openshift-ingress/e2e-migrate-55cbdb16-gw-openshift-default-d46ff9dd7-2qbpc" Jul 16 00:00:48.152090 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:00:48.152022 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb7ce561-2cfa-4232-b6cf-1786997379a0-workload-certs\") pod \"e2e-migrate-55cbdb16-gw-openshift-default-d46ff9dd7-2qbpc\" (UID: \"fb7ce561-2cfa-4232-b6cf-1786997379a0\") " pod="openshift-ingress/e2e-migrate-55cbdb16-gw-openshift-default-d46ff9dd7-2qbpc" Jul 16 00:00:48.152203 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:00:48.152101 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb7ce561-2cfa-4232-b6cf-1786997379a0-istio-podinfo\") pod \"e2e-migrate-55cbdb16-gw-openshift-default-d46ff9dd7-2qbpc\" (UID: \"fb7ce561-2cfa-4232-b6cf-1786997379a0\") " pod="openshift-ingress/e2e-migrate-55cbdb16-gw-openshift-default-d46ff9dd7-2qbpc" Jul 16 00:00:48.152203 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:00:48.152148 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb7ce561-2cfa-4232-b6cf-1786997379a0-istiod-ca-cert\") pod \"e2e-migrate-55cbdb16-gw-openshift-default-d46ff9dd7-2qbpc\" (UID: \"fb7ce561-2cfa-4232-b6cf-1786997379a0\") " pod="openshift-ingress/e2e-migrate-55cbdb16-gw-openshift-default-d46ff9dd7-2qbpc" Jul 16 00:00:48.152305 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:00:48.152200 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb7ce561-2cfa-4232-b6cf-1786997379a0-workload-socket\") pod \"e2e-migrate-55cbdb16-gw-openshift-default-d46ff9dd7-2qbpc\" (UID: \"fb7ce561-2cfa-4232-b6cf-1786997379a0\") " pod="openshift-ingress/e2e-migrate-55cbdb16-gw-openshift-default-d46ff9dd7-2qbpc" Jul 16 00:00:48.152305 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:00:48.152253 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rbjc7\" (UniqueName: \"kubernetes.io/projected/fb7ce561-2cfa-4232-b6cf-1786997379a0-kube-api-access-rbjc7\") pod \"e2e-migrate-55cbdb16-gw-openshift-default-d46ff9dd7-2qbpc\" (UID: \"fb7ce561-2cfa-4232-b6cf-1786997379a0\") " pod="openshift-ingress/e2e-migrate-55cbdb16-gw-openshift-default-d46ff9dd7-2qbpc" Jul 16 00:00:48.152403 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:00:48.152309 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb7ce561-2cfa-4232-b6cf-1786997379a0-istio-envoy\") pod \"e2e-migrate-55cbdb16-gw-openshift-default-d46ff9dd7-2qbpc\" (UID: \"fb7ce561-2cfa-4232-b6cf-1786997379a0\") " pod="openshift-ingress/e2e-migrate-55cbdb16-gw-openshift-default-d46ff9dd7-2qbpc" Jul 16 00:00:48.152403 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:00:48.152356 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb7ce561-2cfa-4232-b6cf-1786997379a0-credential-socket\") pod \"e2e-migrate-55cbdb16-gw-openshift-default-d46ff9dd7-2qbpc\" (UID: \"fb7ce561-2cfa-4232-b6cf-1786997379a0\") " pod="openshift-ingress/e2e-migrate-55cbdb16-gw-openshift-default-d46ff9dd7-2qbpc" Jul 16 00:00:48.152899 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:00:48.152880 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:00:48.252859 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:00:48.252833 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb7ce561-2cfa-4232-b6cf-1786997379a0-istio-token\") pod \"e2e-migrate-55cbdb16-gw-openshift-default-d46ff9dd7-2qbpc\" (UID: \"fb7ce561-2cfa-4232-b6cf-1786997379a0\") " pod="openshift-ingress/e2e-migrate-55cbdb16-gw-openshift-default-d46ff9dd7-2qbpc" Jul 16 00:00:48.253024 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:00:48.252869 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb7ce561-2cfa-4232-b6cf-1786997379a0-istio-data\") pod \"e2e-migrate-55cbdb16-gw-openshift-default-d46ff9dd7-2qbpc\" (UID: \"fb7ce561-2cfa-4232-b6cf-1786997379a0\") " pod="openshift-ingress/e2e-migrate-55cbdb16-gw-openshift-default-d46ff9dd7-2qbpc" Jul 16 00:00:48.253024 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:00:48.252892 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb7ce561-2cfa-4232-b6cf-1786997379a0-workload-certs\") pod \"e2e-migrate-55cbdb16-gw-openshift-default-d46ff9dd7-2qbpc\" (UID: \"fb7ce561-2cfa-4232-b6cf-1786997379a0\") " pod="openshift-ingress/e2e-migrate-55cbdb16-gw-openshift-default-d46ff9dd7-2qbpc" Jul 16 00:00:48.253108 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:00:48.253017 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb7ce561-2cfa-4232-b6cf-1786997379a0-istio-podinfo\") pod \"e2e-migrate-55cbdb16-gw-openshift-default-d46ff9dd7-2qbpc\" (UID: \"fb7ce561-2cfa-4232-b6cf-1786997379a0\") " pod="openshift-ingress/e2e-migrate-55cbdb16-gw-openshift-default-d46ff9dd7-2qbpc" Jul 16 00:00:48.253108 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:00:48.253064 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb7ce561-2cfa-4232-b6cf-1786997379a0-istiod-ca-cert\") pod \"e2e-migrate-55cbdb16-gw-openshift-default-d46ff9dd7-2qbpc\" (UID: \"fb7ce561-2cfa-4232-b6cf-1786997379a0\") " pod="openshift-ingress/e2e-migrate-55cbdb16-gw-openshift-default-d46ff9dd7-2qbpc" Jul 16 00:00:48.253181 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:00:48.253111 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb7ce561-2cfa-4232-b6cf-1786997379a0-workload-socket\") pod \"e2e-migrate-55cbdb16-gw-openshift-default-d46ff9dd7-2qbpc\" (UID: \"fb7ce561-2cfa-4232-b6cf-1786997379a0\") " pod="openshift-ingress/e2e-migrate-55cbdb16-gw-openshift-default-d46ff9dd7-2qbpc" Jul 16 00:00:48.253181 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:00:48.253141 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbjc7\" (UniqueName: \"kubernetes.io/projected/fb7ce561-2cfa-4232-b6cf-1786997379a0-kube-api-access-rbjc7\") pod \"e2e-migrate-55cbdb16-gw-openshift-default-d46ff9dd7-2qbpc\" (UID: \"fb7ce561-2cfa-4232-b6cf-1786997379a0\") " pod="openshift-ingress/e2e-migrate-55cbdb16-gw-openshift-default-d46ff9dd7-2qbpc" Jul 16 00:00:48.253280 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:00:48.253182 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb7ce561-2cfa-4232-b6cf-1786997379a0-istio-envoy\") pod \"e2e-migrate-55cbdb16-gw-openshift-default-d46ff9dd7-2qbpc\" (UID: \"fb7ce561-2cfa-4232-b6cf-1786997379a0\") " pod="openshift-ingress/e2e-migrate-55cbdb16-gw-openshift-default-d46ff9dd7-2qbpc" Jul 16 00:00:48.253280 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:00:48.253232 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb7ce561-2cfa-4232-b6cf-1786997379a0-credential-socket\") pod \"e2e-migrate-55cbdb16-gw-openshift-default-d46ff9dd7-2qbpc\" (UID: \"fb7ce561-2cfa-4232-b6cf-1786997379a0\") " pod="openshift-ingress/e2e-migrate-55cbdb16-gw-openshift-default-d46ff9dd7-2qbpc" Jul 16 00:00:48.253280 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:00:48.253242 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb7ce561-2cfa-4232-b6cf-1786997379a0-workload-certs\") pod \"e2e-migrate-55cbdb16-gw-openshift-default-d46ff9dd7-2qbpc\" (UID: \"fb7ce561-2cfa-4232-b6cf-1786997379a0\") " pod="openshift-ingress/e2e-migrate-55cbdb16-gw-openshift-default-d46ff9dd7-2qbpc" Jul 16 00:00:48.253440 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:00:48.253294 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb7ce561-2cfa-4232-b6cf-1786997379a0-istio-data\") pod \"e2e-migrate-55cbdb16-gw-openshift-default-d46ff9dd7-2qbpc\" (UID: \"fb7ce561-2cfa-4232-b6cf-1786997379a0\") " pod="openshift-ingress/e2e-migrate-55cbdb16-gw-openshift-default-d46ff9dd7-2qbpc" Jul 16 00:00:48.253529 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:00:48.253507 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb7ce561-2cfa-4232-b6cf-1786997379a0-workload-socket\") pod \"e2e-migrate-55cbdb16-gw-openshift-default-d46ff9dd7-2qbpc\" (UID: \"fb7ce561-2cfa-4232-b6cf-1786997379a0\") " pod="openshift-ingress/e2e-migrate-55cbdb16-gw-openshift-default-d46ff9dd7-2qbpc" Jul 16 00:00:48.253583 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:00:48.253543 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb7ce561-2cfa-4232-b6cf-1786997379a0-credential-socket\") pod \"e2e-migrate-55cbdb16-gw-openshift-default-d46ff9dd7-2qbpc\" (UID: \"fb7ce561-2cfa-4232-b6cf-1786997379a0\") " pod="openshift-ingress/e2e-migrate-55cbdb16-gw-openshift-default-d46ff9dd7-2qbpc" Jul 16 00:00:48.253818 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:00:48.253800 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb7ce561-2cfa-4232-b6cf-1786997379a0-istiod-ca-cert\") pod \"e2e-migrate-55cbdb16-gw-openshift-default-d46ff9dd7-2qbpc\" (UID: \"fb7ce561-2cfa-4232-b6cf-1786997379a0\") " pod="openshift-ingress/e2e-migrate-55cbdb16-gw-openshift-default-d46ff9dd7-2qbpc" Jul 16 00:00:48.255288 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:00:48.255256 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb7ce561-2cfa-4232-b6cf-1786997379a0-istio-envoy\") pod \"e2e-migrate-55cbdb16-gw-openshift-default-d46ff9dd7-2qbpc\" (UID: \"fb7ce561-2cfa-4232-b6cf-1786997379a0\") " pod="openshift-ingress/e2e-migrate-55cbdb16-gw-openshift-default-d46ff9dd7-2qbpc" Jul 16 00:00:48.255594 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:00:48.255575 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb7ce561-2cfa-4232-b6cf-1786997379a0-istio-podinfo\") pod \"e2e-migrate-55cbdb16-gw-openshift-default-d46ff9dd7-2qbpc\" (UID: \"fb7ce561-2cfa-4232-b6cf-1786997379a0\") " pod="openshift-ingress/e2e-migrate-55cbdb16-gw-openshift-default-d46ff9dd7-2qbpc" Jul 16 00:00:48.261239 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:00:48.261213 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb7ce561-2cfa-4232-b6cf-1786997379a0-istio-token\") pod \"e2e-migrate-55cbdb16-gw-openshift-default-d46ff9dd7-2qbpc\" (UID: \"fb7ce561-2cfa-4232-b6cf-1786997379a0\") " pod="openshift-ingress/e2e-migrate-55cbdb16-gw-openshift-default-d46ff9dd7-2qbpc" Jul 16 00:00:48.261342 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:00:48.261322 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbjc7\" (UniqueName: \"kubernetes.io/projected/fb7ce561-2cfa-4232-b6cf-1786997379a0-kube-api-access-rbjc7\") pod \"e2e-migrate-55cbdb16-gw-openshift-default-d46ff9dd7-2qbpc\" (UID: \"fb7ce561-2cfa-4232-b6cf-1786997379a0\") " pod="openshift-ingress/e2e-migrate-55cbdb16-gw-openshift-default-d46ff9dd7-2qbpc" Jul 16 00:00:48.296237 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:00:48.296183 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:00:48.357062 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:00:48.357039 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-55cbdb16-gw-openshift-default-d46ff9dd7-2qbpc" Jul 16 00:00:48.491397 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:00:48.491374 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-55cbdb16-gw-openshift-default-d46ff9dd7-2qbpc"] Jul 16 00:00:48.493775 ip-10-0-128-197 kubenswrapper[2563]: W0716 00:00:48.493750 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb7ce561_2cfa_4232_b6cf_1786997379a0.slice/crio-8d4188f98bc23b0e8c40c424e6fdd49a17ecc3964353d1e52b6df715612f166e WatchSource:0}: Error finding container 8d4188f98bc23b0e8c40c424e6fdd49a17ecc3964353d1e52b6df715612f166e: Status 404 returned error can't find the container with id 8d4188f98bc23b0e8c40c424e6fdd49a17ecc3964353d1e52b6df715612f166e Jul 16 00:00:48.496371 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:00:48.496335 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 16 00:00:48.496501 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:00:48.496411 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 16 00:00:48.496501 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:00:48.496464 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 16 00:00:48.549665 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:00:48.549588 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:00:48.740742 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:00:48.740715 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:00:49.139806 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:00:49.139773 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:00:49.170164 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:00:49.170131 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="228b51b2-2b58-4c65-a338-022863017e85" path="/var/lib/kubelet/pods/228b51b2-2b58-4c65-a338-022863017e85/volumes" Jul 16 00:00:49.219323 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:00:49.219288 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-55cbdb16-gw-openshift-default-d46ff9dd7-2qbpc" event={"ID":"fb7ce561-2cfa-4232-b6cf-1786997379a0","Type":"ContainerStarted","Data":"9a3acda20b426b0c9fd69aa90203d2020a0a26ecc4ef2a870a9c7b93e0ea6df1"} Jul 16 00:00:49.219448 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:00:49.219347 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-55cbdb16-gw-openshift-default-d46ff9dd7-2qbpc" event={"ID":"fb7ce561-2cfa-4232-b6cf-1786997379a0","Type":"ContainerStarted","Data":"8d4188f98bc23b0e8c40c424e6fdd49a17ecc3964353d1e52b6df715612f166e"} Jul 16 00:00:49.240515 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:00:49.240486 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:00:49.240695 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:00:49.240654 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-55cbdb16-gw-openshift-default-d46ff9dd7-2qbpc" podStartSLOduration=1.2406412900000001 podStartE2EDuration="1.24064129s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 00:00:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 00:00:49.237786518 +0000 UTC m=+1464.611244500" watchObservedRunningTime="2026-07-16 00:00:49.24064129 +0000 UTC m=+1464.614099253" Jul 16 00:00:49.357851 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:00:49.357821 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-55cbdb16-gw-openshift-default-d46ff9dd7-2qbpc" Jul 16 00:00:49.362791 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:00:49.362766 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-55cbdb16-gw-openshift-default-d46ff9dd7-2qbpc" Jul 16 00:00:50.224283 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:00:50.224248 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-55cbdb16-gw-openshift-default-d46ff9dd7-2qbpc" Jul 16 00:00:50.225404 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:00:50.225385 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-55cbdb16-gw-openshift-default-d46ff9dd7-2qbpc" Jul 16 00:00:51.373565 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:00:51.373540 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:00:53.910702 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:00:53.910657 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-55cbdb16-gw-openshift-default-d46ff9dd7-2qbpc"] Jul 16 00:00:53.911844 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:00:53.911808 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-55cbdb16-gw-openshift-default-d46ff9dd7-2qbpc" podUID="fb7ce561-2cfa-4232-b6cf-1786997379a0" containerName="istio-proxy" containerID="cri-o://9a3acda20b426b0c9fd69aa90203d2020a0a26ecc4ef2a870a9c7b93e0ea6df1" gracePeriod=30 Jul 16 00:00:53.946080 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:00:53.946050 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:00:54.423510 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:00:54.423475 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:00:56.061167 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:00:56.061130 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:00:59.583699 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:00:59.583665 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:00:59.644377 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:00:59.644345 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:00:59.745062 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:00:59.744957 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-5a03fa09-openshift-default-c77997b9f-lhbsz"] Jul 16 00:00:59.759646 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:00:59.748197 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-5a03fa09-openshift-default-c77997b9f-lhbsz" Jul 16 00:00:59.759646 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:00:59.750599 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-5a03fa09-openshift-default-dockercfg-qk4hx\"" Jul 16 00:00:59.764285 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:00:59.764260 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-5a03fa09-openshift-default-c77997b9f-lhbsz"] Jul 16 00:00:59.776836 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:00:59.776813 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/596c881b-ea8c-45c5-b115-ee790a485d95-istio-podinfo\") pod \"e2e-derive-5a03fa09-openshift-default-c77997b9f-lhbsz\" (UID: \"596c881b-ea8c-45c5-b115-ee790a485d95\") " pod="openshift-ingress/e2e-derive-5a03fa09-openshift-default-c77997b9f-lhbsz" Jul 16 00:00:59.776957 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:00:59.776853 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/596c881b-ea8c-45c5-b115-ee790a485d95-istio-data\") pod \"e2e-derive-5a03fa09-openshift-default-c77997b9f-lhbsz\" (UID: \"596c881b-ea8c-45c5-b115-ee790a485d95\") " pod="openshift-ingress/e2e-derive-5a03fa09-openshift-default-c77997b9f-lhbsz" Jul 16 00:00:59.776957 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:00:59.776900 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/596c881b-ea8c-45c5-b115-ee790a485d95-credential-socket\") pod \"e2e-derive-5a03fa09-openshift-default-c77997b9f-lhbsz\" (UID: \"596c881b-ea8c-45c5-b115-ee790a485d95\") " pod="openshift-ingress/e2e-derive-5a03fa09-openshift-default-c77997b9f-lhbsz" Jul 16 00:00:59.776957 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:00:59.776937 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/596c881b-ea8c-45c5-b115-ee790a485d95-istio-token\") pod \"e2e-derive-5a03fa09-openshift-default-c77997b9f-lhbsz\" (UID: \"596c881b-ea8c-45c5-b115-ee790a485d95\") " pod="openshift-ingress/e2e-derive-5a03fa09-openshift-default-c77997b9f-lhbsz" Jul 16 00:00:59.777142 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:00:59.776997 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/596c881b-ea8c-45c5-b115-ee790a485d95-workload-socket\") pod \"e2e-derive-5a03fa09-openshift-default-c77997b9f-lhbsz\" (UID: \"596c881b-ea8c-45c5-b115-ee790a485d95\") " pod="openshift-ingress/e2e-derive-5a03fa09-openshift-default-c77997b9f-lhbsz" Jul 16 00:00:59.777142 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:00:59.777048 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dr557\" (UniqueName: \"kubernetes.io/projected/596c881b-ea8c-45c5-b115-ee790a485d95-kube-api-access-dr557\") pod \"e2e-derive-5a03fa09-openshift-default-c77997b9f-lhbsz\" (UID: \"596c881b-ea8c-45c5-b115-ee790a485d95\") " pod="openshift-ingress/e2e-derive-5a03fa09-openshift-default-c77997b9f-lhbsz" Jul 16 00:00:59.777142 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:00:59.777091 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/596c881b-ea8c-45c5-b115-ee790a485d95-workload-certs\") pod \"e2e-derive-5a03fa09-openshift-default-c77997b9f-lhbsz\" (UID: \"596c881b-ea8c-45c5-b115-ee790a485d95\") " pod="openshift-ingress/e2e-derive-5a03fa09-openshift-default-c77997b9f-lhbsz" Jul 16 00:00:59.777304 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:00:59.777217 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/596c881b-ea8c-45c5-b115-ee790a485d95-istiod-ca-cert\") pod \"e2e-derive-5a03fa09-openshift-default-c77997b9f-lhbsz\" (UID: \"596c881b-ea8c-45c5-b115-ee790a485d95\") " pod="openshift-ingress/e2e-derive-5a03fa09-openshift-default-c77997b9f-lhbsz" Jul 16 00:00:59.777304 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:00:59.777244 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/596c881b-ea8c-45c5-b115-ee790a485d95-istio-envoy\") pod \"e2e-derive-5a03fa09-openshift-default-c77997b9f-lhbsz\" (UID: \"596c881b-ea8c-45c5-b115-ee790a485d95\") " pod="openshift-ingress/e2e-derive-5a03fa09-openshift-default-c77997b9f-lhbsz" Jul 16 00:00:59.787089 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:00:59.787064 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:00:59.864814 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:00:59.864787 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:00:59.878136 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:00:59.878113 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/596c881b-ea8c-45c5-b115-ee790a485d95-credential-socket\") pod \"e2e-derive-5a03fa09-openshift-default-c77997b9f-lhbsz\" (UID: \"596c881b-ea8c-45c5-b115-ee790a485d95\") " pod="openshift-ingress/e2e-derive-5a03fa09-openshift-default-c77997b9f-lhbsz" Jul 16 00:00:59.878216 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:00:59.878149 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/596c881b-ea8c-45c5-b115-ee790a485d95-istio-token\") pod \"e2e-derive-5a03fa09-openshift-default-c77997b9f-lhbsz\" (UID: \"596c881b-ea8c-45c5-b115-ee790a485d95\") " pod="openshift-ingress/e2e-derive-5a03fa09-openshift-default-c77997b9f-lhbsz" Jul 16 00:00:59.878216 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:00:59.878169 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/596c881b-ea8c-45c5-b115-ee790a485d95-workload-socket\") pod \"e2e-derive-5a03fa09-openshift-default-c77997b9f-lhbsz\" (UID: \"596c881b-ea8c-45c5-b115-ee790a485d95\") " pod="openshift-ingress/e2e-derive-5a03fa09-openshift-default-c77997b9f-lhbsz" Jul 16 00:00:59.878216 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:00:59.878198 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dr557\" (UniqueName: \"kubernetes.io/projected/596c881b-ea8c-45c5-b115-ee790a485d95-kube-api-access-dr557\") pod \"e2e-derive-5a03fa09-openshift-default-c77997b9f-lhbsz\" (UID: \"596c881b-ea8c-45c5-b115-ee790a485d95\") " pod="openshift-ingress/e2e-derive-5a03fa09-openshift-default-c77997b9f-lhbsz" Jul 16 00:00:59.878339 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:00:59.878231 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/596c881b-ea8c-45c5-b115-ee790a485d95-workload-certs\") pod \"e2e-derive-5a03fa09-openshift-default-c77997b9f-lhbsz\" (UID: \"596c881b-ea8c-45c5-b115-ee790a485d95\") " pod="openshift-ingress/e2e-derive-5a03fa09-openshift-default-c77997b9f-lhbsz" Jul 16 00:00:59.878339 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:00:59.878312 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/596c881b-ea8c-45c5-b115-ee790a485d95-istiod-ca-cert\") pod \"e2e-derive-5a03fa09-openshift-default-c77997b9f-lhbsz\" (UID: \"596c881b-ea8c-45c5-b115-ee790a485d95\") " pod="openshift-ingress/e2e-derive-5a03fa09-openshift-default-c77997b9f-lhbsz" Jul 16 00:00:59.878446 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:00:59.878337 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/596c881b-ea8c-45c5-b115-ee790a485d95-istio-envoy\") pod \"e2e-derive-5a03fa09-openshift-default-c77997b9f-lhbsz\" (UID: \"596c881b-ea8c-45c5-b115-ee790a485d95\") " pod="openshift-ingress/e2e-derive-5a03fa09-openshift-default-c77997b9f-lhbsz" Jul 16 00:00:59.878446 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:00:59.878376 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/596c881b-ea8c-45c5-b115-ee790a485d95-istio-podinfo\") pod \"e2e-derive-5a03fa09-openshift-default-c77997b9f-lhbsz\" (UID: \"596c881b-ea8c-45c5-b115-ee790a485d95\") " pod="openshift-ingress/e2e-derive-5a03fa09-openshift-default-c77997b9f-lhbsz" Jul 16 00:00:59.878446 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:00:59.878423 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/596c881b-ea8c-45c5-b115-ee790a485d95-istio-data\") pod \"e2e-derive-5a03fa09-openshift-default-c77997b9f-lhbsz\" (UID: \"596c881b-ea8c-45c5-b115-ee790a485d95\") " pod="openshift-ingress/e2e-derive-5a03fa09-openshift-default-c77997b9f-lhbsz" Jul 16 00:00:59.878639 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:00:59.878524 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/596c881b-ea8c-45c5-b115-ee790a485d95-credential-socket\") pod \"e2e-derive-5a03fa09-openshift-default-c77997b9f-lhbsz\" (UID: \"596c881b-ea8c-45c5-b115-ee790a485d95\") " pod="openshift-ingress/e2e-derive-5a03fa09-openshift-default-c77997b9f-lhbsz" Jul 16 00:00:59.878639 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:00:59.878546 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/596c881b-ea8c-45c5-b115-ee790a485d95-workload-socket\") pod \"e2e-derive-5a03fa09-openshift-default-c77997b9f-lhbsz\" (UID: \"596c881b-ea8c-45c5-b115-ee790a485d95\") " pod="openshift-ingress/e2e-derive-5a03fa09-openshift-default-c77997b9f-lhbsz" Jul 16 00:00:59.878639 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:00:59.878620 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/596c881b-ea8c-45c5-b115-ee790a485d95-workload-certs\") pod \"e2e-derive-5a03fa09-openshift-default-c77997b9f-lhbsz\" (UID: \"596c881b-ea8c-45c5-b115-ee790a485d95\") " pod="openshift-ingress/e2e-derive-5a03fa09-openshift-default-c77997b9f-lhbsz" Jul 16 00:00:59.878745 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:00:59.878717 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/596c881b-ea8c-45c5-b115-ee790a485d95-istio-data\") pod \"e2e-derive-5a03fa09-openshift-default-c77997b9f-lhbsz\" (UID: \"596c881b-ea8c-45c5-b115-ee790a485d95\") " pod="openshift-ingress/e2e-derive-5a03fa09-openshift-default-c77997b9f-lhbsz" Jul 16 00:00:59.878920 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:00:59.878889 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/596c881b-ea8c-45c5-b115-ee790a485d95-istiod-ca-cert\") pod \"e2e-derive-5a03fa09-openshift-default-c77997b9f-lhbsz\" (UID: \"596c881b-ea8c-45c5-b115-ee790a485d95\") " pod="openshift-ingress/e2e-derive-5a03fa09-openshift-default-c77997b9f-lhbsz" Jul 16 00:00:59.880602 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:00:59.880577 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/596c881b-ea8c-45c5-b115-ee790a485d95-istio-envoy\") pod \"e2e-derive-5a03fa09-openshift-default-c77997b9f-lhbsz\" (UID: \"596c881b-ea8c-45c5-b115-ee790a485d95\") " pod="openshift-ingress/e2e-derive-5a03fa09-openshift-default-c77997b9f-lhbsz" Jul 16 00:00:59.880687 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:00:59.880670 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/596c881b-ea8c-45c5-b115-ee790a485d95-istio-podinfo\") pod \"e2e-derive-5a03fa09-openshift-default-c77997b9f-lhbsz\" (UID: \"596c881b-ea8c-45c5-b115-ee790a485d95\") " pod="openshift-ingress/e2e-derive-5a03fa09-openshift-default-c77997b9f-lhbsz" Jul 16 00:00:59.885567 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:00:59.885543 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dr557\" (UniqueName: \"kubernetes.io/projected/596c881b-ea8c-45c5-b115-ee790a485d95-kube-api-access-dr557\") pod \"e2e-derive-5a03fa09-openshift-default-c77997b9f-lhbsz\" (UID: \"596c881b-ea8c-45c5-b115-ee790a485d95\") " pod="openshift-ingress/e2e-derive-5a03fa09-openshift-default-c77997b9f-lhbsz" Jul 16 00:00:59.885899 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:00:59.885882 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/596c881b-ea8c-45c5-b115-ee790a485d95-istio-token\") pod \"e2e-derive-5a03fa09-openshift-default-c77997b9f-lhbsz\" (UID: \"596c881b-ea8c-45c5-b115-ee790a485d95\") " pod="openshift-ingress/e2e-derive-5a03fa09-openshift-default-c77997b9f-lhbsz" Jul 16 00:01:00.060096 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:01:00.060073 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-55cbdb16-gw-openshift-default-d46ff9dd7-2qbpc" Jul 16 00:01:00.060372 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:01:00.060354 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-5a03fa09-openshift-default-c77997b9f-lhbsz" Jul 16 00:01:00.080743 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:01:00.080719 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rbjc7\" (UniqueName: \"kubernetes.io/projected/fb7ce561-2cfa-4232-b6cf-1786997379a0-kube-api-access-rbjc7\") pod \"fb7ce561-2cfa-4232-b6cf-1786997379a0\" (UID: \"fb7ce561-2cfa-4232-b6cf-1786997379a0\") " Jul 16 00:01:00.080871 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:01:00.080752 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb7ce561-2cfa-4232-b6cf-1786997379a0-istio-data\") pod \"fb7ce561-2cfa-4232-b6cf-1786997379a0\" (UID: \"fb7ce561-2cfa-4232-b6cf-1786997379a0\") " Jul 16 00:01:00.080871 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:01:00.080786 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb7ce561-2cfa-4232-b6cf-1786997379a0-istio-envoy\") pod \"fb7ce561-2cfa-4232-b6cf-1786997379a0\" (UID: \"fb7ce561-2cfa-4232-b6cf-1786997379a0\") " Jul 16 00:01:00.080871 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:01:00.080832 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb7ce561-2cfa-4232-b6cf-1786997379a0-workload-socket\") pod \"fb7ce561-2cfa-4232-b6cf-1786997379a0\" (UID: \"fb7ce561-2cfa-4232-b6cf-1786997379a0\") " Jul 16 00:01:00.081015 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:01:00.080902 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb7ce561-2cfa-4232-b6cf-1786997379a0-workload-certs\") pod \"fb7ce561-2cfa-4232-b6cf-1786997379a0\" (UID: \"fb7ce561-2cfa-4232-b6cf-1786997379a0\") " Jul 16 00:01:00.081015 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:01:00.080929 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb7ce561-2cfa-4232-b6cf-1786997379a0-credential-socket\") pod \"fb7ce561-2cfa-4232-b6cf-1786997379a0\" (UID: \"fb7ce561-2cfa-4232-b6cf-1786997379a0\") " Jul 16 00:01:00.081015 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:01:00.080990 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb7ce561-2cfa-4232-b6cf-1786997379a0-istio-token\") pod \"fb7ce561-2cfa-4232-b6cf-1786997379a0\" (UID: \"fb7ce561-2cfa-4232-b6cf-1786997379a0\") " Jul 16 00:01:00.081149 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:01:00.081056 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb7ce561-2cfa-4232-b6cf-1786997379a0-istiod-ca-cert\") pod \"fb7ce561-2cfa-4232-b6cf-1786997379a0\" (UID: \"fb7ce561-2cfa-4232-b6cf-1786997379a0\") " Jul 16 00:01:00.081149 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:01:00.081094 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb7ce561-2cfa-4232-b6cf-1786997379a0-istio-podinfo\") pod \"fb7ce561-2cfa-4232-b6cf-1786997379a0\" (UID: \"fb7ce561-2cfa-4232-b6cf-1786997379a0\") " Jul 16 00:01:00.081392 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:01:00.081317 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb7ce561-2cfa-4232-b6cf-1786997379a0-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "fb7ce561-2cfa-4232-b6cf-1786997379a0" (UID: "fb7ce561-2cfa-4232-b6cf-1786997379a0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 00:01:00.081392 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:01:00.081367 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb7ce561-2cfa-4232-b6cf-1786997379a0-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "fb7ce561-2cfa-4232-b6cf-1786997379a0" (UID: "fb7ce561-2cfa-4232-b6cf-1786997379a0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 00:01:00.081618 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:01:00.081388 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb7ce561-2cfa-4232-b6cf-1786997379a0-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "fb7ce561-2cfa-4232-b6cf-1786997379a0" (UID: "fb7ce561-2cfa-4232-b6cf-1786997379a0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 00:01:00.081618 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:01:00.081537 2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb7ce561-2cfa-4232-b6cf-1786997379a0-workload-socket\") on node \"ip-10-0-128-197.ec2.internal\" DevicePath \"\"" Jul 16 00:01:00.081618 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:01:00.081557 2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb7ce561-2cfa-4232-b6cf-1786997379a0-workload-certs\") on node \"ip-10-0-128-197.ec2.internal\" DevicePath \"\"" Jul 16 00:01:00.081618 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:01:00.081572 2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb7ce561-2cfa-4232-b6cf-1786997379a0-credential-socket\") on node \"ip-10-0-128-197.ec2.internal\" DevicePath \"\"" Jul 16 00:01:00.082307 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:01:00.082259 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb7ce561-2cfa-4232-b6cf-1786997379a0-istio-data" (OuterVolumeSpecName: "istio-data") pod "fb7ce561-2cfa-4232-b6cf-1786997379a0" (UID: "fb7ce561-2cfa-4232-b6cf-1786997379a0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 00:01:00.082650 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:01:00.082624 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb7ce561-2cfa-4232-b6cf-1786997379a0-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "fb7ce561-2cfa-4232-b6cf-1786997379a0" (UID: "fb7ce561-2cfa-4232-b6cf-1786997379a0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 00:01:00.083820 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:01:00.083778 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb7ce561-2cfa-4232-b6cf-1786997379a0-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "fb7ce561-2cfa-4232-b6cf-1786997379a0" (UID: "fb7ce561-2cfa-4232-b6cf-1786997379a0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 00:01:00.084167 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:01:00.084119 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb7ce561-2cfa-4232-b6cf-1786997379a0-istio-token" (OuterVolumeSpecName: "istio-token") pod "fb7ce561-2cfa-4232-b6cf-1786997379a0" (UID: "fb7ce561-2cfa-4232-b6cf-1786997379a0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 00:01:00.084256 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:01:00.084158 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb7ce561-2cfa-4232-b6cf-1786997379a0-kube-api-access-rbjc7" (OuterVolumeSpecName: "kube-api-access-rbjc7") pod "fb7ce561-2cfa-4232-b6cf-1786997379a0" (UID: "fb7ce561-2cfa-4232-b6cf-1786997379a0"). InnerVolumeSpecName "kube-api-access-rbjc7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 00:01:00.085176 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:01:00.085143 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fb7ce561-2cfa-4232-b6cf-1786997379a0-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "fb7ce561-2cfa-4232-b6cf-1786997379a0" (UID: "fb7ce561-2cfa-4232-b6cf-1786997379a0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 16 00:01:00.182887 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:01:00.182854 2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb7ce561-2cfa-4232-b6cf-1786997379a0-istio-envoy\") on node \"ip-10-0-128-197.ec2.internal\" DevicePath \"\"" Jul 16 00:01:00.182887 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:01:00.182883 2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb7ce561-2cfa-4232-b6cf-1786997379a0-istio-token\") on node \"ip-10-0-128-197.ec2.internal\" DevicePath \"\"" Jul 16 00:01:00.183091 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:01:00.182895 2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb7ce561-2cfa-4232-b6cf-1786997379a0-istiod-ca-cert\") on node \"ip-10-0-128-197.ec2.internal\" DevicePath \"\"" Jul 16 00:01:00.183091 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:01:00.182908 2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb7ce561-2cfa-4232-b6cf-1786997379a0-istio-podinfo\") on node \"ip-10-0-128-197.ec2.internal\" DevicePath \"\"" Jul 16 00:01:00.183091 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:01:00.182920 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rbjc7\" (UniqueName: \"kubernetes.io/projected/fb7ce561-2cfa-4232-b6cf-1786997379a0-kube-api-access-rbjc7\") on node \"ip-10-0-128-197.ec2.internal\" DevicePath \"\"" Jul 16 00:01:00.183091 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:01:00.182933 2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb7ce561-2cfa-4232-b6cf-1786997379a0-istio-data\") on node \"ip-10-0-128-197.ec2.internal\" DevicePath \"\"" Jul 16 00:01:00.208547 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:01:00.208523 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-5a03fa09-openshift-default-c77997b9f-lhbsz"] Jul 16 00:01:00.210211 ip-10-0-128-197 kubenswrapper[2563]: W0716 00:01:00.210189 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod596c881b_ea8c_45c5_b115_ee790a485d95.slice/crio-77eae7dcdf8707d87f11f5f6230c118fbe7203572ca8f4158370f68e3c3d2f53 WatchSource:0}: Error finding container 77eae7dcdf8707d87f11f5f6230c118fbe7203572ca8f4158370f68e3c3d2f53: Status 404 returned error can't find the container with id 77eae7dcdf8707d87f11f5f6230c118fbe7203572ca8f4158370f68e3c3d2f53 Jul 16 00:01:00.212385 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:01:00.212352 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 16 00:01:00.212476 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:01:00.212413 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 16 00:01:00.212476 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:01:00.212445 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 16 00:01:00.276093 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:01:00.276066 2563 generic.go:356] "Generic (PLEG): container finished" podID="fb7ce561-2cfa-4232-b6cf-1786997379a0" containerID="9a3acda20b426b0c9fd69aa90203d2020a0a26ecc4ef2a870a9c7b93e0ea6df1" exitCode=0 Jul 16 00:01:00.276174 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:01:00.276107 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-55cbdb16-gw-openshift-default-d46ff9dd7-2qbpc" Jul 16 00:01:00.276174 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:01:00.276154 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-55cbdb16-gw-openshift-default-d46ff9dd7-2qbpc" event={"ID":"fb7ce561-2cfa-4232-b6cf-1786997379a0","Type":"ContainerDied","Data":"9a3acda20b426b0c9fd69aa90203d2020a0a26ecc4ef2a870a9c7b93e0ea6df1"} Jul 16 00:01:00.276267 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:01:00.276199 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-55cbdb16-gw-openshift-default-d46ff9dd7-2qbpc" event={"ID":"fb7ce561-2cfa-4232-b6cf-1786997379a0","Type":"ContainerDied","Data":"8d4188f98bc23b0e8c40c424e6fdd49a17ecc3964353d1e52b6df715612f166e"} Jul 16 00:01:00.276267 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:01:00.276220 2563 scope.go:117] "RemoveContainer" containerID="9a3acda20b426b0c9fd69aa90203d2020a0a26ecc4ef2a870a9c7b93e0ea6df1" Jul 16 00:01:00.277886 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:01:00.277860 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-5a03fa09-openshift-default-c77997b9f-lhbsz" event={"ID":"596c881b-ea8c-45c5-b115-ee790a485d95","Type":"ContainerStarted","Data":"77eae7dcdf8707d87f11f5f6230c118fbe7203572ca8f4158370f68e3c3d2f53"} Jul 16 00:01:00.288665 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:01:00.288645 2563 scope.go:117] "RemoveContainer" containerID="9a3acda20b426b0c9fd69aa90203d2020a0a26ecc4ef2a870a9c7b93e0ea6df1" Jul 16 00:01:00.288942 ip-10-0-128-197 kubenswrapper[2563]: E0716 00:01:00.288920 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a3acda20b426b0c9fd69aa90203d2020a0a26ecc4ef2a870a9c7b93e0ea6df1\": container with ID starting with 9a3acda20b426b0c9fd69aa90203d2020a0a26ecc4ef2a870a9c7b93e0ea6df1 not found: ID does not exist" containerID="9a3acda20b426b0c9fd69aa90203d2020a0a26ecc4ef2a870a9c7b93e0ea6df1" Jul 16 00:01:00.289038 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:01:00.288955 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a3acda20b426b0c9fd69aa90203d2020a0a26ecc4ef2a870a9c7b93e0ea6df1"} err="failed to get container status \"9a3acda20b426b0c9fd69aa90203d2020a0a26ecc4ef2a870a9c7b93e0ea6df1\": rpc error: code = NotFound desc = could not find container \"9a3acda20b426b0c9fd69aa90203d2020a0a26ecc4ef2a870a9c7b93e0ea6df1\": container with ID starting with 9a3acda20b426b0c9fd69aa90203d2020a0a26ecc4ef2a870a9c7b93e0ea6df1 not found: ID does not exist" Jul 16 00:01:00.299980 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:01:00.299940 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-55cbdb16-gw-openshift-default-d46ff9dd7-2qbpc"] Jul 16 00:01:00.305410 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:01:00.305389 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-55cbdb16-gw-openshift-default-d46ff9dd7-2qbpc"] Jul 16 00:01:00.384671 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:01:00.384582 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:01:00.482580 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:01:00.482548 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:01:00.585629 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:01:00.585601 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:01:00.683307 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:01:00.683196 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:01:01.174984 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:01:01.173186 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fb7ce561-2cfa-4232-b6cf-1786997379a0" path="/var/lib/kubelet/pods/fb7ce561-2cfa-4232-b6cf-1786997379a0/volumes" Jul 16 00:01:01.285441 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:01:01.285406 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-5a03fa09-openshift-default-c77997b9f-lhbsz" event={"ID":"596c881b-ea8c-45c5-b115-ee790a485d95","Type":"ContainerStarted","Data":"8e2033c17027cd482920c56a8bd6205427171a7ebfbf4eaae07c53fe64161987"} Jul 16 00:01:01.306307 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:01:01.306258 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-5a03fa09-openshift-default-c77997b9f-lhbsz" podStartSLOduration=2.306242971 podStartE2EDuration="2.306242971s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 00:00:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 00:01:01.304689065 +0000 UTC m=+1476.678147031" watchObservedRunningTime="2026-07-16 00:01:01.306242971 +0000 UTC m=+1476.679700933" Jul 16 00:01:02.060540 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:01:02.060507 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-5a03fa09-openshift-default-c77997b9f-lhbsz" Jul 16 00:01:02.065684 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:01:02.065659 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-5a03fa09-openshift-default-c77997b9f-lhbsz" Jul 16 00:01:02.291466 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:01:02.291438 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-5a03fa09-openshift-default-c77997b9f-lhbsz" Jul 16 00:01:02.292582 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:01:02.292562 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-5a03fa09-openshift-default-c77997b9f-lhbsz" Jul 16 00:01:03.164151 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:01:03.164111 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:01:03.258936 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:01:03.258901 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:01:05.413700 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:01:05.413657 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-5a03fa09-openshift-default-c77997b9f-lhbsz"] Jul 16 00:01:05.414261 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:01:05.413912 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-5a03fa09-openshift-default-c77997b9f-lhbsz" podUID="596c881b-ea8c-45c5-b115-ee790a485d95" containerName="istio-proxy" containerID="cri-o://8e2033c17027cd482920c56a8bd6205427171a7ebfbf4eaae07c53fe64161987" gracePeriod=30 Jul 16 00:01:05.425917 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:01:05.425886 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:01:05.544390 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:01:05.544357 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:01:07.254139 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:01:07.254109 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:01:11.681754 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:01:11.681718 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:01:11.782389 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:01:11.782359 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:01:12.139925 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:01:12.139855 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:01:12.168197 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:01:12.168178 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-5a03fa09-openshift-default-c77997b9f-lhbsz" Jul 16 00:01:12.239073 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:01:12.239046 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:01:12.303754 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:01:12.303728 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/596c881b-ea8c-45c5-b115-ee790a485d95-istio-token\") pod \"596c881b-ea8c-45c5-b115-ee790a485d95\" (UID: \"596c881b-ea8c-45c5-b115-ee790a485d95\") " Jul 16 00:01:12.303956 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:01:12.303759 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/596c881b-ea8c-45c5-b115-ee790a485d95-workload-socket\") pod \"596c881b-ea8c-45c5-b115-ee790a485d95\" (UID: \"596c881b-ea8c-45c5-b115-ee790a485d95\") " Jul 16 00:01:12.303956 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:01:12.303777 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dr557\" (UniqueName: \"kubernetes.io/projected/596c881b-ea8c-45c5-b115-ee790a485d95-kube-api-access-dr557\") pod \"596c881b-ea8c-45c5-b115-ee790a485d95\" (UID: \"596c881b-ea8c-45c5-b115-ee790a485d95\") " Jul 16 00:01:12.303956 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:01:12.303821 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/596c881b-ea8c-45c5-b115-ee790a485d95-workload-certs\") pod \"596c881b-ea8c-45c5-b115-ee790a485d95\" (UID: \"596c881b-ea8c-45c5-b115-ee790a485d95\") " Jul 16 00:01:12.303956 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:01:12.303838 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/596c881b-ea8c-45c5-b115-ee790a485d95-istio-podinfo\") pod \"596c881b-ea8c-45c5-b115-ee790a485d95\" (UID: \"596c881b-ea8c-45c5-b115-ee790a485d95\") " Jul 16 00:01:12.303956 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:01:12.303861 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/596c881b-ea8c-45c5-b115-ee790a485d95-istio-data\") pod \"596c881b-ea8c-45c5-b115-ee790a485d95\" (UID: \"596c881b-ea8c-45c5-b115-ee790a485d95\") " Jul 16 00:01:12.303956 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:01:12.303898 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/596c881b-ea8c-45c5-b115-ee790a485d95-credential-socket\") pod \"596c881b-ea8c-45c5-b115-ee790a485d95\" (UID: \"596c881b-ea8c-45c5-b115-ee790a485d95\") " Jul 16 00:01:12.304309 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:01:12.304052 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/596c881b-ea8c-45c5-b115-ee790a485d95-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "596c881b-ea8c-45c5-b115-ee790a485d95" (UID: "596c881b-ea8c-45c5-b115-ee790a485d95"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 00:01:12.304309 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:01:12.304067 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/596c881b-ea8c-45c5-b115-ee790a485d95-istio-envoy\") pod \"596c881b-ea8c-45c5-b115-ee790a485d95\" (UID: \"596c881b-ea8c-45c5-b115-ee790a485d95\") " Jul 16 00:01:12.304309 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:01:12.304119 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/596c881b-ea8c-45c5-b115-ee790a485d95-istiod-ca-cert\") pod \"596c881b-ea8c-45c5-b115-ee790a485d95\" (UID: \"596c881b-ea8c-45c5-b115-ee790a485d95\") " Jul 16 00:01:12.304309 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:01:12.304183 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/596c881b-ea8c-45c5-b115-ee790a485d95-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "596c881b-ea8c-45c5-b115-ee790a485d95" (UID: "596c881b-ea8c-45c5-b115-ee790a485d95"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 00:01:12.304499 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:01:12.304442 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/596c881b-ea8c-45c5-b115-ee790a485d95-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "596c881b-ea8c-45c5-b115-ee790a485d95" (UID: "596c881b-ea8c-45c5-b115-ee790a485d95"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 00:01:12.304557 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:01:12.304496 2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/596c881b-ea8c-45c5-b115-ee790a485d95-workload-socket\") on node \"ip-10-0-128-197.ec2.internal\" DevicePath \"\"" Jul 16 00:01:12.304557 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:01:12.304516 2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/596c881b-ea8c-45c5-b115-ee790a485d95-workload-certs\") on node \"ip-10-0-128-197.ec2.internal\" DevicePath \"\"" Jul 16 00:01:12.304645 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:01:12.304553 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/596c881b-ea8c-45c5-b115-ee790a485d95-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "596c881b-ea8c-45c5-b115-ee790a485d95" (UID: "596c881b-ea8c-45c5-b115-ee790a485d95"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 00:01:12.304733 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:01:12.304702 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/596c881b-ea8c-45c5-b115-ee790a485d95-istio-data" (OuterVolumeSpecName: "istio-data") pod "596c881b-ea8c-45c5-b115-ee790a485d95" (UID: "596c881b-ea8c-45c5-b115-ee790a485d95"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 00:01:12.306006 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:01:12.305984 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/596c881b-ea8c-45c5-b115-ee790a485d95-istio-token" (OuterVolumeSpecName: "istio-token") pod "596c881b-ea8c-45c5-b115-ee790a485d95" (UID: "596c881b-ea8c-45c5-b115-ee790a485d95"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 00:01:12.306144 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:01:12.306129 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/596c881b-ea8c-45c5-b115-ee790a485d95-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "596c881b-ea8c-45c5-b115-ee790a485d95" (UID: "596c881b-ea8c-45c5-b115-ee790a485d95"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 00:01:12.306381 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:01:12.306359 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/596c881b-ea8c-45c5-b115-ee790a485d95-kube-api-access-dr557" (OuterVolumeSpecName: "kube-api-access-dr557") pod "596c881b-ea8c-45c5-b115-ee790a485d95" (UID: "596c881b-ea8c-45c5-b115-ee790a485d95"). InnerVolumeSpecName "kube-api-access-dr557". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 00:01:12.306454 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:01:12.306372 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/596c881b-ea8c-45c5-b115-ee790a485d95-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "596c881b-ea8c-45c5-b115-ee790a485d95" (UID: "596c881b-ea8c-45c5-b115-ee790a485d95"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 16 00:01:12.339477 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:01:12.339448 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:01:12.348427 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:01:12.348401 2563 generic.go:356] "Generic (PLEG): container finished" podID="596c881b-ea8c-45c5-b115-ee790a485d95" containerID="8e2033c17027cd482920c56a8bd6205427171a7ebfbf4eaae07c53fe64161987" exitCode=0 Jul 16 00:01:12.348532 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:01:12.348442 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-5a03fa09-openshift-default-c77997b9f-lhbsz" event={"ID":"596c881b-ea8c-45c5-b115-ee790a485d95","Type":"ContainerDied","Data":"8e2033c17027cd482920c56a8bd6205427171a7ebfbf4eaae07c53fe64161987"} Jul 16 00:01:12.348532 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:01:12.348495 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-5a03fa09-openshift-default-c77997b9f-lhbsz" event={"ID":"596c881b-ea8c-45c5-b115-ee790a485d95","Type":"ContainerDied","Data":"77eae7dcdf8707d87f11f5f6230c118fbe7203572ca8f4158370f68e3c3d2f53"} Jul 16 00:01:12.348532 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:01:12.348448 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-5a03fa09-openshift-default-c77997b9f-lhbsz" Jul 16 00:01:12.348532 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:01:12.348517 2563 scope.go:117] "RemoveContainer" containerID="8e2033c17027cd482920c56a8bd6205427171a7ebfbf4eaae07c53fe64161987" Jul 16 00:01:12.357431 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:01:12.357417 2563 scope.go:117] "RemoveContainer" containerID="8e2033c17027cd482920c56a8bd6205427171a7ebfbf4eaae07c53fe64161987" Jul 16 00:01:12.357692 ip-10-0-128-197 kubenswrapper[2563]: E0716 00:01:12.357673 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e2033c17027cd482920c56a8bd6205427171a7ebfbf4eaae07c53fe64161987\": container with ID starting with 8e2033c17027cd482920c56a8bd6205427171a7ebfbf4eaae07c53fe64161987 not found: ID does not exist" containerID="8e2033c17027cd482920c56a8bd6205427171a7ebfbf4eaae07c53fe64161987" Jul 16 00:01:12.357756 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:01:12.357698 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e2033c17027cd482920c56a8bd6205427171a7ebfbf4eaae07c53fe64161987"} err="failed to get container status \"8e2033c17027cd482920c56a8bd6205427171a7ebfbf4eaae07c53fe64161987\": rpc error: code = NotFound desc = could not find container \"8e2033c17027cd482920c56a8bd6205427171a7ebfbf4eaae07c53fe64161987\": container with ID starting with 8e2033c17027cd482920c56a8bd6205427171a7ebfbf4eaae07c53fe64161987 not found: ID does not exist" Jul 16 00:01:12.370074 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:01:12.370054 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-5a03fa09-openshift-default-c77997b9f-lhbsz"] Jul 16 00:01:12.375662 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:01:12.375642 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-5a03fa09-openshift-default-c77997b9f-lhbsz"] Jul 16 00:01:12.405481 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:01:12.405458 2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/596c881b-ea8c-45c5-b115-ee790a485d95-istio-podinfo\") on node \"ip-10-0-128-197.ec2.internal\" DevicePath \"\"" Jul 16 00:01:12.405481 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:01:12.405480 2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/596c881b-ea8c-45c5-b115-ee790a485d95-istio-data\") on node \"ip-10-0-128-197.ec2.internal\" DevicePath \"\"" Jul 16 00:01:12.405613 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:01:12.405491 2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/596c881b-ea8c-45c5-b115-ee790a485d95-credential-socket\") on node \"ip-10-0-128-197.ec2.internal\" DevicePath \"\"" Jul 16 00:01:12.405613 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:01:12.405500 2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/596c881b-ea8c-45c5-b115-ee790a485d95-istio-envoy\") on node \"ip-10-0-128-197.ec2.internal\" DevicePath \"\"" Jul 16 00:01:12.405613 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:01:12.405508 2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/596c881b-ea8c-45c5-b115-ee790a485d95-istiod-ca-cert\") on node \"ip-10-0-128-197.ec2.internal\" DevicePath \"\"" Jul 16 00:01:12.405613 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:01:12.405515 2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/596c881b-ea8c-45c5-b115-ee790a485d95-istio-token\") on node \"ip-10-0-128-197.ec2.internal\" DevicePath \"\"" Jul 16 00:01:12.405613 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:01:12.405525 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dr557\" (UniqueName: \"kubernetes.io/projected/596c881b-ea8c-45c5-b115-ee790a485d95-kube-api-access-dr557\") on node \"ip-10-0-128-197.ec2.internal\" DevicePath \"\"" Jul 16 00:01:13.170422 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:01:13.170386 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="596c881b-ea8c-45c5-b115-ee790a485d95" path="/var/lib/kubelet/pods/596c881b-ea8c-45c5-b115-ee790a485d95/volumes" Jul 16 00:01:13.790179 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:01:13.790147 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:01:14.722595 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:01:14.722565 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:01:15.020257 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:01:15.020227 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:01:19.705377 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:01:19.705342 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:01:19.774577 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:01:19.774539 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:01:19.855827 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:01:19.855796 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:01:19.955928 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:01:19.955851 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:01:20.056312 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:01:20.056282 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:01:20.522905 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:01:20.522869 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:01:25.115530 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:01:25.115501 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d8f6k_2f64cab3-104a-44f6-b125-ba458f54f058/ovn-acl-logging/0.log" Jul 16 00:01:25.116114 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:01:25.116097 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d8f6k_2f64cab3-104a-44f6-b125-ba458f54f058/ovn-acl-logging/0.log" Jul 16 00:01:46.371777 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:01:46.371748 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:01:51.991461 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:01:51.991384 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:01:52.096149 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:01:52.096117 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:01:52.192737 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:01:52.192699 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:02:08.074147 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:02:08.074111 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:02:13.902994 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:02:13.902949 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:02:13.963922 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:02:13.963885 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:02:14.035592 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:02:14.035558 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:02:14.137376 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:02:14.137344 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:02:24.602038 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:02:24.602000 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:02:25.382072 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:02:25.382040 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:02:31.140114 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:02:31.140079 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:02:31.196960 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:02:31.196931 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:02:31.272718 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:02:31.272688 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:02:31.371361 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:02:31.371328 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:02:31.470704 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:02:31.470621 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:02:32.184146 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:02:32.184116 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:02:32.283648 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:02:32.283618 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:02:32.338781 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:02:32.338747 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:02:33.986458 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:02:33.986432 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:02:39.644750 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:02:39.644719 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:02:39.746366 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:02:39.746334 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:02:40.297318 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:02:40.297281 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:02:46.279717 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:02:46.279669 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:02:46.335758 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:02:46.335726 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:02:46.559576 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:02:46.559485 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:02:46.818156 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:02:46.818082 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-df4149da-openshift-default-6d4688bf67-ngnnw"] Jul 16 00:02:46.818487 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:02:46.818475 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb7ce561-2cfa-4232-b6cf-1786997379a0" containerName="istio-proxy" Jul 16 00:02:46.818528 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:02:46.818488 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb7ce561-2cfa-4232-b6cf-1786997379a0" containerName="istio-proxy" Jul 16 00:02:46.818528 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:02:46.818504 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="596c881b-ea8c-45c5-b115-ee790a485d95" containerName="istio-proxy" Jul 16 00:02:46.818528 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:02:46.818510 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="596c881b-ea8c-45c5-b115-ee790a485d95" containerName="istio-proxy" Jul 16 00:02:46.818620 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:02:46.818569 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="596c881b-ea8c-45c5-b115-ee790a485d95" containerName="istio-proxy" Jul 16 00:02:46.818620 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:02:46.818580 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="fb7ce561-2cfa-4232-b6cf-1786997379a0" containerName="istio-proxy" Jul 16 00:02:46.821641 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:02:46.821622 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-df4149da-openshift-default-6d4688bf67-ngnnw" Jul 16 00:02:46.823926 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:02:46.823908 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-df4149da-openshift-default-dockercfg-6nbr9\"" Jul 16 00:02:46.841838 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:02:46.841810 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-df4149da-openshift-default-6d4688bf67-ngnnw"] Jul 16 00:02:46.857303 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:02:46.857269 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:02:46.951711 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:02:46.951685 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:02:46.993983 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:02:46.993935 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd7f43d9-6c38-4afe-9d46-62ce207e5e92-istio-token\") pod \"e2e-mt-df4149da-openshift-default-6d4688bf67-ngnnw\" (UID: \"fd7f43d9-6c38-4afe-9d46-62ce207e5e92\") " pod="openshift-ingress/e2e-mt-df4149da-openshift-default-6d4688bf67-ngnnw" Jul 16 00:02:46.994186 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:02:46.994048 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd7f43d9-6c38-4afe-9d46-62ce207e5e92-istio-data\") pod \"e2e-mt-df4149da-openshift-default-6d4688bf67-ngnnw\" (UID: \"fd7f43d9-6c38-4afe-9d46-62ce207e5e92\") " pod="openshift-ingress/e2e-mt-df4149da-openshift-default-6d4688bf67-ngnnw" Jul 16 00:02:46.994186 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:02:46.994095 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd7f43d9-6c38-4afe-9d46-62ce207e5e92-istio-envoy\") pod \"e2e-mt-df4149da-openshift-default-6d4688bf67-ngnnw\" (UID: \"fd7f43d9-6c38-4afe-9d46-62ce207e5e92\") " pod="openshift-ingress/e2e-mt-df4149da-openshift-default-6d4688bf67-ngnnw" Jul 16 00:02:46.994186 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:02:46.994121 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd7f43d9-6c38-4afe-9d46-62ce207e5e92-workload-socket\") pod \"e2e-mt-df4149da-openshift-default-6d4688bf67-ngnnw\" (UID: \"fd7f43d9-6c38-4afe-9d46-62ce207e5e92\") " pod="openshift-ingress/e2e-mt-df4149da-openshift-default-6d4688bf67-ngnnw" Jul 16 00:02:46.994332 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:02:46.994208 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9bxm4\" (UniqueName: \"kubernetes.io/projected/fd7f43d9-6c38-4afe-9d46-62ce207e5e92-kube-api-access-9bxm4\") pod \"e2e-mt-df4149da-openshift-default-6d4688bf67-ngnnw\" (UID: \"fd7f43d9-6c38-4afe-9d46-62ce207e5e92\") " pod="openshift-ingress/e2e-mt-df4149da-openshift-default-6d4688bf67-ngnnw" Jul 16 00:02:46.994332 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:02:46.994269 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd7f43d9-6c38-4afe-9d46-62ce207e5e92-workload-certs\") pod \"e2e-mt-df4149da-openshift-default-6d4688bf67-ngnnw\" (UID: \"fd7f43d9-6c38-4afe-9d46-62ce207e5e92\") " pod="openshift-ingress/e2e-mt-df4149da-openshift-default-6d4688bf67-ngnnw" Jul 16 00:02:46.994332 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:02:46.994318 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd7f43d9-6c38-4afe-9d46-62ce207e5e92-istio-podinfo\") pod \"e2e-mt-df4149da-openshift-default-6d4688bf67-ngnnw\" (UID: \"fd7f43d9-6c38-4afe-9d46-62ce207e5e92\") " pod="openshift-ingress/e2e-mt-df4149da-openshift-default-6d4688bf67-ngnnw" Jul 16 00:02:46.994450 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:02:46.994354 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd7f43d9-6c38-4afe-9d46-62ce207e5e92-credential-socket\") pod \"e2e-mt-df4149da-openshift-default-6d4688bf67-ngnnw\" (UID: \"fd7f43d9-6c38-4afe-9d46-62ce207e5e92\") " pod="openshift-ingress/e2e-mt-df4149da-openshift-default-6d4688bf67-ngnnw" Jul 16 00:02:46.994450 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:02:46.994382 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd7f43d9-6c38-4afe-9d46-62ce207e5e92-istiod-ca-cert\") pod \"e2e-mt-df4149da-openshift-default-6d4688bf67-ngnnw\" (UID: \"fd7f43d9-6c38-4afe-9d46-62ce207e5e92\") " pod="openshift-ingress/e2e-mt-df4149da-openshift-default-6d4688bf67-ngnnw" Jul 16 00:02:47.095957 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:02:47.095880 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd7f43d9-6c38-4afe-9d46-62ce207e5e92-istio-data\") pod \"e2e-mt-df4149da-openshift-default-6d4688bf67-ngnnw\" (UID: \"fd7f43d9-6c38-4afe-9d46-62ce207e5e92\") " pod="openshift-ingress/e2e-mt-df4149da-openshift-default-6d4688bf67-ngnnw" Jul 16 00:02:47.095957 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:02:47.095926 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd7f43d9-6c38-4afe-9d46-62ce207e5e92-istio-envoy\") pod \"e2e-mt-df4149da-openshift-default-6d4688bf67-ngnnw\" (UID: \"fd7f43d9-6c38-4afe-9d46-62ce207e5e92\") " pod="openshift-ingress/e2e-mt-df4149da-openshift-default-6d4688bf67-ngnnw" Jul 16 00:02:47.095957 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:02:47.095947 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd7f43d9-6c38-4afe-9d46-62ce207e5e92-workload-socket\") pod \"e2e-mt-df4149da-openshift-default-6d4688bf67-ngnnw\" (UID: \"fd7f43d9-6c38-4afe-9d46-62ce207e5e92\") " pod="openshift-ingress/e2e-mt-df4149da-openshift-default-6d4688bf67-ngnnw" Jul 16 00:02:47.096243 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:02:47.095998 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9bxm4\" (UniqueName: \"kubernetes.io/projected/fd7f43d9-6c38-4afe-9d46-62ce207e5e92-kube-api-access-9bxm4\") pod \"e2e-mt-df4149da-openshift-default-6d4688bf67-ngnnw\" (UID: \"fd7f43d9-6c38-4afe-9d46-62ce207e5e92\") " pod="openshift-ingress/e2e-mt-df4149da-openshift-default-6d4688bf67-ngnnw" Jul 16 00:02:47.096243 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:02:47.096024 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd7f43d9-6c38-4afe-9d46-62ce207e5e92-workload-certs\") pod \"e2e-mt-df4149da-openshift-default-6d4688bf67-ngnnw\" (UID: \"fd7f43d9-6c38-4afe-9d46-62ce207e5e92\") " pod="openshift-ingress/e2e-mt-df4149da-openshift-default-6d4688bf67-ngnnw" Jul 16 00:02:47.096243 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:02:47.096058 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd7f43d9-6c38-4afe-9d46-62ce207e5e92-istio-podinfo\") pod \"e2e-mt-df4149da-openshift-default-6d4688bf67-ngnnw\" (UID: \"fd7f43d9-6c38-4afe-9d46-62ce207e5e92\") " pod="openshift-ingress/e2e-mt-df4149da-openshift-default-6d4688bf67-ngnnw" Jul 16 00:02:47.096243 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:02:47.096099 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd7f43d9-6c38-4afe-9d46-62ce207e5e92-credential-socket\") pod \"e2e-mt-df4149da-openshift-default-6d4688bf67-ngnnw\" (UID: \"fd7f43d9-6c38-4afe-9d46-62ce207e5e92\") " pod="openshift-ingress/e2e-mt-df4149da-openshift-default-6d4688bf67-ngnnw" Jul 16 00:02:47.096243 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:02:47.096133 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd7f43d9-6c38-4afe-9d46-62ce207e5e92-istiod-ca-cert\") pod \"e2e-mt-df4149da-openshift-default-6d4688bf67-ngnnw\" (UID: \"fd7f43d9-6c38-4afe-9d46-62ce207e5e92\") " pod="openshift-ingress/e2e-mt-df4149da-openshift-default-6d4688bf67-ngnnw" Jul 16 00:02:47.096243 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:02:47.096161 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd7f43d9-6c38-4afe-9d46-62ce207e5e92-istio-token\") pod \"e2e-mt-df4149da-openshift-default-6d4688bf67-ngnnw\" (UID: \"fd7f43d9-6c38-4afe-9d46-62ce207e5e92\") " pod="openshift-ingress/e2e-mt-df4149da-openshift-default-6d4688bf67-ngnnw" Jul 16 00:02:47.096523 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:02:47.096357 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd7f43d9-6c38-4afe-9d46-62ce207e5e92-istio-data\") pod \"e2e-mt-df4149da-openshift-default-6d4688bf67-ngnnw\" (UID: \"fd7f43d9-6c38-4afe-9d46-62ce207e5e92\") " pod="openshift-ingress/e2e-mt-df4149da-openshift-default-6d4688bf67-ngnnw" Jul 16 00:02:47.096523 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:02:47.096417 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd7f43d9-6c38-4afe-9d46-62ce207e5e92-workload-socket\") pod \"e2e-mt-df4149da-openshift-default-6d4688bf67-ngnnw\" (UID: \"fd7f43d9-6c38-4afe-9d46-62ce207e5e92\") " pod="openshift-ingress/e2e-mt-df4149da-openshift-default-6d4688bf67-ngnnw" Jul 16 00:02:47.096523 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:02:47.096511 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd7f43d9-6c38-4afe-9d46-62ce207e5e92-workload-certs\") pod \"e2e-mt-df4149da-openshift-default-6d4688bf67-ngnnw\" (UID: \"fd7f43d9-6c38-4afe-9d46-62ce207e5e92\") " pod="openshift-ingress/e2e-mt-df4149da-openshift-default-6d4688bf67-ngnnw" Jul 16 00:02:47.096661 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:02:47.096591 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd7f43d9-6c38-4afe-9d46-62ce207e5e92-credential-socket\") pod \"e2e-mt-df4149da-openshift-default-6d4688bf67-ngnnw\" (UID: \"fd7f43d9-6c38-4afe-9d46-62ce207e5e92\") " pod="openshift-ingress/e2e-mt-df4149da-openshift-default-6d4688bf67-ngnnw" Jul 16 00:02:47.097097 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:02:47.097077 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd7f43d9-6c38-4afe-9d46-62ce207e5e92-istiod-ca-cert\") pod \"e2e-mt-df4149da-openshift-default-6d4688bf67-ngnnw\" (UID: \"fd7f43d9-6c38-4afe-9d46-62ce207e5e92\") " pod="openshift-ingress/e2e-mt-df4149da-openshift-default-6d4688bf67-ngnnw" Jul 16 00:02:47.098393 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:02:47.098369 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd7f43d9-6c38-4afe-9d46-62ce207e5e92-istio-envoy\") pod \"e2e-mt-df4149da-openshift-default-6d4688bf67-ngnnw\" (UID: \"fd7f43d9-6c38-4afe-9d46-62ce207e5e92\") " pod="openshift-ingress/e2e-mt-df4149da-openshift-default-6d4688bf67-ngnnw" Jul 16 00:02:47.098493 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:02:47.098477 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd7f43d9-6c38-4afe-9d46-62ce207e5e92-istio-podinfo\") pod \"e2e-mt-df4149da-openshift-default-6d4688bf67-ngnnw\" (UID: \"fd7f43d9-6c38-4afe-9d46-62ce207e5e92\") " pod="openshift-ingress/e2e-mt-df4149da-openshift-default-6d4688bf67-ngnnw" Jul 16 00:02:47.104226 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:02:47.104207 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd7f43d9-6c38-4afe-9d46-62ce207e5e92-istio-token\") pod \"e2e-mt-df4149da-openshift-default-6d4688bf67-ngnnw\" (UID: \"fd7f43d9-6c38-4afe-9d46-62ce207e5e92\") " pod="openshift-ingress/e2e-mt-df4149da-openshift-default-6d4688bf67-ngnnw" Jul 16 00:02:47.104456 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:02:47.104439 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9bxm4\" (UniqueName: \"kubernetes.io/projected/fd7f43d9-6c38-4afe-9d46-62ce207e5e92-kube-api-access-9bxm4\") pod \"e2e-mt-df4149da-openshift-default-6d4688bf67-ngnnw\" (UID: \"fd7f43d9-6c38-4afe-9d46-62ce207e5e92\") " pod="openshift-ingress/e2e-mt-df4149da-openshift-default-6d4688bf67-ngnnw" Jul 16 00:02:47.132541 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:02:47.132512 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-df4149da-openshift-default-6d4688bf67-ngnnw" Jul 16 00:02:47.276176 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:02:47.276151 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-df4149da-openshift-default-6d4688bf67-ngnnw"] Jul 16 00:02:47.278777 ip-10-0-128-197 kubenswrapper[2563]: W0716 00:02:47.278746 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd7f43d9_6c38_4afe_9d46_62ce207e5e92.slice/crio-60b6706c14427c3c28e7445417433a26c3b1e930e5310fe80e7d53890fee0d3e WatchSource:0}: Error finding container 60b6706c14427c3c28e7445417433a26c3b1e930e5310fe80e7d53890fee0d3e: Status 404 returned error can't find the container with id 60b6706c14427c3c28e7445417433a26c3b1e930e5310fe80e7d53890fee0d3e Jul 16 00:02:47.280955 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:02:47.280924 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 16 00:02:47.281262 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:02:47.281018 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 16 00:02:47.281262 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:02:47.281052 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 16 00:02:47.819756 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:02:47.819715 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-df4149da-openshift-default-6d4688bf67-ngnnw" event={"ID":"fd7f43d9-6c38-4afe-9d46-62ce207e5e92","Type":"ContainerStarted","Data":"30315615dc708bc010d6ae3adcb791985cee8e848080c715f630cbb633c28175"} Jul 16 00:02:47.819940 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:02:47.819810 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-df4149da-openshift-default-6d4688bf67-ngnnw" event={"ID":"fd7f43d9-6c38-4afe-9d46-62ce207e5e92","Type":"ContainerStarted","Data":"60b6706c14427c3c28e7445417433a26c3b1e930e5310fe80e7d53890fee0d3e"} Jul 16 00:02:47.839748 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:02:47.839699 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-df4149da-openshift-default-6d4688bf67-ngnnw" podStartSLOduration=1.839681025 podStartE2EDuration="1.839681025s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 00:02:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 00:02:47.837670492 +0000 UTC m=+1583.211128454" watchObservedRunningTime="2026-07-16 00:02:47.839681025 +0000 UTC m=+1583.213138987" Jul 16 00:02:48.133742 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:02:48.133658 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-df4149da-openshift-default-6d4688bf67-ngnnw" Jul 16 00:02:48.138651 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:02:48.138623 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-df4149da-openshift-default-6d4688bf67-ngnnw" Jul 16 00:02:48.709875 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:02:48.709834 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:02:48.809671 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:02:48.809632 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:02:48.825067 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:02:48.825039 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-df4149da-openshift-default-6d4688bf67-ngnnw" Jul 16 00:02:48.825960 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:02:48.825939 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-df4149da-openshift-default-6d4688bf67-ngnnw" Jul 16 00:02:48.964820 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:02:48.964731 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:02:49.064847 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:02:49.064812 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:02:49.507849 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:02:49.507819 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:02:49.607673 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:02:49.607638 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:02:54.216645 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:02:54.216603 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:02:54.857278 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:02:54.857243 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:02:54.960378 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:02:54.960345 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:02:55.287045 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:02:55.287010 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:03:00.826139 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:03:00.826107 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:03:00.928938 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:03:00.928909 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:03:01.499612 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:03:01.499583 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:03:03.848641 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:03:03.848058 2563 patch_prober.go:28] interesting pod/e2e-mt-df4149da-openshift-default-6d4688bf67-ngnnw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": read tcp 10.134.0.2:56430->10.134.0.43:15021: read: connection reset by peer" start-of-body= Jul 16 00:03:03.848641 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:03:03.848125 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-df4149da-openshift-default-6d4688bf67-ngnnw" podUID="fd7f43d9-6c38-4afe-9d46-62ce207e5e92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": read tcp 10.134.0.2:56430->10.134.0.43:15021: read: connection reset by peer" Jul 16 00:03:03.904017 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:03:03.903986 2563 generic.go:356] "Generic (PLEG): container finished" podID="fd7f43d9-6c38-4afe-9d46-62ce207e5e92" containerID="30315615dc708bc010d6ae3adcb791985cee8e848080c715f630cbb633c28175" exitCode=0 Jul 16 00:03:03.904127 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:03:03.903995 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-df4149da-openshift-default-6d4688bf67-ngnnw" event={"ID":"fd7f43d9-6c38-4afe-9d46-62ce207e5e92","Type":"ContainerDied","Data":"30315615dc708bc010d6ae3adcb791985cee8e848080c715f630cbb633c28175"} Jul 16 00:03:03.904504 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:03:03.904490 2563 scope.go:117] "RemoveContainer" containerID="30315615dc708bc010d6ae3adcb791985cee8e848080c715f630cbb633c28175" Jul 16 00:03:03.905505 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:03:03.905476 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 16 00:03:03.905570 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:03:03.905554 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 16 00:03:03.905608 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:03:03.905589 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 16 00:03:04.912517 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:03:04.912487 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-df4149da-openshift-default-6d4688bf67-ngnnw" event={"ID":"fd7f43d9-6c38-4afe-9d46-62ce207e5e92","Type":"ContainerStarted","Data":"4d9b423e6f6bb916ed8937541f4c067830cc2383accd2ab793618f3d4d305bdc"} Jul 16 00:03:05.133044 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:03:05.133006 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-df4149da-openshift-default-6d4688bf67-ngnnw" Jul 16 00:03:05.133282 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:03:05.133258 2563 patch_prober.go:28] interesting pod/e2e-mt-df4149da-openshift-default-6d4688bf67-ngnnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Jul 16 00:03:05.133336 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:03:05.133317 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-df4149da-openshift-default-6d4688bf67-ngnnw" podUID="fd7f43d9-6c38-4afe-9d46-62ce207e5e92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Jul 16 00:03:06.133221 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:03:06.133191 2563 patch_prober.go:28] interesting pod/e2e-mt-df4149da-openshift-default-6d4688bf67-ngnnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Jul 16 00:03:06.133630 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:03:06.133243 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-df4149da-openshift-default-6d4688bf67-ngnnw" podUID="fd7f43d9-6c38-4afe-9d46-62ce207e5e92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Jul 16 00:03:07.133463 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:03:07.133425 2563 patch_prober.go:28] interesting pod/e2e-mt-df4149da-openshift-default-6d4688bf67-ngnnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Jul 16 00:03:07.133937 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:03:07.133502 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-df4149da-openshift-default-6d4688bf67-ngnnw" podUID="fd7f43d9-6c38-4afe-9d46-62ce207e5e92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Jul 16 00:03:07.211991 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:03:07.207629 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:03:07.298923 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:03:07.298888 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:03:07.503778 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:03:07.503743 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-df4149da-openshift-default-6d4688bf67-ngnnw"] Jul 16 00:03:07.504268 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:03:07.504236 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-df4149da-openshift-default-6d4688bf67-ngnnw" podUID="fd7f43d9-6c38-4afe-9d46-62ce207e5e92" containerName="istio-proxy" containerID="cri-o://4d9b423e6f6bb916ed8937541f4c067830cc2383accd2ab793618f3d4d305bdc" gracePeriod=30 Jul 16 00:03:07.529834 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:03:07.529810 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:03:07.669246 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:03:07.669214 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:03:07.808095 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:03:07.807924 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:03:07.907069 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:03:07.907034 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:03:08.007009 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:03:08.006950 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:03:13.569015 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:03:13.568951 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:03:13.630734 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:03:13.630708 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:03:13.706056 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:03:13.706024 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:03:13.808772 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:03:13.808739 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:03:13.909112 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:03:13.909044 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:03:14.363541 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:03:14.363518 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-df4149da-openshift-default-6d4688bf67-ngnnw" Jul 16 00:03:14.480268 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:03:14.480244 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd7f43d9-6c38-4afe-9d46-62ce207e5e92-istio-podinfo\") pod \"fd7f43d9-6c38-4afe-9d46-62ce207e5e92\" (UID: \"fd7f43d9-6c38-4afe-9d46-62ce207e5e92\") " Jul 16 00:03:14.480415 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:03:14.480287 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd7f43d9-6c38-4afe-9d46-62ce207e5e92-istiod-ca-cert\") pod \"fd7f43d9-6c38-4afe-9d46-62ce207e5e92\" (UID: \"fd7f43d9-6c38-4afe-9d46-62ce207e5e92\") " Jul 16 00:03:14.480415 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:03:14.480355 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd7f43d9-6c38-4afe-9d46-62ce207e5e92-workload-socket\") pod \"fd7f43d9-6c38-4afe-9d46-62ce207e5e92\" (UID: \"fd7f43d9-6c38-4afe-9d46-62ce207e5e92\") " Jul 16 00:03:14.480415 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:03:14.480388 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd7f43d9-6c38-4afe-9d46-62ce207e5e92-istio-envoy\") pod \"fd7f43d9-6c38-4afe-9d46-62ce207e5e92\" (UID: \"fd7f43d9-6c38-4afe-9d46-62ce207e5e92\") " Jul 16 00:03:14.480546 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:03:14.480515 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd7f43d9-6c38-4afe-9d46-62ce207e5e92-istio-data\") pod \"fd7f43d9-6c38-4afe-9d46-62ce207e5e92\" (UID: \"fd7f43d9-6c38-4afe-9d46-62ce207e5e92\") " Jul 16 00:03:14.480605 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:03:14.480564 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd7f43d9-6c38-4afe-9d46-62ce207e5e92-istio-token\") pod \"fd7f43d9-6c38-4afe-9d46-62ce207e5e92\" (UID: \"fd7f43d9-6c38-4afe-9d46-62ce207e5e92\") " Jul 16 00:03:14.480655 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:03:14.480623 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd7f43d9-6c38-4afe-9d46-62ce207e5e92-workload-certs\") pod \"fd7f43d9-6c38-4afe-9d46-62ce207e5e92\" (UID: \"fd7f43d9-6c38-4afe-9d46-62ce207e5e92\") " Jul 16 00:03:14.480701 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:03:14.480678 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9bxm4\" (UniqueName: \"kubernetes.io/projected/fd7f43d9-6c38-4afe-9d46-62ce207e5e92-kube-api-access-9bxm4\") pod \"fd7f43d9-6c38-4afe-9d46-62ce207e5e92\" (UID: \"fd7f43d9-6c38-4afe-9d46-62ce207e5e92\") " Jul 16 00:03:14.480759 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:03:14.480717 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd7f43d9-6c38-4afe-9d46-62ce207e5e92-credential-socket\") pod \"fd7f43d9-6c38-4afe-9d46-62ce207e5e92\" (UID: \"fd7f43d9-6c38-4afe-9d46-62ce207e5e92\") " Jul 16 00:03:14.480759 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:03:14.480725 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd7f43d9-6c38-4afe-9d46-62ce207e5e92-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "fd7f43d9-6c38-4afe-9d46-62ce207e5e92" (UID: "fd7f43d9-6c38-4afe-9d46-62ce207e5e92"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 00:03:14.481069 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:03:14.480927 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd7f43d9-6c38-4afe-9d46-62ce207e5e92-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "fd7f43d9-6c38-4afe-9d46-62ce207e5e92" (UID: "fd7f43d9-6c38-4afe-9d46-62ce207e5e92"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 00:03:14.481197 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:03:14.481161 2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd7f43d9-6c38-4afe-9d46-62ce207e5e92-workload-certs\") on node \"ip-10-0-128-197.ec2.internal\" DevicePath \"\"" Jul 16 00:03:14.481197 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:03:14.481184 2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd7f43d9-6c38-4afe-9d46-62ce207e5e92-workload-socket\") on node \"ip-10-0-128-197.ec2.internal\" DevicePath \"\"" Jul 16 00:03:14.481304 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:03:14.481259 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd7f43d9-6c38-4afe-9d46-62ce207e5e92-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "fd7f43d9-6c38-4afe-9d46-62ce207e5e92" (UID: "fd7f43d9-6c38-4afe-9d46-62ce207e5e92"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 00:03:14.481457 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:03:14.481427 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd7f43d9-6c38-4afe-9d46-62ce207e5e92-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "fd7f43d9-6c38-4afe-9d46-62ce207e5e92" (UID: "fd7f43d9-6c38-4afe-9d46-62ce207e5e92"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 00:03:14.481600 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:03:14.481563 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd7f43d9-6c38-4afe-9d46-62ce207e5e92-istio-data" (OuterVolumeSpecName: "istio-data") pod "fd7f43d9-6c38-4afe-9d46-62ce207e5e92" (UID: "fd7f43d9-6c38-4afe-9d46-62ce207e5e92"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 00:03:14.482665 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:03:14.482632 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd7f43d9-6c38-4afe-9d46-62ce207e5e92-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "fd7f43d9-6c38-4afe-9d46-62ce207e5e92" (UID: "fd7f43d9-6c38-4afe-9d46-62ce207e5e92"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 00:03:14.482665 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:03:14.482651 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fd7f43d9-6c38-4afe-9d46-62ce207e5e92-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "fd7f43d9-6c38-4afe-9d46-62ce207e5e92" (UID: "fd7f43d9-6c38-4afe-9d46-62ce207e5e92"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 16 00:03:14.482820 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:03:14.482749 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd7f43d9-6c38-4afe-9d46-62ce207e5e92-istio-token" (OuterVolumeSpecName: "istio-token") pod "fd7f43d9-6c38-4afe-9d46-62ce207e5e92" (UID: "fd7f43d9-6c38-4afe-9d46-62ce207e5e92"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 00:03:14.482820 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:03:14.482748 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd7f43d9-6c38-4afe-9d46-62ce207e5e92-kube-api-access-9bxm4" (OuterVolumeSpecName: "kube-api-access-9bxm4") pod "fd7f43d9-6c38-4afe-9d46-62ce207e5e92" (UID: "fd7f43d9-6c38-4afe-9d46-62ce207e5e92"). InnerVolumeSpecName "kube-api-access-9bxm4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 00:03:14.581942 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:03:14.581916 2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd7f43d9-6c38-4afe-9d46-62ce207e5e92-credential-socket\") on node \"ip-10-0-128-197.ec2.internal\" DevicePath \"\"" Jul 16 00:03:14.581942 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:03:14.581939 2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd7f43d9-6c38-4afe-9d46-62ce207e5e92-istio-podinfo\") on node \"ip-10-0-128-197.ec2.internal\" DevicePath \"\"" Jul 16 00:03:14.581942 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:03:14.581949 2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd7f43d9-6c38-4afe-9d46-62ce207e5e92-istiod-ca-cert\") on node \"ip-10-0-128-197.ec2.internal\" DevicePath \"\"" Jul 16 00:03:14.582334 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:03:14.581957 2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd7f43d9-6c38-4afe-9d46-62ce207e5e92-istio-envoy\") on node \"ip-10-0-128-197.ec2.internal\" DevicePath \"\"" Jul 16 00:03:14.582334 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:03:14.581985 2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd7f43d9-6c38-4afe-9d46-62ce207e5e92-istio-data\") on node \"ip-10-0-128-197.ec2.internal\" DevicePath \"\"" Jul 16 00:03:14.582334 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:03:14.581993 2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd7f43d9-6c38-4afe-9d46-62ce207e5e92-istio-token\") on node \"ip-10-0-128-197.ec2.internal\" DevicePath \"\"" Jul 16 00:03:14.582334 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:03:14.582000 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9bxm4\" (UniqueName: \"kubernetes.io/projected/fd7f43d9-6c38-4afe-9d46-62ce207e5e92-kube-api-access-9bxm4\") on node \"ip-10-0-128-197.ec2.internal\" DevicePath \"\"" Jul 16 00:03:14.963527 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:03:14.963443 2563 generic.go:356] "Generic (PLEG): container finished" podID="fd7f43d9-6c38-4afe-9d46-62ce207e5e92" containerID="4d9b423e6f6bb916ed8937541f4c067830cc2383accd2ab793618f3d4d305bdc" exitCode=0 Jul 16 00:03:14.963527 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:03:14.963495 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-df4149da-openshift-default-6d4688bf67-ngnnw" Jul 16 00:03:14.963701 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:03:14.963530 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-df4149da-openshift-default-6d4688bf67-ngnnw" event={"ID":"fd7f43d9-6c38-4afe-9d46-62ce207e5e92","Type":"ContainerDied","Data":"4d9b423e6f6bb916ed8937541f4c067830cc2383accd2ab793618f3d4d305bdc"} Jul 16 00:03:14.963701 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:03:14.963586 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-df4149da-openshift-default-6d4688bf67-ngnnw" event={"ID":"fd7f43d9-6c38-4afe-9d46-62ce207e5e92","Type":"ContainerDied","Data":"60b6706c14427c3c28e7445417433a26c3b1e930e5310fe80e7d53890fee0d3e"} Jul 16 00:03:14.963701 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:03:14.963619 2563 scope.go:117] "RemoveContainer" containerID="4d9b423e6f6bb916ed8937541f4c067830cc2383accd2ab793618f3d4d305bdc" Jul 16 00:03:14.972955 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:03:14.972935 2563 scope.go:117] "RemoveContainer" containerID="30315615dc708bc010d6ae3adcb791985cee8e848080c715f630cbb633c28175" Jul 16 00:03:14.981064 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:03:14.981045 2563 scope.go:117] "RemoveContainer" containerID="4d9b423e6f6bb916ed8937541f4c067830cc2383accd2ab793618f3d4d305bdc" Jul 16 00:03:14.981339 ip-10-0-128-197 kubenswrapper[2563]: E0716 00:03:14.981318 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d9b423e6f6bb916ed8937541f4c067830cc2383accd2ab793618f3d4d305bdc\": container with ID starting with 4d9b423e6f6bb916ed8937541f4c067830cc2383accd2ab793618f3d4d305bdc not found: ID does not exist" containerID="4d9b423e6f6bb916ed8937541f4c067830cc2383accd2ab793618f3d4d305bdc" Jul 16 00:03:14.981404 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:03:14.981350 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d9b423e6f6bb916ed8937541f4c067830cc2383accd2ab793618f3d4d305bdc"} err="failed to get container status \"4d9b423e6f6bb916ed8937541f4c067830cc2383accd2ab793618f3d4d305bdc\": rpc error: code = NotFound desc = could not find container \"4d9b423e6f6bb916ed8937541f4c067830cc2383accd2ab793618f3d4d305bdc\": container with ID starting with 4d9b423e6f6bb916ed8937541f4c067830cc2383accd2ab793618f3d4d305bdc not found: ID does not exist" Jul 16 00:03:14.981404 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:03:14.981365 2563 scope.go:117] "RemoveContainer" containerID="30315615dc708bc010d6ae3adcb791985cee8e848080c715f630cbb633c28175" Jul 16 00:03:14.981617 ip-10-0-128-197 kubenswrapper[2563]: E0716 00:03:14.981600 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30315615dc708bc010d6ae3adcb791985cee8e848080c715f630cbb633c28175\": container with ID starting with 30315615dc708bc010d6ae3adcb791985cee8e848080c715f630cbb633c28175 not found: ID does not exist" containerID="30315615dc708bc010d6ae3adcb791985cee8e848080c715f630cbb633c28175" Jul 16 00:03:14.981671 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:03:14.981625 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30315615dc708bc010d6ae3adcb791985cee8e848080c715f630cbb633c28175"} err="failed to get container status \"30315615dc708bc010d6ae3adcb791985cee8e848080c715f630cbb633c28175\": rpc error: code = NotFound desc = could not find container \"30315615dc708bc010d6ae3adcb791985cee8e848080c715f630cbb633c28175\": container with ID starting with 30315615dc708bc010d6ae3adcb791985cee8e848080c715f630cbb633c28175 not found: ID does not exist" Jul 16 00:03:14.985198 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:03:14.985173 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-df4149da-openshift-default-6d4688bf67-ngnnw"] Jul 16 00:03:14.989106 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:03:14.989084 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-df4149da-openshift-default-6d4688bf67-ngnnw"] Jul 16 00:03:15.170216 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:03:15.170186 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fd7f43d9-6c38-4afe-9d46-62ce207e5e92" path="/var/lib/kubelet/pods/fd7f43d9-6c38-4afe-9d46-62ce207e5e92/volumes" Jul 16 00:03:19.732664 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:03:19.732632 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:03:19.829671 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:03:19.829638 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:03:27.273793 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:03:27.273737 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:03:27.370604 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:03:27.370566 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:03:27.468033 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:03:27.467983 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:03:28.166184 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:03:28.166155 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:03:28.265015 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:03:28.264983 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:03:28.366316 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:03:28.366283 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:03:29.753756 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:03:29.753721 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:03:35.358392 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:03:35.358357 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:03:35.458933 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:03:35.458901 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:03:36.035757 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:03:36.035710 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:03:41.858328 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:03:41.858297 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:03:41.941073 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:03:41.941042 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:03:42.087154 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:03:42.087123 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:03:42.214260 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:03:42.214174 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:03:47.639627 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:03:47.639593 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:03:47.738844 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:03:47.738810 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:03:52.681917 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:03:52.681879 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:03:52.925511 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:03:52.925474 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:04:13.873830 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:04:13.873801 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:04:19.743534 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:04:19.743499 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:04:19.844440 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:04:19.844412 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:04:20.085833 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:04:20.085756 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:04:20.372107 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:04:20.372032 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-d3fd63-openshift-default-59db69b4b5-qgt8b"] Jul 16 00:04:20.372634 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:04:20.372618 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fd7f43d9-6c38-4afe-9d46-62ce207e5e92" containerName="istio-proxy" Jul 16 00:04:20.372677 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:04:20.372638 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="fd7f43d9-6c38-4afe-9d46-62ce207e5e92" containerName="istio-proxy" Jul 16 00:04:20.372768 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:04:20.372757 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="fd7f43d9-6c38-4afe-9d46-62ce207e5e92" containerName="istio-proxy" Jul 16 00:04:20.372805 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:04:20.372776 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="fd7f43d9-6c38-4afe-9d46-62ce207e5e92" containerName="istio-proxy" Jul 16 00:04:20.376258 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:04:20.376237 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-d3fd63-openshift-default-59db69b4b5-qgt8b" Jul 16 00:04:20.379517 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:04:20.379492 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-d3fd63-openshift-default-dockercfg-xrp6n\"" Jul 16 00:04:20.390356 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:04:20.390124 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-d3fd63-openshift-default-59db69b4b5-qgt8b"] Jul 16 00:04:20.398372 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:04:20.398342 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:04:20.454595 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:04:20.454570 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:04:20.506321 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:04:20.506286 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c0ace2f-4c34-48df-a593-8b7102595bae-credential-socket\") pod \"e2e-inf-a-d3fd63-openshift-default-59db69b4b5-qgt8b\" (UID: \"6c0ace2f-4c34-48df-a593-8b7102595bae\") " pod="openshift-ingress/e2e-inf-a-d3fd63-openshift-default-59db69b4b5-qgt8b" Jul 16 00:04:20.506491 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:04:20.506333 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c0ace2f-4c34-48df-a593-8b7102595bae-istio-podinfo\") pod \"e2e-inf-a-d3fd63-openshift-default-59db69b4b5-qgt8b\" (UID: \"6c0ace2f-4c34-48df-a593-8b7102595bae\") " pod="openshift-ingress/e2e-inf-a-d3fd63-openshift-default-59db69b4b5-qgt8b" Jul 16 00:04:20.506491 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:04:20.506401 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c0ace2f-4c34-48df-a593-8b7102595bae-workload-socket\") pod \"e2e-inf-a-d3fd63-openshift-default-59db69b4b5-qgt8b\" (UID: \"6c0ace2f-4c34-48df-a593-8b7102595bae\") " pod="openshift-ingress/e2e-inf-a-d3fd63-openshift-default-59db69b4b5-qgt8b" Jul 16 00:04:20.506491 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:04:20.506430 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c0ace2f-4c34-48df-a593-8b7102595bae-istiod-ca-cert\") pod \"e2e-inf-a-d3fd63-openshift-default-59db69b4b5-qgt8b\" (UID: \"6c0ace2f-4c34-48df-a593-8b7102595bae\") " pod="openshift-ingress/e2e-inf-a-d3fd63-openshift-default-59db69b4b5-qgt8b" Jul 16 00:04:20.506491 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:04:20.506450 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c0ace2f-4c34-48df-a593-8b7102595bae-istio-envoy\") pod \"e2e-inf-a-d3fd63-openshift-default-59db69b4b5-qgt8b\" (UID: \"6c0ace2f-4c34-48df-a593-8b7102595bae\") " pod="openshift-ingress/e2e-inf-a-d3fd63-openshift-default-59db69b4b5-qgt8b" Jul 16 00:04:20.506491 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:04:20.506487 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c0ace2f-4c34-48df-a593-8b7102595bae-workload-certs\") pod \"e2e-inf-a-d3fd63-openshift-default-59db69b4b5-qgt8b\" (UID: \"6c0ace2f-4c34-48df-a593-8b7102595bae\") " pod="openshift-ingress/e2e-inf-a-d3fd63-openshift-default-59db69b4b5-qgt8b" Jul 16 00:04:20.506680 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:04:20.506535 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c0ace2f-4c34-48df-a593-8b7102595bae-istio-data\") pod \"e2e-inf-a-d3fd63-openshift-default-59db69b4b5-qgt8b\" (UID: \"6c0ace2f-4c34-48df-a593-8b7102595bae\") " pod="openshift-ingress/e2e-inf-a-d3fd63-openshift-default-59db69b4b5-qgt8b" Jul 16 00:04:20.506680 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:04:20.506558 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2mtqm\" (UniqueName: \"kubernetes.io/projected/6c0ace2f-4c34-48df-a593-8b7102595bae-kube-api-access-2mtqm\") pod \"e2e-inf-a-d3fd63-openshift-default-59db69b4b5-qgt8b\" (UID: \"6c0ace2f-4c34-48df-a593-8b7102595bae\") " pod="openshift-ingress/e2e-inf-a-d3fd63-openshift-default-59db69b4b5-qgt8b" Jul 16 00:04:20.506680 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:04:20.506585 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c0ace2f-4c34-48df-a593-8b7102595bae-istio-token\") pod \"e2e-inf-a-d3fd63-openshift-default-59db69b4b5-qgt8b\" (UID: \"6c0ace2f-4c34-48df-a593-8b7102595bae\") " pod="openshift-ingress/e2e-inf-a-d3fd63-openshift-default-59db69b4b5-qgt8b" Jul 16 00:04:20.607737 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:04:20.607705 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c0ace2f-4c34-48df-a593-8b7102595bae-workload-socket\") pod \"e2e-inf-a-d3fd63-openshift-default-59db69b4b5-qgt8b\" (UID: \"6c0ace2f-4c34-48df-a593-8b7102595bae\") " pod="openshift-ingress/e2e-inf-a-d3fd63-openshift-default-59db69b4b5-qgt8b" Jul 16 00:04:20.607737 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:04:20.607738 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c0ace2f-4c34-48df-a593-8b7102595bae-istiod-ca-cert\") pod \"e2e-inf-a-d3fd63-openshift-default-59db69b4b5-qgt8b\" (UID: \"6c0ace2f-4c34-48df-a593-8b7102595bae\") " pod="openshift-ingress/e2e-inf-a-d3fd63-openshift-default-59db69b4b5-qgt8b" Jul 16 00:04:20.607930 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:04:20.607759 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c0ace2f-4c34-48df-a593-8b7102595bae-istio-envoy\") pod \"e2e-inf-a-d3fd63-openshift-default-59db69b4b5-qgt8b\" (UID: \"6c0ace2f-4c34-48df-a593-8b7102595bae\") " pod="openshift-ingress/e2e-inf-a-d3fd63-openshift-default-59db69b4b5-qgt8b" Jul 16 00:04:20.607930 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:04:20.607857 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c0ace2f-4c34-48df-a593-8b7102595bae-workload-certs\") pod \"e2e-inf-a-d3fd63-openshift-default-59db69b4b5-qgt8b\" (UID: \"6c0ace2f-4c34-48df-a593-8b7102595bae\") " pod="openshift-ingress/e2e-inf-a-d3fd63-openshift-default-59db69b4b5-qgt8b" Jul 16 00:04:20.607930 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:04:20.607879 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c0ace2f-4c34-48df-a593-8b7102595bae-istio-data\") pod \"e2e-inf-a-d3fd63-openshift-default-59db69b4b5-qgt8b\" (UID: \"6c0ace2f-4c34-48df-a593-8b7102595bae\") " pod="openshift-ingress/e2e-inf-a-d3fd63-openshift-default-59db69b4b5-qgt8b" Jul 16 00:04:20.607930 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:04:20.607896 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2mtqm\" (UniqueName: \"kubernetes.io/projected/6c0ace2f-4c34-48df-a593-8b7102595bae-kube-api-access-2mtqm\") pod \"e2e-inf-a-d3fd63-openshift-default-59db69b4b5-qgt8b\" (UID: \"6c0ace2f-4c34-48df-a593-8b7102595bae\") " pod="openshift-ingress/e2e-inf-a-d3fd63-openshift-default-59db69b4b5-qgt8b" Jul 16 00:04:20.607930 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:04:20.607925 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c0ace2f-4c34-48df-a593-8b7102595bae-istio-token\") pod \"e2e-inf-a-d3fd63-openshift-default-59db69b4b5-qgt8b\" (UID: \"6c0ace2f-4c34-48df-a593-8b7102595bae\") " pod="openshift-ingress/e2e-inf-a-d3fd63-openshift-default-59db69b4b5-qgt8b" Jul 16 00:04:20.608229 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:04:20.608012 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c0ace2f-4c34-48df-a593-8b7102595bae-credential-socket\") pod \"e2e-inf-a-d3fd63-openshift-default-59db69b4b5-qgt8b\" (UID: \"6c0ace2f-4c34-48df-a593-8b7102595bae\") " pod="openshift-ingress/e2e-inf-a-d3fd63-openshift-default-59db69b4b5-qgt8b" Jul 16 00:04:20.608229 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:04:20.608058 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c0ace2f-4c34-48df-a593-8b7102595bae-istio-podinfo\") pod \"e2e-inf-a-d3fd63-openshift-default-59db69b4b5-qgt8b\" (UID: \"6c0ace2f-4c34-48df-a593-8b7102595bae\") " pod="openshift-ingress/e2e-inf-a-d3fd63-openshift-default-59db69b4b5-qgt8b" Jul 16 00:04:20.608229 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:04:20.608177 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c0ace2f-4c34-48df-a593-8b7102595bae-workload-socket\") pod \"e2e-inf-a-d3fd63-openshift-default-59db69b4b5-qgt8b\" (UID: \"6c0ace2f-4c34-48df-a593-8b7102595bae\") " pod="openshift-ingress/e2e-inf-a-d3fd63-openshift-default-59db69b4b5-qgt8b" Jul 16 00:04:20.608374 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:04:20.608232 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c0ace2f-4c34-48df-a593-8b7102595bae-workload-certs\") pod \"e2e-inf-a-d3fd63-openshift-default-59db69b4b5-qgt8b\" (UID: \"6c0ace2f-4c34-48df-a593-8b7102595bae\") " pod="openshift-ingress/e2e-inf-a-d3fd63-openshift-default-59db69b4b5-qgt8b" Jul 16 00:04:20.608454 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:04:20.608435 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c0ace2f-4c34-48df-a593-8b7102595bae-credential-socket\") pod \"e2e-inf-a-d3fd63-openshift-default-59db69b4b5-qgt8b\" (UID: \"6c0ace2f-4c34-48df-a593-8b7102595bae\") " pod="openshift-ingress/e2e-inf-a-d3fd63-openshift-default-59db69b4b5-qgt8b" Jul 16 00:04:20.608557 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:04:20.608536 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c0ace2f-4c34-48df-a593-8b7102595bae-istio-data\") pod \"e2e-inf-a-d3fd63-openshift-default-59db69b4b5-qgt8b\" (UID: \"6c0ace2f-4c34-48df-a593-8b7102595bae\") " pod="openshift-ingress/e2e-inf-a-d3fd63-openshift-default-59db69b4b5-qgt8b" Jul 16 00:04:20.608598 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:04:20.608540 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c0ace2f-4c34-48df-a593-8b7102595bae-istiod-ca-cert\") pod \"e2e-inf-a-d3fd63-openshift-default-59db69b4b5-qgt8b\" (UID: \"6c0ace2f-4c34-48df-a593-8b7102595bae\") " pod="openshift-ingress/e2e-inf-a-d3fd63-openshift-default-59db69b4b5-qgt8b" Jul 16 00:04:20.610048 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:04:20.610032 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c0ace2f-4c34-48df-a593-8b7102595bae-istio-envoy\") pod \"e2e-inf-a-d3fd63-openshift-default-59db69b4b5-qgt8b\" (UID: \"6c0ace2f-4c34-48df-a593-8b7102595bae\") " pod="openshift-ingress/e2e-inf-a-d3fd63-openshift-default-59db69b4b5-qgt8b" Jul 16 00:04:20.610300 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:04:20.610285 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c0ace2f-4c34-48df-a593-8b7102595bae-istio-podinfo\") pod \"e2e-inf-a-d3fd63-openshift-default-59db69b4b5-qgt8b\" (UID: \"6c0ace2f-4c34-48df-a593-8b7102595bae\") " pod="openshift-ingress/e2e-inf-a-d3fd63-openshift-default-59db69b4b5-qgt8b" Jul 16 00:04:20.615251 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:04:20.615232 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c0ace2f-4c34-48df-a593-8b7102595bae-istio-token\") pod \"e2e-inf-a-d3fd63-openshift-default-59db69b4b5-qgt8b\" (UID: \"6c0ace2f-4c34-48df-a593-8b7102595bae\") " pod="openshift-ingress/e2e-inf-a-d3fd63-openshift-default-59db69b4b5-qgt8b" Jul 16 00:04:20.615510 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:04:20.615492 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2mtqm\" (UniqueName: \"kubernetes.io/projected/6c0ace2f-4c34-48df-a593-8b7102595bae-kube-api-access-2mtqm\") pod \"e2e-inf-a-d3fd63-openshift-default-59db69b4b5-qgt8b\" (UID: \"6c0ace2f-4c34-48df-a593-8b7102595bae\") " pod="openshift-ingress/e2e-inf-a-d3fd63-openshift-default-59db69b4b5-qgt8b" Jul 16 00:04:20.694650 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:04:20.694590 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-d3fd63-openshift-default-59db69b4b5-qgt8b" Jul 16 00:04:20.826356 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:04:20.826330 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-d3fd63-openshift-default-59db69b4b5-qgt8b"] Jul 16 00:04:20.827852 ip-10-0-128-197 kubenswrapper[2563]: W0716 00:04:20.827828 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c0ace2f_4c34_48df_a593_8b7102595bae.slice/crio-a98078a12419b9017b82f1482b209963abb37f9d1d1d3f734beca7b47855f176 WatchSource:0}: Error finding container a98078a12419b9017b82f1482b209963abb37f9d1d1d3f734beca7b47855f176: Status 404 returned error can't find the container with id a98078a12419b9017b82f1482b209963abb37f9d1d1d3f734beca7b47855f176 Jul 16 00:04:20.830022 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:04:20.829987 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 16 00:04:20.830125 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:04:20.830054 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 16 00:04:20.830125 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:04:20.830088 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 16 00:04:21.293682 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:04:21.293648 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d3fd63-openshift-default-59db69b4b5-qgt8b" event={"ID":"6c0ace2f-4c34-48df-a593-8b7102595bae","Type":"ContainerStarted","Data":"8f5a94152b5f0a4a3a1db97581fc474c9933f118aa445936cf22399a850bbd81"} Jul 16 00:04:21.293893 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:04:21.293707 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d3fd63-openshift-default-59db69b4b5-qgt8b" event={"ID":"6c0ace2f-4c34-48df-a593-8b7102595bae","Type":"ContainerStarted","Data":"a98078a12419b9017b82f1482b209963abb37f9d1d1d3f734beca7b47855f176"} Jul 16 00:04:21.313095 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:04:21.313044 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-d3fd63-openshift-default-59db69b4b5-qgt8b" podStartSLOduration=1.3130276539999999 podStartE2EDuration="1.313027654s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 00:04:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 00:04:21.311408074 +0000 UTC m=+1676.684866042" watchObservedRunningTime="2026-07-16 00:04:21.313027654 +0000 UTC m=+1676.686485617" Jul 16 00:04:21.695010 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:04:21.694922 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-d3fd63-openshift-default-59db69b4b5-qgt8b" Jul 16 00:04:21.699533 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:04:21.699511 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-d3fd63-openshift-default-59db69b4b5-qgt8b" Jul 16 00:04:22.189789 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:04:22.189763 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:04:22.265338 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:04:22.265280 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:04:22.299494 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:04:22.299463 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-d3fd63-openshift-default-59db69b4b5-qgt8b" Jul 16 00:04:22.300399 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:04:22.300380 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-d3fd63-openshift-default-59db69b4b5-qgt8b" Jul 16 00:04:22.328411 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:04:22.328381 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:04:22.743068 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:04:22.743037 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:04:22.844022 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:04:22.843986 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:04:22.943628 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:04:22.943597 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:04:27.695848 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:04:27.695813 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:04:28.019295 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:04:28.019257 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:04:28.289560 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:04:28.289470 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-dd8441-openshift-default-687bb5d445-zkmrm"] Jul 16 00:04:28.289875 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:04:28.289862 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fd7f43d9-6c38-4afe-9d46-62ce207e5e92" containerName="istio-proxy" Jul 16 00:04:28.289875 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:04:28.289874 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="fd7f43d9-6c38-4afe-9d46-62ce207e5e92" containerName="istio-proxy" Jul 16 00:04:28.293040 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:04:28.293020 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-dd8441-openshift-default-687bb5d445-zkmrm" Jul 16 00:04:28.295345 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:04:28.295325 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-dd8441-openshift-default-dockercfg-jfdzm\"" Jul 16 00:04:28.303092 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:04:28.303065 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-dd8441-openshift-default-687bb5d445-zkmrm"] Jul 16 00:04:28.312350 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:04:28.312325 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:04:28.382555 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:04:28.382506 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30af4109-dbe8-4f4d-9778-6d8580cf9b2d-istiod-ca-cert\") pod \"e2e-inf-b-dd8441-openshift-default-687bb5d445-zkmrm\" (UID: \"30af4109-dbe8-4f4d-9778-6d8580cf9b2d\") " pod="openshift-ingress/e2e-inf-b-dd8441-openshift-default-687bb5d445-zkmrm" Jul 16 00:04:28.382732 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:04:28.382610 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30af4109-dbe8-4f4d-9778-6d8580cf9b2d-credential-socket\") pod \"e2e-inf-b-dd8441-openshift-default-687bb5d445-zkmrm\" (UID: \"30af4109-dbe8-4f4d-9778-6d8580cf9b2d\") " pod="openshift-ingress/e2e-inf-b-dd8441-openshift-default-687bb5d445-zkmrm" Jul 16 00:04:28.382732 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:04:28.382700 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30af4109-dbe8-4f4d-9778-6d8580cf9b2d-istio-token\") pod \"e2e-inf-b-dd8441-openshift-default-687bb5d445-zkmrm\" (UID: \"30af4109-dbe8-4f4d-9778-6d8580cf9b2d\") " pod="openshift-ingress/e2e-inf-b-dd8441-openshift-default-687bb5d445-zkmrm" Jul 16 00:04:28.382834 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:04:28.382735 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30af4109-dbe8-4f4d-9778-6d8580cf9b2d-workload-certs\") pod \"e2e-inf-b-dd8441-openshift-default-687bb5d445-zkmrm\" (UID: \"30af4109-dbe8-4f4d-9778-6d8580cf9b2d\") " pod="openshift-ingress/e2e-inf-b-dd8441-openshift-default-687bb5d445-zkmrm" Jul 16 00:04:28.382834 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:04:28.382787 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4nz8p\" (UniqueName: \"kubernetes.io/projected/30af4109-dbe8-4f4d-9778-6d8580cf9b2d-kube-api-access-4nz8p\") pod \"e2e-inf-b-dd8441-openshift-default-687bb5d445-zkmrm\" (UID: \"30af4109-dbe8-4f4d-9778-6d8580cf9b2d\") " pod="openshift-ingress/e2e-inf-b-dd8441-openshift-default-687bb5d445-zkmrm" Jul 16 00:04:28.382834 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:04:28.382823 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30af4109-dbe8-4f4d-9778-6d8580cf9b2d-istio-data\") pod \"e2e-inf-b-dd8441-openshift-default-687bb5d445-zkmrm\" (UID: \"30af4109-dbe8-4f4d-9778-6d8580cf9b2d\") " pod="openshift-ingress/e2e-inf-b-dd8441-openshift-default-687bb5d445-zkmrm" Jul 16 00:04:28.383015 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:04:28.382873 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30af4109-dbe8-4f4d-9778-6d8580cf9b2d-istio-envoy\") pod \"e2e-inf-b-dd8441-openshift-default-687bb5d445-zkmrm\" (UID: \"30af4109-dbe8-4f4d-9778-6d8580cf9b2d\") " pod="openshift-ingress/e2e-inf-b-dd8441-openshift-default-687bb5d445-zkmrm" Jul 16 00:04:28.383015 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:04:28.382909 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30af4109-dbe8-4f4d-9778-6d8580cf9b2d-istio-podinfo\") pod \"e2e-inf-b-dd8441-openshift-default-687bb5d445-zkmrm\" (UID: \"30af4109-dbe8-4f4d-9778-6d8580cf9b2d\") " pod="openshift-ingress/e2e-inf-b-dd8441-openshift-default-687bb5d445-zkmrm" Jul 16 00:04:28.383122 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:04:28.383012 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30af4109-dbe8-4f4d-9778-6d8580cf9b2d-workload-socket\") pod \"e2e-inf-b-dd8441-openshift-default-687bb5d445-zkmrm\" (UID: \"30af4109-dbe8-4f4d-9778-6d8580cf9b2d\") " pod="openshift-ingress/e2e-inf-b-dd8441-openshift-default-687bb5d445-zkmrm" Jul 16 00:04:28.483649 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:04:28.483615 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30af4109-dbe8-4f4d-9778-6d8580cf9b2d-istio-data\") pod \"e2e-inf-b-dd8441-openshift-default-687bb5d445-zkmrm\" (UID: \"30af4109-dbe8-4f4d-9778-6d8580cf9b2d\") " pod="openshift-ingress/e2e-inf-b-dd8441-openshift-default-687bb5d445-zkmrm" Jul 16 00:04:28.483649 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:04:28.483655 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30af4109-dbe8-4f4d-9778-6d8580cf9b2d-istio-envoy\") pod \"e2e-inf-b-dd8441-openshift-default-687bb5d445-zkmrm\" (UID: \"30af4109-dbe8-4f4d-9778-6d8580cf9b2d\") " pod="openshift-ingress/e2e-inf-b-dd8441-openshift-default-687bb5d445-zkmrm" Jul 16 00:04:28.483891 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:04:28.483678 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30af4109-dbe8-4f4d-9778-6d8580cf9b2d-istio-podinfo\") pod \"e2e-inf-b-dd8441-openshift-default-687bb5d445-zkmrm\" (UID: \"30af4109-dbe8-4f4d-9778-6d8580cf9b2d\") " pod="openshift-ingress/e2e-inf-b-dd8441-openshift-default-687bb5d445-zkmrm" Jul 16 00:04:28.483891 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:04:28.483796 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30af4109-dbe8-4f4d-9778-6d8580cf9b2d-workload-socket\") pod \"e2e-inf-b-dd8441-openshift-default-687bb5d445-zkmrm\" (UID: \"30af4109-dbe8-4f4d-9778-6d8580cf9b2d\") " pod="openshift-ingress/e2e-inf-b-dd8441-openshift-default-687bb5d445-zkmrm" Jul 16 00:04:28.483891 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:04:28.483845 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30af4109-dbe8-4f4d-9778-6d8580cf9b2d-istiod-ca-cert\") pod \"e2e-inf-b-dd8441-openshift-default-687bb5d445-zkmrm\" (UID: \"30af4109-dbe8-4f4d-9778-6d8580cf9b2d\") " pod="openshift-ingress/e2e-inf-b-dd8441-openshift-default-687bb5d445-zkmrm" Jul 16 00:04:28.484083 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:04:28.483916 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30af4109-dbe8-4f4d-9778-6d8580cf9b2d-credential-socket\") pod \"e2e-inf-b-dd8441-openshift-default-687bb5d445-zkmrm\" (UID: \"30af4109-dbe8-4f4d-9778-6d8580cf9b2d\") " pod="openshift-ingress/e2e-inf-b-dd8441-openshift-default-687bb5d445-zkmrm" Jul 16 00:04:28.484083 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:04:28.484035 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30af4109-dbe8-4f4d-9778-6d8580cf9b2d-istio-token\") pod \"e2e-inf-b-dd8441-openshift-default-687bb5d445-zkmrm\" (UID: \"30af4109-dbe8-4f4d-9778-6d8580cf9b2d\") " pod="openshift-ingress/e2e-inf-b-dd8441-openshift-default-687bb5d445-zkmrm" Jul 16 00:04:28.484083 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:04:28.484069 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30af4109-dbe8-4f4d-9778-6d8580cf9b2d-istio-data\") pod \"e2e-inf-b-dd8441-openshift-default-687bb5d445-zkmrm\" (UID: \"30af4109-dbe8-4f4d-9778-6d8580cf9b2d\") " pod="openshift-ingress/e2e-inf-b-dd8441-openshift-default-687bb5d445-zkmrm" Jul 16 00:04:28.484233 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:04:28.484135 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30af4109-dbe8-4f4d-9778-6d8580cf9b2d-workload-socket\") pod \"e2e-inf-b-dd8441-openshift-default-687bb5d445-zkmrm\" (UID: \"30af4109-dbe8-4f4d-9778-6d8580cf9b2d\") " pod="openshift-ingress/e2e-inf-b-dd8441-openshift-default-687bb5d445-zkmrm" Jul 16 00:04:28.484339 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:04:28.484313 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30af4109-dbe8-4f4d-9778-6d8580cf9b2d-workload-certs\") pod \"e2e-inf-b-dd8441-openshift-default-687bb5d445-zkmrm\" (UID: \"30af4109-dbe8-4f4d-9778-6d8580cf9b2d\") " pod="openshift-ingress/e2e-inf-b-dd8441-openshift-default-687bb5d445-zkmrm" Jul 16 00:04:28.484339 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:04:28.484324 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30af4109-dbe8-4f4d-9778-6d8580cf9b2d-credential-socket\") pod \"e2e-inf-b-dd8441-openshift-default-687bb5d445-zkmrm\" (UID: \"30af4109-dbe8-4f4d-9778-6d8580cf9b2d\") " pod="openshift-ingress/e2e-inf-b-dd8441-openshift-default-687bb5d445-zkmrm" Jul 16 00:04:28.484523 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:04:28.484386 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30af4109-dbe8-4f4d-9778-6d8580cf9b2d-workload-certs\") pod \"e2e-inf-b-dd8441-openshift-default-687bb5d445-zkmrm\" (UID: \"30af4109-dbe8-4f4d-9778-6d8580cf9b2d\") " pod="openshift-ingress/e2e-inf-b-dd8441-openshift-default-687bb5d445-zkmrm" Jul 16 00:04:28.484523 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:04:28.484399 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4nz8p\" (UniqueName: \"kubernetes.io/projected/30af4109-dbe8-4f4d-9778-6d8580cf9b2d-kube-api-access-4nz8p\") pod \"e2e-inf-b-dd8441-openshift-default-687bb5d445-zkmrm\" (UID: \"30af4109-dbe8-4f4d-9778-6d8580cf9b2d\") " pod="openshift-ingress/e2e-inf-b-dd8441-openshift-default-687bb5d445-zkmrm" Jul 16 00:04:28.484686 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:04:28.484657 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30af4109-dbe8-4f4d-9778-6d8580cf9b2d-istiod-ca-cert\") pod \"e2e-inf-b-dd8441-openshift-default-687bb5d445-zkmrm\" (UID: \"30af4109-dbe8-4f4d-9778-6d8580cf9b2d\") " pod="openshift-ingress/e2e-inf-b-dd8441-openshift-default-687bb5d445-zkmrm" Jul 16 00:04:28.486335 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:04:28.486315 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30af4109-dbe8-4f4d-9778-6d8580cf9b2d-istio-envoy\") pod \"e2e-inf-b-dd8441-openshift-default-687bb5d445-zkmrm\" (UID: \"30af4109-dbe8-4f4d-9778-6d8580cf9b2d\") " pod="openshift-ingress/e2e-inf-b-dd8441-openshift-default-687bb5d445-zkmrm" Jul 16 00:04:28.486554 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:04:28.486526 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30af4109-dbe8-4f4d-9778-6d8580cf9b2d-istio-podinfo\") pod \"e2e-inf-b-dd8441-openshift-default-687bb5d445-zkmrm\" (UID: \"30af4109-dbe8-4f4d-9778-6d8580cf9b2d\") " pod="openshift-ingress/e2e-inf-b-dd8441-openshift-default-687bb5d445-zkmrm" Jul 16 00:04:28.491044 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:04:28.491017 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:04:28.491782 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:04:28.491757 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30af4109-dbe8-4f4d-9778-6d8580cf9b2d-istio-token\") pod \"e2e-inf-b-dd8441-openshift-default-687bb5d445-zkmrm\" (UID: \"30af4109-dbe8-4f4d-9778-6d8580cf9b2d\") " pod="openshift-ingress/e2e-inf-b-dd8441-openshift-default-687bb5d445-zkmrm" Jul 16 00:04:28.492030 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:04:28.491958 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4nz8p\" (UniqueName: \"kubernetes.io/projected/30af4109-dbe8-4f4d-9778-6d8580cf9b2d-kube-api-access-4nz8p\") pod \"e2e-inf-b-dd8441-openshift-default-687bb5d445-zkmrm\" (UID: \"30af4109-dbe8-4f4d-9778-6d8580cf9b2d\") " pod="openshift-ingress/e2e-inf-b-dd8441-openshift-default-687bb5d445-zkmrm" Jul 16 00:04:28.604067 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:04:28.603980 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-dd8441-openshift-default-687bb5d445-zkmrm" Jul 16 00:04:28.735903 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:04:28.735873 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-dd8441-openshift-default-687bb5d445-zkmrm"] Jul 16 00:04:28.737974 ip-10-0-128-197 kubenswrapper[2563]: W0716 00:04:28.737942 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30af4109_dbe8_4f4d_9778_6d8580cf9b2d.slice/crio-6c7a0407038c143e7b1f34e770652f3ba032d0fab442aac6b4fc9a21afad5a37 WatchSource:0}: Error finding container 6c7a0407038c143e7b1f34e770652f3ba032d0fab442aac6b4fc9a21afad5a37: Status 404 returned error can't find the container with id 6c7a0407038c143e7b1f34e770652f3ba032d0fab442aac6b4fc9a21afad5a37 Jul 16 00:04:28.740329 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:04:28.740295 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 16 00:04:28.740450 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:04:28.740370 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 16 00:04:28.740450 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:04:28.740416 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 16 00:04:29.334218 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:04:29.334184 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-dd8441-openshift-default-687bb5d445-zkmrm" event={"ID":"30af4109-dbe8-4f4d-9778-6d8580cf9b2d","Type":"ContainerStarted","Data":"d200119b763859d1bd570c15a43558ed2e96f891d0f33e65a1c53edc8e2444ee"} Jul 16 00:04:29.334410 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:04:29.334231 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-dd8441-openshift-default-687bb5d445-zkmrm" event={"ID":"30af4109-dbe8-4f4d-9778-6d8580cf9b2d","Type":"ContainerStarted","Data":"6c7a0407038c143e7b1f34e770652f3ba032d0fab442aac6b4fc9a21afad5a37"} Jul 16 00:04:29.356153 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:04:29.356106 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-dd8441-openshift-default-687bb5d445-zkmrm" podStartSLOduration=1.356087606 podStartE2EDuration="1.356087606s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 00:04:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 00:04:29.355406384 +0000 UTC m=+1684.728864348" watchObservedRunningTime="2026-07-16 00:04:29.356087606 +0000 UTC m=+1684.729545585" Jul 16 00:04:29.605084 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:04:29.605004 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-dd8441-openshift-default-687bb5d445-zkmrm" Jul 16 00:04:29.609976 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:04:29.609939 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-dd8441-openshift-default-687bb5d445-zkmrm" Jul 16 00:04:30.208252 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:04:30.208217 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:04:30.297681 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:04:30.297653 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:04:30.339352 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:04:30.339325 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-dd8441-openshift-default-687bb5d445-zkmrm" Jul 16 00:04:30.340347 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:04:30.340322 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-dd8441-openshift-default-687bb5d445-zkmrm" Jul 16 00:04:30.366491 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:04:30.366465 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:04:30.788083 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:04:30.788048 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:04:30.885476 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:04:30.885443 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:04:30.985112 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:04:30.985081 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:04:35.729587 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:04:35.729555 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:04:41.400666 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:04:41.400564 2563 generic.go:356] "Generic (PLEG): container finished" podID="6c0ace2f-4c34-48df-a593-8b7102595bae" containerID="8f5a94152b5f0a4a3a1db97581fc474c9933f118aa445936cf22399a850bbd81" exitCode=0 Jul 16 00:04:41.401195 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:04:41.400671 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d3fd63-openshift-default-59db69b4b5-qgt8b" event={"ID":"6c0ace2f-4c34-48df-a593-8b7102595bae","Type":"ContainerDied","Data":"8f5a94152b5f0a4a3a1db97581fc474c9933f118aa445936cf22399a850bbd81"} Jul 16 00:04:41.401195 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:04:41.401164 2563 scope.go:117] "RemoveContainer" containerID="8f5a94152b5f0a4a3a1db97581fc474c9933f118aa445936cf22399a850bbd81" Jul 16 00:04:41.407480 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:04:41.407435 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 16 00:04:41.407601 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:04:41.407536 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 16 00:04:41.407601 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:04:41.407587 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 16 00:04:42.411866 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:04:42.411818 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d3fd63-openshift-default-59db69b4b5-qgt8b" event={"ID":"6c0ace2f-4c34-48df-a593-8b7102595bae","Type":"ContainerStarted","Data":"ec518538d31cc527f4b8a62face68e4c2bb66072bce21c7faade768ddb722e44"} Jul 16 00:04:42.696077 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:04:42.695958 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-d3fd63-openshift-default-59db69b4b5-qgt8b" Jul 16 00:04:42.696444 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:04:42.696352 2563 patch_prober.go:28] interesting pod/e2e-inf-a-d3fd63-openshift-default-59db69b4b5-qgt8b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Jul 16 00:04:42.696615 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:04:42.696592 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d3fd63-openshift-default-59db69b4b5-qgt8b" podUID="6c0ace2f-4c34-48df-a593-8b7102595bae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Jul 16 00:04:43.495898 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:04:43.495844 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:04:43.596767 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:04:43.596692 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:04:43.695488 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:04:43.695019 2563 patch_prober.go:28] interesting pod/e2e-inf-a-d3fd63-openshift-default-59db69b4b5-qgt8b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Jul 16 00:04:43.695488 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:04:43.695082 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d3fd63-openshift-default-59db69b4b5-qgt8b" podUID="6c0ace2f-4c34-48df-a593-8b7102595bae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Jul 16 00:04:44.695114 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:04:44.695070 2563 patch_prober.go:28] interesting pod/e2e-inf-a-d3fd63-openshift-default-59db69b4b5-qgt8b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Jul 16 00:04:44.695603 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:04:44.695138 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d3fd63-openshift-default-59db69b4b5-qgt8b" podUID="6c0ace2f-4c34-48df-a593-8b7102595bae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Jul 16 00:04:45.696017 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:04:45.695956 2563 patch_prober.go:28] interesting pod/e2e-inf-a-d3fd63-openshift-default-59db69b4b5-qgt8b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Jul 16 00:04:45.696496 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:04:45.696048 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d3fd63-openshift-default-59db69b4b5-qgt8b" podUID="6c0ace2f-4c34-48df-a593-8b7102595bae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Jul 16 00:04:46.380139 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:04:46.380098 2563 patch_prober.go:28] interesting pod/e2e-inf-b-dd8441-openshift-default-687bb5d445-zkmrm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 16 00:04:46.380321 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:04:46.380165 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-dd8441-openshift-default-687bb5d445-zkmrm" podUID="30af4109-dbe8-4f4d-9778-6d8580cf9b2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 16 00:04:46.699048 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:04:46.698957 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-d3fd63-openshift-default-59db69b4b5-qgt8b" Jul 16 00:04:46.699398 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:04:46.699345 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-d3fd63-openshift-default-59db69b4b5-qgt8b" Jul 16 00:04:46.700085 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:04:46.700067 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-d3fd63-openshift-default-59db69b4b5-qgt8b" Jul 16 00:04:47.441511 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:04:47.441410 2563 generic.go:356] "Generic (PLEG): container finished" podID="30af4109-dbe8-4f4d-9778-6d8580cf9b2d" containerID="d200119b763859d1bd570c15a43558ed2e96f891d0f33e65a1c53edc8e2444ee" exitCode=0 Jul 16 00:04:47.441511 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:04:47.441488 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-dd8441-openshift-default-687bb5d445-zkmrm" event={"ID":"30af4109-dbe8-4f4d-9778-6d8580cf9b2d","Type":"ContainerDied","Data":"d200119b763859d1bd570c15a43558ed2e96f891d0f33e65a1c53edc8e2444ee"} Jul 16 00:04:47.442102 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:04:47.442084 2563 scope.go:117] "RemoveContainer" containerID="d200119b763859d1bd570c15a43558ed2e96f891d0f33e65a1c53edc8e2444ee" Jul 16 00:04:47.443136 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:04:47.443106 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 16 00:04:47.443254 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:04:47.443186 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 16 00:04:47.443254 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:04:47.443233 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 16 00:04:48.448858 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:04:48.448823 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-dd8441-openshift-default-687bb5d445-zkmrm" event={"ID":"30af4109-dbe8-4f4d-9778-6d8580cf9b2d","Type":"ContainerStarted","Data":"cef0b425c46b87d99e17d1c8b8ff5195f21a176150dc10729afc44ce6860b39b"} Jul 16 00:04:48.605398 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:04:48.605364 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-dd8441-openshift-default-687bb5d445-zkmrm" Jul 16 00:04:48.605751 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:04:48.605730 2563 patch_prober.go:28] interesting pod/e2e-inf-b-dd8441-openshift-default-687bb5d445-zkmrm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Jul 16 00:04:48.605925 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:04:48.605898 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-dd8441-openshift-default-687bb5d445-zkmrm" podUID="30af4109-dbe8-4f4d-9778-6d8580cf9b2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Jul 16 00:04:48.880274 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:04:48.880238 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:04:49.605321 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:04:49.605283 2563 patch_prober.go:28] interesting pod/e2e-inf-b-dd8441-openshift-default-687bb5d445-zkmrm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Jul 16 00:04:49.605930 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:04:49.605897 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-dd8441-openshift-default-687bb5d445-zkmrm" podUID="30af4109-dbe8-4f4d-9778-6d8580cf9b2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Jul 16 00:04:50.604589 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:04:50.604548 2563 patch_prober.go:28] interesting pod/e2e-inf-b-dd8441-openshift-default-687bb5d445-zkmrm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Jul 16 00:04:50.604800 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:04:50.604621 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-dd8441-openshift-default-687bb5d445-zkmrm" podUID="30af4109-dbe8-4f4d-9778-6d8580cf9b2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Jul 16 00:04:51.605045 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:04:51.605009 2563 patch_prober.go:28] interesting pod/e2e-inf-b-dd8441-openshift-default-687bb5d445-zkmrm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Jul 16 00:04:51.606144 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:04:51.605719 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-dd8441-openshift-default-687bb5d445-zkmrm" podUID="30af4109-dbe8-4f4d-9778-6d8580cf9b2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Jul 16 00:04:51.749294 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:04:51.749251 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:04:51.845993 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:04:51.845947 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:04:52.608386 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:04:52.608355 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-dd8441-openshift-default-687bb5d445-zkmrm" Jul 16 00:04:52.608802 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:04:52.608643 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-dd8441-openshift-default-687bb5d445-zkmrm" Jul 16 00:04:52.609396 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:04:52.609380 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-dd8441-openshift-default-687bb5d445-zkmrm" Jul 16 00:05:19.627142 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:05:19.627101 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:05:29.755236 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:05:29.755204 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:05:30.032406 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:05:30.032316 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:05:49.454785 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:05:49.454750 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:05:50.329132 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:05:50.329091 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:05:50.807413 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:05:50.807379 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:05:55.884302 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:05:55.884271 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:06:25.152340 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:06:25.152303 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d8f6k_2f64cab3-104a-44f6-b125-ba458f54f058/ovn-acl-logging/0.log" Jul 16 00:06:25.154442 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:06:25.154422 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d8f6k_2f64cab3-104a-44f6-b125-ba458f54f058/ovn-acl-logging/0.log" Jul 16 00:06:32.580318 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:06:32.580280 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:06:32.889018 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:06:32.888914 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-d3fd63-openshift-default-59db69b4b5-qgt8b"] Jul 16 00:06:32.889164 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:06:32.889126 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-d3fd63-openshift-default-59db69b4b5-qgt8b" podUID="6c0ace2f-4c34-48df-a593-8b7102595bae" containerName="istio-proxy" containerID="cri-o://ec518538d31cc527f4b8a62face68e4c2bb66072bce21c7faade768ddb722e44" gracePeriod=30 Jul 16 00:06:32.942210 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:06:32.942183 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:06:33.101709 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:06:33.101680 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:06:33.584751 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:06:33.584722 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:06:38.011118 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:06:38.011091 2563 generic.go:356] "Generic (PLEG): container finished" podID="6c0ace2f-4c34-48df-a593-8b7102595bae" containerID="ec518538d31cc527f4b8a62face68e4c2bb66072bce21c7faade768ddb722e44" exitCode=0 Jul 16 00:06:38.011433 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:06:38.011127 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d3fd63-openshift-default-59db69b4b5-qgt8b" event={"ID":"6c0ace2f-4c34-48df-a593-8b7102595bae","Type":"ContainerDied","Data":"ec518538d31cc527f4b8a62face68e4c2bb66072bce21c7faade768ddb722e44"} Jul 16 00:06:38.011433 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:06:38.011176 2563 scope.go:117] "RemoveContainer" containerID="8f5a94152b5f0a4a3a1db97581fc474c9933f118aa445936cf22399a850bbd81" Jul 16 00:06:38.139454 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:06:38.139430 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-d3fd63-openshift-default-59db69b4b5-qgt8b" Jul 16 00:06:38.265321 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:06:38.265297 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c0ace2f-4c34-48df-a593-8b7102595bae-credential-socket\") pod \"6c0ace2f-4c34-48df-a593-8b7102595bae\" (UID: \"6c0ace2f-4c34-48df-a593-8b7102595bae\") " Jul 16 00:06:38.265514 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:06:38.265331 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2mtqm\" (UniqueName: \"kubernetes.io/projected/6c0ace2f-4c34-48df-a593-8b7102595bae-kube-api-access-2mtqm\") pod \"6c0ace2f-4c34-48df-a593-8b7102595bae\" (UID: \"6c0ace2f-4c34-48df-a593-8b7102595bae\") " Jul 16 00:06:38.265514 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:06:38.265362 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c0ace2f-4c34-48df-a593-8b7102595bae-workload-socket\") pod \"6c0ace2f-4c34-48df-a593-8b7102595bae\" (UID: \"6c0ace2f-4c34-48df-a593-8b7102595bae\") " Jul 16 00:06:38.265514 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:06:38.265384 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c0ace2f-4c34-48df-a593-8b7102595bae-istio-podinfo\") pod \"6c0ace2f-4c34-48df-a593-8b7102595bae\" (UID: \"6c0ace2f-4c34-48df-a593-8b7102595bae\") " Jul 16 00:06:38.265514 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:06:38.265410 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c0ace2f-4c34-48df-a593-8b7102595bae-istiod-ca-cert\") pod \"6c0ace2f-4c34-48df-a593-8b7102595bae\" (UID: \"6c0ace2f-4c34-48df-a593-8b7102595bae\") " Jul 16 00:06:38.265514 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:06:38.265429 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c0ace2f-4c34-48df-a593-8b7102595bae-istio-data\") pod \"6c0ace2f-4c34-48df-a593-8b7102595bae\" (UID: \"6c0ace2f-4c34-48df-a593-8b7102595bae\") " Jul 16 00:06:38.265514 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:06:38.265451 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c0ace2f-4c34-48df-a593-8b7102595bae-workload-certs\") pod \"6c0ace2f-4c34-48df-a593-8b7102595bae\" (UID: \"6c0ace2f-4c34-48df-a593-8b7102595bae\") " Jul 16 00:06:38.265514 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:06:38.265476 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c0ace2f-4c34-48df-a593-8b7102595bae-istio-envoy\") pod \"6c0ace2f-4c34-48df-a593-8b7102595bae\" (UID: \"6c0ace2f-4c34-48df-a593-8b7102595bae\") " Jul 16 00:06:38.265813 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:06:38.265524 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c0ace2f-4c34-48df-a593-8b7102595bae-istio-token\") pod \"6c0ace2f-4c34-48df-a593-8b7102595bae\" (UID: \"6c0ace2f-4c34-48df-a593-8b7102595bae\") " Jul 16 00:06:38.265813 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:06:38.265584 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c0ace2f-4c34-48df-a593-8b7102595bae-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "6c0ace2f-4c34-48df-a593-8b7102595bae" (UID: "6c0ace2f-4c34-48df-a593-8b7102595bae"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 00:06:38.265813 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:06:38.265600 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c0ace2f-4c34-48df-a593-8b7102595bae-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "6c0ace2f-4c34-48df-a593-8b7102595bae" (UID: "6c0ace2f-4c34-48df-a593-8b7102595bae"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 00:06:38.266008 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:06:38.265878 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c0ace2f-4c34-48df-a593-8b7102595bae-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "6c0ace2f-4c34-48df-a593-8b7102595bae" (UID: "6c0ace2f-4c34-48df-a593-8b7102595bae"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 00:06:38.266076 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:06:38.266045 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c0ace2f-4c34-48df-a593-8b7102595bae-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "6c0ace2f-4c34-48df-a593-8b7102595bae" (UID: "6c0ace2f-4c34-48df-a593-8b7102595bae"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 00:06:38.266327 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:06:38.266281 2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c0ace2f-4c34-48df-a593-8b7102595bae-credential-socket\") on node \"ip-10-0-128-197.ec2.internal\" DevicePath \"\"" Jul 16 00:06:38.266327 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:06:38.266311 2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c0ace2f-4c34-48df-a593-8b7102595bae-workload-socket\") on node \"ip-10-0-128-197.ec2.internal\" DevicePath \"\"" Jul 16 00:06:38.266327 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:06:38.266328 2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c0ace2f-4c34-48df-a593-8b7102595bae-istiod-ca-cert\") on node \"ip-10-0-128-197.ec2.internal\" DevicePath \"\"" Jul 16 00:06:38.266769 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:06:38.266743 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c0ace2f-4c34-48df-a593-8b7102595bae-istio-data" (OuterVolumeSpecName: "istio-data") pod "6c0ace2f-4c34-48df-a593-8b7102595bae" (UID: "6c0ace2f-4c34-48df-a593-8b7102595bae"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 00:06:38.267772 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:06:38.267752 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6c0ace2f-4c34-48df-a593-8b7102595bae-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "6c0ace2f-4c34-48df-a593-8b7102595bae" (UID: "6c0ace2f-4c34-48df-a593-8b7102595bae"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 16 00:06:38.267861 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:06:38.267782 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c0ace2f-4c34-48df-a593-8b7102595bae-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "6c0ace2f-4c34-48df-a593-8b7102595bae" (UID: "6c0ace2f-4c34-48df-a593-8b7102595bae"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 00:06:38.268232 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:06:38.268211 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c0ace2f-4c34-48df-a593-8b7102595bae-istio-token" (OuterVolumeSpecName: "istio-token") pod "6c0ace2f-4c34-48df-a593-8b7102595bae" (UID: "6c0ace2f-4c34-48df-a593-8b7102595bae"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 00:06:38.268303 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:06:38.268236 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c0ace2f-4c34-48df-a593-8b7102595bae-kube-api-access-2mtqm" (OuterVolumeSpecName: "kube-api-access-2mtqm") pod "6c0ace2f-4c34-48df-a593-8b7102595bae" (UID: "6c0ace2f-4c34-48df-a593-8b7102595bae"). InnerVolumeSpecName "kube-api-access-2mtqm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 00:06:38.367668 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:06:38.367644 2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c0ace2f-4c34-48df-a593-8b7102595bae-istio-envoy\") on node \"ip-10-0-128-197.ec2.internal\" DevicePath \"\"" Jul 16 00:06:38.367668 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:06:38.367666 2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c0ace2f-4c34-48df-a593-8b7102595bae-istio-token\") on node \"ip-10-0-128-197.ec2.internal\" DevicePath \"\"" Jul 16 00:06:38.367827 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:06:38.367675 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2mtqm\" (UniqueName: \"kubernetes.io/projected/6c0ace2f-4c34-48df-a593-8b7102595bae-kube-api-access-2mtqm\") on node \"ip-10-0-128-197.ec2.internal\" DevicePath \"\"" Jul 16 00:06:38.367827 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:06:38.367685 2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c0ace2f-4c34-48df-a593-8b7102595bae-istio-podinfo\") on node \"ip-10-0-128-197.ec2.internal\" DevicePath \"\"" Jul 16 00:06:38.367827 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:06:38.367693 2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c0ace2f-4c34-48df-a593-8b7102595bae-istio-data\") on node \"ip-10-0-128-197.ec2.internal\" DevicePath \"\"" Jul 16 00:06:38.367827 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:06:38.367701 2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c0ace2f-4c34-48df-a593-8b7102595bae-workload-certs\") on node \"ip-10-0-128-197.ec2.internal\" DevicePath \"\"" Jul 16 00:06:39.017360 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:06:39.017327 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-d3fd63-openshift-default-59db69b4b5-qgt8b" Jul 16 00:06:39.017360 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:06:39.017332 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d3fd63-openshift-default-59db69b4b5-qgt8b" event={"ID":"6c0ace2f-4c34-48df-a593-8b7102595bae","Type":"ContainerDied","Data":"a98078a12419b9017b82f1482b209963abb37f9d1d1d3f734beca7b47855f176"} Jul 16 00:06:39.017893 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:06:39.017398 2563 scope.go:117] "RemoveContainer" containerID="ec518538d31cc527f4b8a62face68e4c2bb66072bce21c7faade768ddb722e44" Jul 16 00:06:39.038003 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:06:39.037978 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-d3fd63-openshift-default-59db69b4b5-qgt8b"] Jul 16 00:06:39.041883 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:06:39.041863 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-d3fd63-openshift-default-59db69b4b5-qgt8b"] Jul 16 00:06:39.168899 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:06:39.168873 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6c0ace2f-4c34-48df-a593-8b7102595bae" path="/var/lib/kubelet/pods/6c0ace2f-4c34-48df-a593-8b7102595bae/volumes" Jul 16 00:06:39.446792 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:06:39.446714 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:06:39.706296 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:06:39.706203 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-dd8441-openshift-default-687bb5d445-zkmrm"] Jul 16 00:06:39.707291 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:06:39.707243 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-dd8441-openshift-default-687bb5d445-zkmrm" podUID="30af4109-dbe8-4f4d-9778-6d8580cf9b2d" containerName="istio-proxy" containerID="cri-o://cef0b425c46b87d99e17d1c8b8ff5195f21a176150dc10729afc44ce6860b39b" gracePeriod=30 Jul 16 00:06:40.050446 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:06:40.050413 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:06:40.108880 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:06:40.108844 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:06:40.270461 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:06:40.270427 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:06:40.543714 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:06:40.543677 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-a22338-openshift-default-75b5ff7965-zlgpp"] Jul 16 00:06:40.544158 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:06:40.544140 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6c0ace2f-4c34-48df-a593-8b7102595bae" containerName="istio-proxy" Jul 16 00:06:40.544158 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:06:40.544158 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="6c0ace2f-4c34-48df-a593-8b7102595bae" containerName="istio-proxy" Jul 16 00:06:40.544280 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:06:40.544234 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="6c0ace2f-4c34-48df-a593-8b7102595bae" containerName="istio-proxy" Jul 16 00:06:40.544280 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:06:40.544242 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="6c0ace2f-4c34-48df-a593-8b7102595bae" containerName="istio-proxy" Jul 16 00:06:40.548595 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:06:40.548577 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-a22338-openshift-default-75b5ff7965-zlgpp" Jul 16 00:06:40.550697 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:06:40.550679 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-a22338-openshift-default-dockercfg-kbxj8\"" Jul 16 00:06:40.558420 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:06:40.558401 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-a22338-openshift-default-75b5ff7965-zlgpp"] Jul 16 00:06:40.562775 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:06:40.562754 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:06:40.689572 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:06:40.689527 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e98c531-00d6-4bf7-b75a-21ecf44f0c15-workload-certs\") pod \"e2e-api-a-a22338-openshift-default-75b5ff7965-zlgpp\" (UID: \"8e98c531-00d6-4bf7-b75a-21ecf44f0c15\") " pod="openshift-ingress/e2e-api-a-a22338-openshift-default-75b5ff7965-zlgpp" Jul 16 00:06:40.689773 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:06:40.689592 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e98c531-00d6-4bf7-b75a-21ecf44f0c15-istiod-ca-cert\") pod \"e2e-api-a-a22338-openshift-default-75b5ff7965-zlgpp\" (UID: \"8e98c531-00d6-4bf7-b75a-21ecf44f0c15\") " pod="openshift-ingress/e2e-api-a-a22338-openshift-default-75b5ff7965-zlgpp" Jul 16 00:06:40.689773 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:06:40.689646 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e98c531-00d6-4bf7-b75a-21ecf44f0c15-credential-socket\") pod \"e2e-api-a-a22338-openshift-default-75b5ff7965-zlgpp\" (UID: \"8e98c531-00d6-4bf7-b75a-21ecf44f0c15\") " pod="openshift-ingress/e2e-api-a-a22338-openshift-default-75b5ff7965-zlgpp" Jul 16 00:06:40.689773 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:06:40.689736 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e98c531-00d6-4bf7-b75a-21ecf44f0c15-istio-envoy\") pod \"e2e-api-a-a22338-openshift-default-75b5ff7965-zlgpp\" (UID: \"8e98c531-00d6-4bf7-b75a-21ecf44f0c15\") " pod="openshift-ingress/e2e-api-a-a22338-openshift-default-75b5ff7965-zlgpp" Jul 16 00:06:40.689773 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:06:40.689765 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e98c531-00d6-4bf7-b75a-21ecf44f0c15-istio-token\") pod \"e2e-api-a-a22338-openshift-default-75b5ff7965-zlgpp\" (UID: \"8e98c531-00d6-4bf7-b75a-21ecf44f0c15\") " pod="openshift-ingress/e2e-api-a-a22338-openshift-default-75b5ff7965-zlgpp" Jul 16 00:06:40.689942 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:06:40.689817 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cgswv\" (UniqueName: \"kubernetes.io/projected/8e98c531-00d6-4bf7-b75a-21ecf44f0c15-kube-api-access-cgswv\") pod \"e2e-api-a-a22338-openshift-default-75b5ff7965-zlgpp\" (UID: \"8e98c531-00d6-4bf7-b75a-21ecf44f0c15\") " pod="openshift-ingress/e2e-api-a-a22338-openshift-default-75b5ff7965-zlgpp" Jul 16 00:06:40.689942 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:06:40.689840 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e98c531-00d6-4bf7-b75a-21ecf44f0c15-istio-podinfo\") pod \"e2e-api-a-a22338-openshift-default-75b5ff7965-zlgpp\" (UID: \"8e98c531-00d6-4bf7-b75a-21ecf44f0c15\") " pod="openshift-ingress/e2e-api-a-a22338-openshift-default-75b5ff7965-zlgpp" Jul 16 00:06:40.689942 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:06:40.689879 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e98c531-00d6-4bf7-b75a-21ecf44f0c15-istio-data\") pod \"e2e-api-a-a22338-openshift-default-75b5ff7965-zlgpp\" (UID: \"8e98c531-00d6-4bf7-b75a-21ecf44f0c15\") " pod="openshift-ingress/e2e-api-a-a22338-openshift-default-75b5ff7965-zlgpp" Jul 16 00:06:40.689942 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:06:40.689923 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e98c531-00d6-4bf7-b75a-21ecf44f0c15-workload-socket\") pod \"e2e-api-a-a22338-openshift-default-75b5ff7965-zlgpp\" (UID: \"8e98c531-00d6-4bf7-b75a-21ecf44f0c15\") " pod="openshift-ingress/e2e-api-a-a22338-openshift-default-75b5ff7965-zlgpp" Jul 16 00:06:40.790787 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:06:40.790757 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cgswv\" (UniqueName: \"kubernetes.io/projected/8e98c531-00d6-4bf7-b75a-21ecf44f0c15-kube-api-access-cgswv\") pod \"e2e-api-a-a22338-openshift-default-75b5ff7965-zlgpp\" (UID: \"8e98c531-00d6-4bf7-b75a-21ecf44f0c15\") " pod="openshift-ingress/e2e-api-a-a22338-openshift-default-75b5ff7965-zlgpp" Jul 16 00:06:40.790962 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:06:40.790798 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e98c531-00d6-4bf7-b75a-21ecf44f0c15-istio-podinfo\") pod \"e2e-api-a-a22338-openshift-default-75b5ff7965-zlgpp\" (UID: \"8e98c531-00d6-4bf7-b75a-21ecf44f0c15\") " pod="openshift-ingress/e2e-api-a-a22338-openshift-default-75b5ff7965-zlgpp" Jul 16 00:06:40.790962 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:06:40.790830 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e98c531-00d6-4bf7-b75a-21ecf44f0c15-istio-data\") pod \"e2e-api-a-a22338-openshift-default-75b5ff7965-zlgpp\" (UID: \"8e98c531-00d6-4bf7-b75a-21ecf44f0c15\") " pod="openshift-ingress/e2e-api-a-a22338-openshift-default-75b5ff7965-zlgpp" Jul 16 00:06:40.790962 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:06:40.790850 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e98c531-00d6-4bf7-b75a-21ecf44f0c15-workload-socket\") pod \"e2e-api-a-a22338-openshift-default-75b5ff7965-zlgpp\" (UID: \"8e98c531-00d6-4bf7-b75a-21ecf44f0c15\") " pod="openshift-ingress/e2e-api-a-a22338-openshift-default-75b5ff7965-zlgpp" Jul 16 00:06:40.790962 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:06:40.790875 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e98c531-00d6-4bf7-b75a-21ecf44f0c15-workload-certs\") pod \"e2e-api-a-a22338-openshift-default-75b5ff7965-zlgpp\" (UID: \"8e98c531-00d6-4bf7-b75a-21ecf44f0c15\") " pod="openshift-ingress/e2e-api-a-a22338-openshift-default-75b5ff7965-zlgpp" Jul 16 00:06:40.791217 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:06:40.791024 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e98c531-00d6-4bf7-b75a-21ecf44f0c15-istiod-ca-cert\") pod \"e2e-api-a-a22338-openshift-default-75b5ff7965-zlgpp\" (UID: \"8e98c531-00d6-4bf7-b75a-21ecf44f0c15\") " pod="openshift-ingress/e2e-api-a-a22338-openshift-default-75b5ff7965-zlgpp" Jul 16 00:06:40.791217 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:06:40.791074 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e98c531-00d6-4bf7-b75a-21ecf44f0c15-credential-socket\") pod \"e2e-api-a-a22338-openshift-default-75b5ff7965-zlgpp\" (UID: \"8e98c531-00d6-4bf7-b75a-21ecf44f0c15\") " pod="openshift-ingress/e2e-api-a-a22338-openshift-default-75b5ff7965-zlgpp" Jul 16 00:06:40.791217 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:06:40.791147 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e98c531-00d6-4bf7-b75a-21ecf44f0c15-istio-envoy\") pod \"e2e-api-a-a22338-openshift-default-75b5ff7965-zlgpp\" (UID: \"8e98c531-00d6-4bf7-b75a-21ecf44f0c15\") " pod="openshift-ingress/e2e-api-a-a22338-openshift-default-75b5ff7965-zlgpp" Jul 16 00:06:40.791217 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:06:40.791179 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e98c531-00d6-4bf7-b75a-21ecf44f0c15-istio-token\") pod \"e2e-api-a-a22338-openshift-default-75b5ff7965-zlgpp\" (UID: \"8e98c531-00d6-4bf7-b75a-21ecf44f0c15\") " pod="openshift-ingress/e2e-api-a-a22338-openshift-default-75b5ff7965-zlgpp" Jul 16 00:06:40.791491 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:06:40.791467 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e98c531-00d6-4bf7-b75a-21ecf44f0c15-workload-certs\") pod \"e2e-api-a-a22338-openshift-default-75b5ff7965-zlgpp\" (UID: \"8e98c531-00d6-4bf7-b75a-21ecf44f0c15\") " pod="openshift-ingress/e2e-api-a-a22338-openshift-default-75b5ff7965-zlgpp" Jul 16 00:06:40.791635 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:06:40.791507 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e98c531-00d6-4bf7-b75a-21ecf44f0c15-istio-data\") pod \"e2e-api-a-a22338-openshift-default-75b5ff7965-zlgpp\" (UID: \"8e98c531-00d6-4bf7-b75a-21ecf44f0c15\") " pod="openshift-ingress/e2e-api-a-a22338-openshift-default-75b5ff7965-zlgpp" Jul 16 00:06:40.791635 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:06:40.791575 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e98c531-00d6-4bf7-b75a-21ecf44f0c15-credential-socket\") pod \"e2e-api-a-a22338-openshift-default-75b5ff7965-zlgpp\" (UID: \"8e98c531-00d6-4bf7-b75a-21ecf44f0c15\") " pod="openshift-ingress/e2e-api-a-a22338-openshift-default-75b5ff7965-zlgpp" Jul 16 00:06:40.791732 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:06:40.791691 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e98c531-00d6-4bf7-b75a-21ecf44f0c15-workload-socket\") pod \"e2e-api-a-a22338-openshift-default-75b5ff7965-zlgpp\" (UID: \"8e98c531-00d6-4bf7-b75a-21ecf44f0c15\") " pod="openshift-ingress/e2e-api-a-a22338-openshift-default-75b5ff7965-zlgpp" Jul 16 00:06:40.791884 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:06:40.791866 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e98c531-00d6-4bf7-b75a-21ecf44f0c15-istiod-ca-cert\") pod \"e2e-api-a-a22338-openshift-default-75b5ff7965-zlgpp\" (UID: \"8e98c531-00d6-4bf7-b75a-21ecf44f0c15\") " pod="openshift-ingress/e2e-api-a-a22338-openshift-default-75b5ff7965-zlgpp" Jul 16 00:06:40.793276 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:06:40.793256 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e98c531-00d6-4bf7-b75a-21ecf44f0c15-istio-envoy\") pod \"e2e-api-a-a22338-openshift-default-75b5ff7965-zlgpp\" (UID: \"8e98c531-00d6-4bf7-b75a-21ecf44f0c15\") " pod="openshift-ingress/e2e-api-a-a22338-openshift-default-75b5ff7965-zlgpp" Jul 16 00:06:40.793414 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:06:40.793397 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e98c531-00d6-4bf7-b75a-21ecf44f0c15-istio-podinfo\") pod \"e2e-api-a-a22338-openshift-default-75b5ff7965-zlgpp\" (UID: \"8e98c531-00d6-4bf7-b75a-21ecf44f0c15\") " pod="openshift-ingress/e2e-api-a-a22338-openshift-default-75b5ff7965-zlgpp" Jul 16 00:06:40.797928 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:06:40.797864 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e98c531-00d6-4bf7-b75a-21ecf44f0c15-istio-token\") pod \"e2e-api-a-a22338-openshift-default-75b5ff7965-zlgpp\" (UID: \"8e98c531-00d6-4bf7-b75a-21ecf44f0c15\") " pod="openshift-ingress/e2e-api-a-a22338-openshift-default-75b5ff7965-zlgpp" Jul 16 00:06:40.798040 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:06:40.797951 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cgswv\" (UniqueName: \"kubernetes.io/projected/8e98c531-00d6-4bf7-b75a-21ecf44f0c15-kube-api-access-cgswv\") pod \"e2e-api-a-a22338-openshift-default-75b5ff7965-zlgpp\" (UID: \"8e98c531-00d6-4bf7-b75a-21ecf44f0c15\") " pod="openshift-ingress/e2e-api-a-a22338-openshift-default-75b5ff7965-zlgpp" Jul 16 00:06:40.860198 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:06:40.860173 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-a22338-openshift-default-75b5ff7965-zlgpp" Jul 16 00:06:40.992907 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:06:40.992871 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-a22338-openshift-default-75b5ff7965-zlgpp"] Jul 16 00:06:40.995092 ip-10-0-128-197 kubenswrapper[2563]: W0716 00:06:40.995067 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e98c531_00d6_4bf7_b75a_21ecf44f0c15.slice/crio-c2518e71746a72a0e1f74886c99408d759de78f460e716faa77fc63b06bed126 WatchSource:0}: Error finding container c2518e71746a72a0e1f74886c99408d759de78f460e716faa77fc63b06bed126: Status 404 returned error can't find the container with id c2518e71746a72a0e1f74886c99408d759de78f460e716faa77fc63b06bed126 Jul 16 00:06:40.997086 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:06:40.997058 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 16 00:06:40.997203 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:06:40.997117 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 16 00:06:40.997203 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:06:40.997144 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 16 00:06:41.029323 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:06:41.029298 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a22338-openshift-default-75b5ff7965-zlgpp" event={"ID":"8e98c531-00d6-4bf7-b75a-21ecf44f0c15","Type":"ContainerStarted","Data":"c2518e71746a72a0e1f74886c99408d759de78f460e716faa77fc63b06bed126"} Jul 16 00:06:41.378533 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:06:41.378444 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:06:42.035485 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:06:42.035449 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a22338-openshift-default-75b5ff7965-zlgpp" event={"ID":"8e98c531-00d6-4bf7-b75a-21ecf44f0c15","Type":"ContainerStarted","Data":"4729c9d99925d377b4d028db5a1a309db274f5f40b802b8df6b89d49c6085ae1"} Jul 16 00:06:42.054960 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:06:42.054910 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-a22338-openshift-default-75b5ff7965-zlgpp" podStartSLOduration=2.054895182 podStartE2EDuration="2.054895182s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 00:06:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 00:06:42.052105811 +0000 UTC m=+1817.425563785" watchObservedRunningTime="2026-07-16 00:06:42.054895182 +0000 UTC m=+1817.428353144" Jul 16 00:06:42.860717 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:06:42.860686 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-a22338-openshift-default-75b5ff7965-zlgpp" Jul 16 00:06:42.865517 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:06:42.865493 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-a22338-openshift-default-75b5ff7965-zlgpp" Jul 16 00:06:42.939740 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:06:42.939715 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:06:43.041115 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:06:43.041086 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-a22338-openshift-default-75b5ff7965-zlgpp" Jul 16 00:06:43.042029 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:06:43.041953 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-a22338-openshift-default-75b5ff7965-zlgpp" Jul 16 00:06:43.539627 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:06:43.539593 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:06:43.641990 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:06:43.641944 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:06:43.741399 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:06:43.741370 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:06:44.960178 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:06:44.960158 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-dd8441-openshift-default-687bb5d445-zkmrm" Jul 16 00:06:45.030697 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:06:45.030671 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30af4109-dbe8-4f4d-9778-6d8580cf9b2d-istio-token\") pod \"30af4109-dbe8-4f4d-9778-6d8580cf9b2d\" (UID: \"30af4109-dbe8-4f4d-9778-6d8580cf9b2d\") " Jul 16 00:06:45.030845 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:06:45.030728 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30af4109-dbe8-4f4d-9778-6d8580cf9b2d-workload-socket\") pod \"30af4109-dbe8-4f4d-9778-6d8580cf9b2d\" (UID: \"30af4109-dbe8-4f4d-9778-6d8580cf9b2d\") " Jul 16 00:06:45.030937 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:06:45.030918 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30af4109-dbe8-4f4d-9778-6d8580cf9b2d-istio-data\") pod \"30af4109-dbe8-4f4d-9778-6d8580cf9b2d\" (UID: \"30af4109-dbe8-4f4d-9778-6d8580cf9b2d\") " Jul 16 00:06:45.030993 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:06:45.030955 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30af4109-dbe8-4f4d-9778-6d8580cf9b2d-credential-socket\") pod \"30af4109-dbe8-4f4d-9778-6d8580cf9b2d\" (UID: \"30af4109-dbe8-4f4d-9778-6d8580cf9b2d\") " Jul 16 00:06:45.031049 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:06:45.030991 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30af4109-dbe8-4f4d-9778-6d8580cf9b2d-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "30af4109-dbe8-4f4d-9778-6d8580cf9b2d" (UID: "30af4109-dbe8-4f4d-9778-6d8580cf9b2d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 00:06:45.031049 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:06:45.031023 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30af4109-dbe8-4f4d-9778-6d8580cf9b2d-workload-certs\") pod \"30af4109-dbe8-4f4d-9778-6d8580cf9b2d\" (UID: \"30af4109-dbe8-4f4d-9778-6d8580cf9b2d\") " Jul 16 00:06:45.031155 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:06:45.031051 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30af4109-dbe8-4f4d-9778-6d8580cf9b2d-istiod-ca-cert\") pod \"30af4109-dbe8-4f4d-9778-6d8580cf9b2d\" (UID: \"30af4109-dbe8-4f4d-9778-6d8580cf9b2d\") " Jul 16 00:06:45.031155 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:06:45.031078 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4nz8p\" (UniqueName: \"kubernetes.io/projected/30af4109-dbe8-4f4d-9778-6d8580cf9b2d-kube-api-access-4nz8p\") pod \"30af4109-dbe8-4f4d-9778-6d8580cf9b2d\" (UID: \"30af4109-dbe8-4f4d-9778-6d8580cf9b2d\") " Jul 16 00:06:45.031155 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:06:45.031102 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30af4109-dbe8-4f4d-9778-6d8580cf9b2d-istio-podinfo\") pod \"30af4109-dbe8-4f4d-9778-6d8580cf9b2d\" (UID: \"30af4109-dbe8-4f4d-9778-6d8580cf9b2d\") " Jul 16 00:06:45.031307 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:06:45.031175 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30af4109-dbe8-4f4d-9778-6d8580cf9b2d-istio-envoy\") pod \"30af4109-dbe8-4f4d-9778-6d8580cf9b2d\" (UID: \"30af4109-dbe8-4f4d-9778-6d8580cf9b2d\") " Jul 16 00:06:45.031422 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:06:45.031399 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30af4109-dbe8-4f4d-9778-6d8580cf9b2d-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "30af4109-dbe8-4f4d-9778-6d8580cf9b2d" (UID: "30af4109-dbe8-4f4d-9778-6d8580cf9b2d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 00:06:45.031509 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:06:45.031461 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30af4109-dbe8-4f4d-9778-6d8580cf9b2d-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "30af4109-dbe8-4f4d-9778-6d8580cf9b2d" (UID: "30af4109-dbe8-4f4d-9778-6d8580cf9b2d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 00:06:45.031594 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:06:45.031565 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30af4109-dbe8-4f4d-9778-6d8580cf9b2d-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "30af4109-dbe8-4f4d-9778-6d8580cf9b2d" (UID: "30af4109-dbe8-4f4d-9778-6d8580cf9b2d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 00:06:45.031725 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:06:45.031573 2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30af4109-dbe8-4f4d-9778-6d8580cf9b2d-workload-socket\") on node \"ip-10-0-128-197.ec2.internal\" DevicePath \"\"" Jul 16 00:06:45.031725 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:06:45.031635 2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30af4109-dbe8-4f4d-9778-6d8580cf9b2d-workload-certs\") on node \"ip-10-0-128-197.ec2.internal\" DevicePath \"\"" Jul 16 00:06:45.031725 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:06:45.031702 2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30af4109-dbe8-4f4d-9778-6d8580cf9b2d-istiod-ca-cert\") on node \"ip-10-0-128-197.ec2.internal\" DevicePath \"\"" Jul 16 00:06:45.031900 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:06:45.031732 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30af4109-dbe8-4f4d-9778-6d8580cf9b2d-istio-data" (OuterVolumeSpecName: "istio-data") pod "30af4109-dbe8-4f4d-9778-6d8580cf9b2d" (UID: "30af4109-dbe8-4f4d-9778-6d8580cf9b2d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 00:06:45.032896 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:06:45.032861 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30af4109-dbe8-4f4d-9778-6d8580cf9b2d-istio-token" (OuterVolumeSpecName: "istio-token") pod "30af4109-dbe8-4f4d-9778-6d8580cf9b2d" (UID: "30af4109-dbe8-4f4d-9778-6d8580cf9b2d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 00:06:45.033224 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:06:45.033194 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30af4109-dbe8-4f4d-9778-6d8580cf9b2d-kube-api-access-4nz8p" (OuterVolumeSpecName: "kube-api-access-4nz8p") pod "30af4109-dbe8-4f4d-9778-6d8580cf9b2d" (UID: "30af4109-dbe8-4f4d-9778-6d8580cf9b2d"). InnerVolumeSpecName "kube-api-access-4nz8p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 00:06:45.033399 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:06:45.033379 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/30af4109-dbe8-4f4d-9778-6d8580cf9b2d-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "30af4109-dbe8-4f4d-9778-6d8580cf9b2d" (UID: "30af4109-dbe8-4f4d-9778-6d8580cf9b2d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 16 00:06:45.033528 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:06:45.033508 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30af4109-dbe8-4f4d-9778-6d8580cf9b2d-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "30af4109-dbe8-4f4d-9778-6d8580cf9b2d" (UID: "30af4109-dbe8-4f4d-9778-6d8580cf9b2d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 00:06:45.052853 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:06:45.052825 2563 generic.go:356] "Generic (PLEG): container finished" podID="30af4109-dbe8-4f4d-9778-6d8580cf9b2d" containerID="cef0b425c46b87d99e17d1c8b8ff5195f21a176150dc10729afc44ce6860b39b" exitCode=0 Jul 16 00:06:45.052929 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:06:45.052882 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-dd8441-openshift-default-687bb5d445-zkmrm" Jul 16 00:06:45.052929 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:06:45.052899 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-dd8441-openshift-default-687bb5d445-zkmrm" event={"ID":"30af4109-dbe8-4f4d-9778-6d8580cf9b2d","Type":"ContainerDied","Data":"cef0b425c46b87d99e17d1c8b8ff5195f21a176150dc10729afc44ce6860b39b"} Jul 16 00:06:45.053039 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:06:45.052986 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-dd8441-openshift-default-687bb5d445-zkmrm" event={"ID":"30af4109-dbe8-4f4d-9778-6d8580cf9b2d","Type":"ContainerDied","Data":"6c7a0407038c143e7b1f34e770652f3ba032d0fab442aac6b4fc9a21afad5a37"} Jul 16 00:06:45.053039 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:06:45.053010 2563 scope.go:117] "RemoveContainer" containerID="cef0b425c46b87d99e17d1c8b8ff5195f21a176150dc10729afc44ce6860b39b" Jul 16 00:06:45.062376 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:06:45.062357 2563 scope.go:117] "RemoveContainer" containerID="d200119b763859d1bd570c15a43558ed2e96f891d0f33e65a1c53edc8e2444ee" Jul 16 00:06:45.070402 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:06:45.070386 2563 scope.go:117] "RemoveContainer" containerID="cef0b425c46b87d99e17d1c8b8ff5195f21a176150dc10729afc44ce6860b39b" Jul 16 00:06:45.070619 ip-10-0-128-197 kubenswrapper[2563]: E0716 00:06:45.070598 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cef0b425c46b87d99e17d1c8b8ff5195f21a176150dc10729afc44ce6860b39b\": container with ID starting with cef0b425c46b87d99e17d1c8b8ff5195f21a176150dc10729afc44ce6860b39b not found: ID does not exist" containerID="cef0b425c46b87d99e17d1c8b8ff5195f21a176150dc10729afc44ce6860b39b" Jul 16 00:06:45.070677 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:06:45.070625 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cef0b425c46b87d99e17d1c8b8ff5195f21a176150dc10729afc44ce6860b39b"} err="failed to get container status \"cef0b425c46b87d99e17d1c8b8ff5195f21a176150dc10729afc44ce6860b39b\": rpc error: code = NotFound desc = could not find container \"cef0b425c46b87d99e17d1c8b8ff5195f21a176150dc10729afc44ce6860b39b\": container with ID starting with cef0b425c46b87d99e17d1c8b8ff5195f21a176150dc10729afc44ce6860b39b not found: ID does not exist" Jul 16 00:06:45.070677 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:06:45.070642 2563 scope.go:117] "RemoveContainer" containerID="d200119b763859d1bd570c15a43558ed2e96f891d0f33e65a1c53edc8e2444ee" Jul 16 00:06:45.070883 ip-10-0-128-197 kubenswrapper[2563]: E0716 00:06:45.070868 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d200119b763859d1bd570c15a43558ed2e96f891d0f33e65a1c53edc8e2444ee\": container with ID starting with d200119b763859d1bd570c15a43558ed2e96f891d0f33e65a1c53edc8e2444ee not found: ID does not exist" containerID="d200119b763859d1bd570c15a43558ed2e96f891d0f33e65a1c53edc8e2444ee" Jul 16 00:06:45.070920 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:06:45.070888 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d200119b763859d1bd570c15a43558ed2e96f891d0f33e65a1c53edc8e2444ee"} err="failed to get container status \"d200119b763859d1bd570c15a43558ed2e96f891d0f33e65a1c53edc8e2444ee\": rpc error: code = NotFound desc = could not find container \"d200119b763859d1bd570c15a43558ed2e96f891d0f33e65a1c53edc8e2444ee\": container with ID starting with d200119b763859d1bd570c15a43558ed2e96f891d0f33e65a1c53edc8e2444ee not found: ID does not exist" Jul 16 00:06:45.073996 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:06:45.073951 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-dd8441-openshift-default-687bb5d445-zkmrm"] Jul 16 00:06:45.078138 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:06:45.078112 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-dd8441-openshift-default-687bb5d445-zkmrm"] Jul 16 00:06:45.132974 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:06:45.132950 2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30af4109-dbe8-4f4d-9778-6d8580cf9b2d-istio-data\") on node \"ip-10-0-128-197.ec2.internal\" DevicePath \"\"" Jul 16 00:06:45.133066 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:06:45.132997 2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30af4109-dbe8-4f4d-9778-6d8580cf9b2d-credential-socket\") on node \"ip-10-0-128-197.ec2.internal\" DevicePath \"\"" Jul 16 00:06:45.133066 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:06:45.133013 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4nz8p\" (UniqueName: \"kubernetes.io/projected/30af4109-dbe8-4f4d-9778-6d8580cf9b2d-kube-api-access-4nz8p\") on node \"ip-10-0-128-197.ec2.internal\" DevicePath \"\"" Jul 16 00:06:45.133066 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:06:45.133028 2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30af4109-dbe8-4f4d-9778-6d8580cf9b2d-istio-podinfo\") on node \"ip-10-0-128-197.ec2.internal\" DevicePath \"\"" Jul 16 00:06:45.133066 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:06:45.133043 2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30af4109-dbe8-4f4d-9778-6d8580cf9b2d-istio-envoy\") on node \"ip-10-0-128-197.ec2.internal\" DevicePath \"\"" Jul 16 00:06:45.133066 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:06:45.133057 2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30af4109-dbe8-4f4d-9778-6d8580cf9b2d-istio-token\") on node \"ip-10-0-128-197.ec2.internal\" DevicePath \"\"" Jul 16 00:06:45.169831 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:06:45.169807 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="30af4109-dbe8-4f4d-9778-6d8580cf9b2d" path="/var/lib/kubelet/pods/30af4109-dbe8-4f4d-9778-6d8580cf9b2d/volumes" Jul 16 00:06:48.454632 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:06:48.454602 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:06:48.893812 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:06:48.893771 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:06:49.171330 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:06:49.171253 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-eae7a4-openshift-default-655765b455-87hdc"] Jul 16 00:06:49.171678 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:06:49.171663 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6c0ace2f-4c34-48df-a593-8b7102595bae" containerName="istio-proxy" Jul 16 00:06:49.171678 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:06:49.171679 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="6c0ace2f-4c34-48df-a593-8b7102595bae" containerName="istio-proxy" Jul 16 00:06:49.171803 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:06:49.171692 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="30af4109-dbe8-4f4d-9778-6d8580cf9b2d" containerName="istio-proxy" Jul 16 00:06:49.171803 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:06:49.171701 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="30af4109-dbe8-4f4d-9778-6d8580cf9b2d" containerName="istio-proxy" Jul 16 00:06:49.171803 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:06:49.171711 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="30af4109-dbe8-4f4d-9778-6d8580cf9b2d" containerName="istio-proxy" Jul 16 00:06:49.171803 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:06:49.171721 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="30af4109-dbe8-4f4d-9778-6d8580cf9b2d" containerName="istio-proxy" Jul 16 00:06:49.171939 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:06:49.171839 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="30af4109-dbe8-4f4d-9778-6d8580cf9b2d" containerName="istio-proxy" Jul 16 00:06:49.171939 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:06:49.171852 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="30af4109-dbe8-4f4d-9778-6d8580cf9b2d" containerName="istio-proxy" Jul 16 00:06:49.176282 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:06:49.176262 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-eae7a4-openshift-default-655765b455-87hdc" Jul 16 00:06:49.178564 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:06:49.178544 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-eae7a4-openshift-default-dockercfg-vn75x\"" Jul 16 00:06:49.186426 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:06:49.186403 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-eae7a4-openshift-default-655765b455-87hdc"] Jul 16 00:06:49.214899 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:06:49.214839 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:06:49.274047 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:06:49.274009 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f53994d-00f8-49b1-9743-427275ddf86e-istio-data\") pod \"e2e-api-b-eae7a4-openshift-default-655765b455-87hdc\" (UID: \"7f53994d-00f8-49b1-9743-427275ddf86e\") " pod="openshift-ingress/e2e-api-b-eae7a4-openshift-default-655765b455-87hdc" Jul 16 00:06:49.274239 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:06:49.274053 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-db5kd\" (UniqueName: \"kubernetes.io/projected/7f53994d-00f8-49b1-9743-427275ddf86e-kube-api-access-db5kd\") pod \"e2e-api-b-eae7a4-openshift-default-655765b455-87hdc\" (UID: \"7f53994d-00f8-49b1-9743-427275ddf86e\") " pod="openshift-ingress/e2e-api-b-eae7a4-openshift-default-655765b455-87hdc" Jul 16 00:06:49.274239 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:06:49.274094 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f53994d-00f8-49b1-9743-427275ddf86e-workload-certs\") pod \"e2e-api-b-eae7a4-openshift-default-655765b455-87hdc\" (UID: \"7f53994d-00f8-49b1-9743-427275ddf86e\") " pod="openshift-ingress/e2e-api-b-eae7a4-openshift-default-655765b455-87hdc" Jul 16 00:06:49.274239 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:06:49.274192 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f53994d-00f8-49b1-9743-427275ddf86e-istio-token\") pod \"e2e-api-b-eae7a4-openshift-default-655765b455-87hdc\" (UID: \"7f53994d-00f8-49b1-9743-427275ddf86e\") " pod="openshift-ingress/e2e-api-b-eae7a4-openshift-default-655765b455-87hdc" Jul 16 00:06:49.274434 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:06:49.274331 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f53994d-00f8-49b1-9743-427275ddf86e-istio-podinfo\") pod \"e2e-api-b-eae7a4-openshift-default-655765b455-87hdc\" (UID: \"7f53994d-00f8-49b1-9743-427275ddf86e\") " pod="openshift-ingress/e2e-api-b-eae7a4-openshift-default-655765b455-87hdc" Jul 16 00:06:49.274434 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:06:49.274353 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f53994d-00f8-49b1-9743-427275ddf86e-istiod-ca-cert\") pod \"e2e-api-b-eae7a4-openshift-default-655765b455-87hdc\" (UID: \"7f53994d-00f8-49b1-9743-427275ddf86e\") " pod="openshift-ingress/e2e-api-b-eae7a4-openshift-default-655765b455-87hdc" Jul 16 00:06:49.274550 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:06:49.274523 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f53994d-00f8-49b1-9743-427275ddf86e-credential-socket\") pod \"e2e-api-b-eae7a4-openshift-default-655765b455-87hdc\" (UID: \"7f53994d-00f8-49b1-9743-427275ddf86e\") " pod="openshift-ingress/e2e-api-b-eae7a4-openshift-default-655765b455-87hdc" Jul 16 00:06:49.274609 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:06:49.274562 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f53994d-00f8-49b1-9743-427275ddf86e-istio-envoy\") pod \"e2e-api-b-eae7a4-openshift-default-655765b455-87hdc\" (UID: \"7f53994d-00f8-49b1-9743-427275ddf86e\") " pod="openshift-ingress/e2e-api-b-eae7a4-openshift-default-655765b455-87hdc" Jul 16 00:06:49.274660 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:06:49.274629 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f53994d-00f8-49b1-9743-427275ddf86e-workload-socket\") pod \"e2e-api-b-eae7a4-openshift-default-655765b455-87hdc\" (UID: \"7f53994d-00f8-49b1-9743-427275ddf86e\") " pod="openshift-ingress/e2e-api-b-eae7a4-openshift-default-655765b455-87hdc" Jul 16 00:06:49.376220 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:06:49.376188 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f53994d-00f8-49b1-9743-427275ddf86e-istio-data\") pod \"e2e-api-b-eae7a4-openshift-default-655765b455-87hdc\" (UID: \"7f53994d-00f8-49b1-9743-427275ddf86e\") " pod="openshift-ingress/e2e-api-b-eae7a4-openshift-default-655765b455-87hdc" Jul 16 00:06:49.376417 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:06:49.376227 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-db5kd\" (UniqueName: \"kubernetes.io/projected/7f53994d-00f8-49b1-9743-427275ddf86e-kube-api-access-db5kd\") pod \"e2e-api-b-eae7a4-openshift-default-655765b455-87hdc\" (UID: \"7f53994d-00f8-49b1-9743-427275ddf86e\") " pod="openshift-ingress/e2e-api-b-eae7a4-openshift-default-655765b455-87hdc" Jul 16 00:06:49.376417 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:06:49.376264 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f53994d-00f8-49b1-9743-427275ddf86e-workload-certs\") pod \"e2e-api-b-eae7a4-openshift-default-655765b455-87hdc\" (UID: \"7f53994d-00f8-49b1-9743-427275ddf86e\") " pod="openshift-ingress/e2e-api-b-eae7a4-openshift-default-655765b455-87hdc" Jul 16 00:06:49.376417 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:06:49.376305 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f53994d-00f8-49b1-9743-427275ddf86e-istio-token\") pod \"e2e-api-b-eae7a4-openshift-default-655765b455-87hdc\" (UID: \"7f53994d-00f8-49b1-9743-427275ddf86e\") " pod="openshift-ingress/e2e-api-b-eae7a4-openshift-default-655765b455-87hdc" Jul 16 00:06:49.376417 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:06:49.376402 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f53994d-00f8-49b1-9743-427275ddf86e-istio-podinfo\") pod \"e2e-api-b-eae7a4-openshift-default-655765b455-87hdc\" (UID: \"7f53994d-00f8-49b1-9743-427275ddf86e\") " pod="openshift-ingress/e2e-api-b-eae7a4-openshift-default-655765b455-87hdc" Jul 16 00:06:49.376642 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:06:49.376424 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f53994d-00f8-49b1-9743-427275ddf86e-istiod-ca-cert\") pod \"e2e-api-b-eae7a4-openshift-default-655765b455-87hdc\" (UID: \"7f53994d-00f8-49b1-9743-427275ddf86e\") " pod="openshift-ingress/e2e-api-b-eae7a4-openshift-default-655765b455-87hdc" Jul 16 00:06:49.376642 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:06:49.376471 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f53994d-00f8-49b1-9743-427275ddf86e-credential-socket\") pod \"e2e-api-b-eae7a4-openshift-default-655765b455-87hdc\" (UID: \"7f53994d-00f8-49b1-9743-427275ddf86e\") " pod="openshift-ingress/e2e-api-b-eae7a4-openshift-default-655765b455-87hdc" Jul 16 00:06:49.376642 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:06:49.376496 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f53994d-00f8-49b1-9743-427275ddf86e-istio-envoy\") pod \"e2e-api-b-eae7a4-openshift-default-655765b455-87hdc\" (UID: \"7f53994d-00f8-49b1-9743-427275ddf86e\") " pod="openshift-ingress/e2e-api-b-eae7a4-openshift-default-655765b455-87hdc" Jul 16 00:06:49.376642 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:06:49.376545 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f53994d-00f8-49b1-9743-427275ddf86e-workload-socket\") pod \"e2e-api-b-eae7a4-openshift-default-655765b455-87hdc\" (UID: \"7f53994d-00f8-49b1-9743-427275ddf86e\") " pod="openshift-ingress/e2e-api-b-eae7a4-openshift-default-655765b455-87hdc" Jul 16 00:06:49.376642 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:06:49.376574 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f53994d-00f8-49b1-9743-427275ddf86e-istio-data\") pod \"e2e-api-b-eae7a4-openshift-default-655765b455-87hdc\" (UID: \"7f53994d-00f8-49b1-9743-427275ddf86e\") " pod="openshift-ingress/e2e-api-b-eae7a4-openshift-default-655765b455-87hdc" Jul 16 00:06:49.376902 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:06:49.376659 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f53994d-00f8-49b1-9743-427275ddf86e-workload-certs\") pod \"e2e-api-b-eae7a4-openshift-default-655765b455-87hdc\" (UID: \"7f53994d-00f8-49b1-9743-427275ddf86e\") " pod="openshift-ingress/e2e-api-b-eae7a4-openshift-default-655765b455-87hdc" Jul 16 00:06:49.376992 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:06:49.376929 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f53994d-00f8-49b1-9743-427275ddf86e-credential-socket\") pod \"e2e-api-b-eae7a4-openshift-default-655765b455-87hdc\" (UID: \"7f53994d-00f8-49b1-9743-427275ddf86e\") " pod="openshift-ingress/e2e-api-b-eae7a4-openshift-default-655765b455-87hdc" Jul 16 00:06:49.377060 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:06:49.377015 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f53994d-00f8-49b1-9743-427275ddf86e-workload-socket\") pod \"e2e-api-b-eae7a4-openshift-default-655765b455-87hdc\" (UID: \"7f53994d-00f8-49b1-9743-427275ddf86e\") " pod="openshift-ingress/e2e-api-b-eae7a4-openshift-default-655765b455-87hdc" Jul 16 00:06:49.377202 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:06:49.377176 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f53994d-00f8-49b1-9743-427275ddf86e-istiod-ca-cert\") pod \"e2e-api-b-eae7a4-openshift-default-655765b455-87hdc\" (UID: \"7f53994d-00f8-49b1-9743-427275ddf86e\") " pod="openshift-ingress/e2e-api-b-eae7a4-openshift-default-655765b455-87hdc" Jul 16 00:06:49.379115 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:06:49.379029 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f53994d-00f8-49b1-9743-427275ddf86e-istio-envoy\") pod \"e2e-api-b-eae7a4-openshift-default-655765b455-87hdc\" (UID: \"7f53994d-00f8-49b1-9743-427275ddf86e\") " pod="openshift-ingress/e2e-api-b-eae7a4-openshift-default-655765b455-87hdc" Jul 16 00:06:49.379285 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:06:49.379249 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f53994d-00f8-49b1-9743-427275ddf86e-istio-podinfo\") pod \"e2e-api-b-eae7a4-openshift-default-655765b455-87hdc\" (UID: \"7f53994d-00f8-49b1-9743-427275ddf86e\") " pod="openshift-ingress/e2e-api-b-eae7a4-openshift-default-655765b455-87hdc" Jul 16 00:06:49.387175 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:06:49.387132 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-db5kd\" (UniqueName: \"kubernetes.io/projected/7f53994d-00f8-49b1-9743-427275ddf86e-kube-api-access-db5kd\") pod \"e2e-api-b-eae7a4-openshift-default-655765b455-87hdc\" (UID: \"7f53994d-00f8-49b1-9743-427275ddf86e\") " pod="openshift-ingress/e2e-api-b-eae7a4-openshift-default-655765b455-87hdc" Jul 16 00:06:49.387632 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:06:49.387591 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f53994d-00f8-49b1-9743-427275ddf86e-istio-token\") pod \"e2e-api-b-eae7a4-openshift-default-655765b455-87hdc\" (UID: \"7f53994d-00f8-49b1-9743-427275ddf86e\") " pod="openshift-ingress/e2e-api-b-eae7a4-openshift-default-655765b455-87hdc" Jul 16 00:06:49.423579 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:06:49.423489 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:06:49.487409 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:06:49.487375 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-eae7a4-openshift-default-655765b455-87hdc" Jul 16 00:06:49.663174 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:06:49.663151 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-eae7a4-openshift-default-655765b455-87hdc"] Jul 16 00:06:49.665041 ip-10-0-128-197 kubenswrapper[2563]: W0716 00:06:49.665013 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f53994d_00f8_49b1_9743_427275ddf86e.slice/crio-5d4402cd7f37ecb29e20607a00d46e68cd5cfb38d9577cc10e5ce1723433a5df WatchSource:0}: Error finding container 5d4402cd7f37ecb29e20607a00d46e68cd5cfb38d9577cc10e5ce1723433a5df: Status 404 returned error can't find the container with id 5d4402cd7f37ecb29e20607a00d46e68cd5cfb38d9577cc10e5ce1723433a5df Jul 16 00:06:49.667150 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:06:49.667117 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 16 00:06:49.667277 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:06:49.667194 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 16 00:06:49.667277 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:06:49.667237 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 16 00:06:50.081978 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:06:50.081928 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-eae7a4-openshift-default-655765b455-87hdc" event={"ID":"7f53994d-00f8-49b1-9743-427275ddf86e","Type":"ContainerStarted","Data":"7ca382af7d76f4f278711fbe5c54c7c9343b443368b3ec7ec3ffc2bd3d83e734"} Jul 16 00:06:50.082156 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:06:50.082008 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-eae7a4-openshift-default-655765b455-87hdc" event={"ID":"7f53994d-00f8-49b1-9743-427275ddf86e","Type":"ContainerStarted","Data":"5d4402cd7f37ecb29e20607a00d46e68cd5cfb38d9577cc10e5ce1723433a5df"} Jul 16 00:06:50.101009 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:06:50.100933 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-eae7a4-openshift-default-655765b455-87hdc" podStartSLOduration=1.100911087 podStartE2EDuration="1.100911087s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 00:06:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 00:06:50.098580857 +0000 UTC m=+1825.472038823" watchObservedRunningTime="2026-07-16 00:06:50.100911087 +0000 UTC m=+1825.474369051" Jul 16 00:06:50.487984 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:06:50.487923 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-eae7a4-openshift-default-655765b455-87hdc" Jul 16 00:06:50.503418 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:06:50.503384 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-eae7a4-openshift-default-655765b455-87hdc" Jul 16 00:06:51.017169 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:06:51.017137 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:06:51.087389 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:06:51.087360 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-eae7a4-openshift-default-655765b455-87hdc" Jul 16 00:06:51.088500 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:06:51.088480 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-eae7a4-openshift-default-655765b455-87hdc" Jul 16 00:06:51.098527 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:06:51.098492 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:06:51.168103 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:06:51.168074 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:06:51.618021 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:06:51.617983 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:06:51.719112 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:06:51.719079 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:06:56.116481 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:06:56.116449 2563 generic.go:356] "Generic (PLEG): container finished" podID="8e98c531-00d6-4bf7-b75a-21ecf44f0c15" containerID="4729c9d99925d377b4d028db5a1a309db274f5f40b802b8df6b89d49c6085ae1" exitCode=0 Jul 16 00:06:56.116889 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:06:56.116516 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a22338-openshift-default-75b5ff7965-zlgpp" event={"ID":"8e98c531-00d6-4bf7-b75a-21ecf44f0c15","Type":"ContainerDied","Data":"4729c9d99925d377b4d028db5a1a309db274f5f40b802b8df6b89d49c6085ae1"} Jul 16 00:06:56.117000 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:06:56.116982 2563 scope.go:117] "RemoveContainer" containerID="4729c9d99925d377b4d028db5a1a309db274f5f40b802b8df6b89d49c6085ae1" Jul 16 00:06:56.117952 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:06:56.117927 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 16 00:06:56.118031 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:06:56.118017 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 16 00:06:56.118069 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:06:56.118054 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 16 00:06:56.577373 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:06:56.577341 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:06:57.123650 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:06:57.123610 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a22338-openshift-default-75b5ff7965-zlgpp" event={"ID":"8e98c531-00d6-4bf7-b75a-21ecf44f0c15","Type":"ContainerStarted","Data":"6a73a7f6b2a9866a786dbb35989027cc8f9ee0624ce18915cf2e3dc9dd8e07d2"} Jul 16 00:06:57.861199 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:06:57.861166 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-a22338-openshift-default-75b5ff7965-zlgpp" Jul 16 00:06:57.862344 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:06:57.862322 2563 patch_prober.go:28] interesting pod/e2e-api-a-a22338-openshift-default-75b5ff7965-zlgpp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Jul 16 00:06:57.862410 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:06:57.862371 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a22338-openshift-default-75b5ff7965-zlgpp" podUID="8e98c531-00d6-4bf7-b75a-21ecf44f0c15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Jul 16 00:06:58.041346 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:06:58.041312 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-a22338-openshift-default-75b5ff7965-zlgpp" Jul 16 00:06:58.861500 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:06:58.861468 2563 patch_prober.go:28] interesting pod/e2e-api-a-a22338-openshift-default-75b5ff7965-zlgpp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Jul 16 00:06:58.861897 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:06:58.861522 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a22338-openshift-default-75b5ff7965-zlgpp" podUID="8e98c531-00d6-4bf7-b75a-21ecf44f0c15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Jul 16 00:06:59.860640 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:06:59.860609 2563 patch_prober.go:28] interesting pod/e2e-api-a-a22338-openshift-default-75b5ff7965-zlgpp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Jul 16 00:06:59.860814 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:06:59.860661 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a22338-openshift-default-75b5ff7965-zlgpp" podUID="8e98c531-00d6-4bf7-b75a-21ecf44f0c15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Jul 16 00:07:00.860800 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:00.860762 2563 patch_prober.go:28] interesting pod/e2e-api-a-a22338-openshift-default-75b5ff7965-zlgpp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Jul 16 00:07:00.861234 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:00.860819 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a22338-openshift-default-75b5ff7965-zlgpp" podUID="8e98c531-00d6-4bf7-b75a-21ecf44f0c15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Jul 16 00:07:01.864556 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:01.864526 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-a22338-openshift-default-75b5ff7965-zlgpp" Jul 16 00:07:01.865480 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:01.865461 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-a22338-openshift-default-75b5ff7965-zlgpp" Jul 16 00:07:04.275279 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:04.275243 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:07:04.376449 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:04.376413 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:07:04.981378 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:04.981349 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:07:10.740348 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:10.740312 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:07:11.204302 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:11.204218 2563 generic.go:356] "Generic (PLEG): container finished" podID="7f53994d-00f8-49b1-9743-427275ddf86e" containerID="7ca382af7d76f4f278711fbe5c54c7c9343b443368b3ec7ec3ffc2bd3d83e734" exitCode=0 Jul 16 00:07:11.204302 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:11.204287 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-eae7a4-openshift-default-655765b455-87hdc" event={"ID":"7f53994d-00f8-49b1-9743-427275ddf86e","Type":"ContainerDied","Data":"7ca382af7d76f4f278711fbe5c54c7c9343b443368b3ec7ec3ffc2bd3d83e734"} Jul 16 00:07:11.204694 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:11.204679 2563 scope.go:117] "RemoveContainer" containerID="7ca382af7d76f4f278711fbe5c54c7c9343b443368b3ec7ec3ffc2bd3d83e734" Jul 16 00:07:11.205718 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:11.205687 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 16 00:07:11.205800 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:11.205756 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 16 00:07:11.205800 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:11.205784 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 16 00:07:12.211869 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:12.211831 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-eae7a4-openshift-default-655765b455-87hdc" event={"ID":"7f53994d-00f8-49b1-9743-427275ddf86e","Type":"ContainerStarted","Data":"f2a8521df85b5518f4e602cb42c2132fb4b502a0e4860789d88535187ad78735"} Jul 16 00:07:12.342435 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:12.342397 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:07:12.442624 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:12.442592 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:07:12.488522 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:12.488492 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-eae7a4-openshift-default-655765b455-87hdc" Jul 16 00:07:12.488702 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:12.488668 2563 patch_prober.go:28] interesting pod/e2e-api-b-eae7a4-openshift-default-655765b455-87hdc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Jul 16 00:07:12.488756 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:12.488723 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-eae7a4-openshift-default-655765b455-87hdc" podUID="7f53994d-00f8-49b1-9743-427275ddf86e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Jul 16 00:07:13.488489 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:13.488451 2563 patch_prober.go:28] interesting pod/e2e-api-b-eae7a4-openshift-default-655765b455-87hdc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Jul 16 00:07:13.488903 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:13.488508 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-eae7a4-openshift-default-655765b455-87hdc" podUID="7f53994d-00f8-49b1-9743-427275ddf86e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Jul 16 00:07:14.488481 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:14.488445 2563 patch_prober.go:28] interesting pod/e2e-api-b-eae7a4-openshift-default-655765b455-87hdc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Jul 16 00:07:14.488663 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:14.488509 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-eae7a4-openshift-default-655765b455-87hdc" podUID="7f53994d-00f8-49b1-9743-427275ddf86e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Jul 16 00:07:15.488086 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:15.488053 2563 patch_prober.go:28] interesting pod/e2e-api-b-eae7a4-openshift-default-655765b455-87hdc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Jul 16 00:07:15.488267 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:15.488106 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-eae7a4-openshift-default-655765b455-87hdc" podUID="7f53994d-00f8-49b1-9743-427275ddf86e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Jul 16 00:07:16.301992 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:16.301903 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:07:16.356714 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:16.356680 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:07:16.488467 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:16.488439 2563 patch_prober.go:28] interesting pod/e2e-api-b-eae7a4-openshift-default-655765b455-87hdc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Jul 16 00:07:16.488631 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:16.488488 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-eae7a4-openshift-default-655765b455-87hdc" podUID="7f53994d-00f8-49b1-9743-427275ddf86e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Jul 16 00:07:16.546351 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:16.546321 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-a22338-openshift-default-75b5ff7965-zlgpp"] Jul 16 00:07:16.546537 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:16.546516 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-a22338-openshift-default-75b5ff7965-zlgpp" podUID="8e98c531-00d6-4bf7-b75a-21ecf44f0c15" containerName="istio-proxy" containerID="cri-o://6a73a7f6b2a9866a786dbb35989027cc8f9ee0624ce18915cf2e3dc9dd8e07d2" gracePeriod=30 Jul 16 00:07:16.565582 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:16.565521 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:07:16.776593 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:16.776560 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:07:16.867515 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:16.867429 2563 patch_prober.go:28] interesting pod/e2e-api-a-a22338-openshift-default-75b5ff7965-zlgpp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 16 00:07:16.867515 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:16.867492 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-a22338-openshift-default-75b5ff7965-zlgpp" podUID="8e98c531-00d6-4bf7-b75a-21ecf44f0c15" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 16 00:07:17.223887 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:17.223768 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:07:17.491538 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:17.491505 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-eae7a4-openshift-default-655765b455-87hdc" Jul 16 00:07:17.491984 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:17.491883 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-eae7a4-openshift-default-655765b455-87hdc" Jul 16 00:07:17.492696 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:17.492678 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-eae7a4-openshift-default-655765b455-87hdc" Jul 16 00:07:21.796513 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:21.796491 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-a22338-openshift-default-75b5ff7965-zlgpp" Jul 16 00:07:21.938071 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:21.937989 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e98c531-00d6-4bf7-b75a-21ecf44f0c15-istio-token\") pod \"8e98c531-00d6-4bf7-b75a-21ecf44f0c15\" (UID: \"8e98c531-00d6-4bf7-b75a-21ecf44f0c15\") " Jul 16 00:07:21.938071 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:21.938045 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e98c531-00d6-4bf7-b75a-21ecf44f0c15-istio-data\") pod \"8e98c531-00d6-4bf7-b75a-21ecf44f0c15\" (UID: \"8e98c531-00d6-4bf7-b75a-21ecf44f0c15\") " Jul 16 00:07:21.938288 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:21.938221 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e98c531-00d6-4bf7-b75a-21ecf44f0c15-istio-podinfo\") pod \"8e98c531-00d6-4bf7-b75a-21ecf44f0c15\" (UID: \"8e98c531-00d6-4bf7-b75a-21ecf44f0c15\") " Jul 16 00:07:21.938288 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:21.938281 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e98c531-00d6-4bf7-b75a-21ecf44f0c15-workload-socket\") pod \"8e98c531-00d6-4bf7-b75a-21ecf44f0c15\" (UID: \"8e98c531-00d6-4bf7-b75a-21ecf44f0c15\") " Jul 16 00:07:21.938400 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:21.938341 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e98c531-00d6-4bf7-b75a-21ecf44f0c15-istio-envoy\") pod \"8e98c531-00d6-4bf7-b75a-21ecf44f0c15\" (UID: \"8e98c531-00d6-4bf7-b75a-21ecf44f0c15\") " Jul 16 00:07:21.938400 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:21.938376 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e98c531-00d6-4bf7-b75a-21ecf44f0c15-credential-socket\") pod \"8e98c531-00d6-4bf7-b75a-21ecf44f0c15\" (UID: \"8e98c531-00d6-4bf7-b75a-21ecf44f0c15\") " Jul 16 00:07:21.938503 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:21.938415 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e98c531-00d6-4bf7-b75a-21ecf44f0c15-workload-certs\") pod \"8e98c531-00d6-4bf7-b75a-21ecf44f0c15\" (UID: \"8e98c531-00d6-4bf7-b75a-21ecf44f0c15\") " Jul 16 00:07:21.938503 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:21.938444 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e98c531-00d6-4bf7-b75a-21ecf44f0c15-istiod-ca-cert\") pod \"8e98c531-00d6-4bf7-b75a-21ecf44f0c15\" (UID: \"8e98c531-00d6-4bf7-b75a-21ecf44f0c15\") " Jul 16 00:07:21.938503 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:21.938468 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cgswv\" (UniqueName: \"kubernetes.io/projected/8e98c531-00d6-4bf7-b75a-21ecf44f0c15-kube-api-access-cgswv\") pod \"8e98c531-00d6-4bf7-b75a-21ecf44f0c15\" (UID: \"8e98c531-00d6-4bf7-b75a-21ecf44f0c15\") " Jul 16 00:07:21.938669 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:21.938626 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e98c531-00d6-4bf7-b75a-21ecf44f0c15-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "8e98c531-00d6-4bf7-b75a-21ecf44f0c15" (UID: "8e98c531-00d6-4bf7-b75a-21ecf44f0c15"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 00:07:21.938669 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:21.938642 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e98c531-00d6-4bf7-b75a-21ecf44f0c15-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "8e98c531-00d6-4bf7-b75a-21ecf44f0c15" (UID: "8e98c531-00d6-4bf7-b75a-21ecf44f0c15"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 00:07:21.938857 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:21.938837 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e98c531-00d6-4bf7-b75a-21ecf44f0c15-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "8e98c531-00d6-4bf7-b75a-21ecf44f0c15" (UID: "8e98c531-00d6-4bf7-b75a-21ecf44f0c15"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 00:07:21.938931 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:21.938863 2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e98c531-00d6-4bf7-b75a-21ecf44f0c15-workload-socket\") on node \"ip-10-0-128-197.ec2.internal\" DevicePath \"\"" Jul 16 00:07:21.938931 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:21.938883 2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e98c531-00d6-4bf7-b75a-21ecf44f0c15-workload-certs\") on node \"ip-10-0-128-197.ec2.internal\" DevicePath \"\"" Jul 16 00:07:21.939094 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:21.939068 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e98c531-00d6-4bf7-b75a-21ecf44f0c15-istio-data" (OuterVolumeSpecName: "istio-data") pod "8e98c531-00d6-4bf7-b75a-21ecf44f0c15" (UID: "8e98c531-00d6-4bf7-b75a-21ecf44f0c15"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 00:07:21.939189 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:21.939170 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e98c531-00d6-4bf7-b75a-21ecf44f0c15-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "8e98c531-00d6-4bf7-b75a-21ecf44f0c15" (UID: "8e98c531-00d6-4bf7-b75a-21ecf44f0c15"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 00:07:21.940571 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:21.940547 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e98c531-00d6-4bf7-b75a-21ecf44f0c15-istio-token" (OuterVolumeSpecName: "istio-token") pod "8e98c531-00d6-4bf7-b75a-21ecf44f0c15" (UID: "8e98c531-00d6-4bf7-b75a-21ecf44f0c15"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 00:07:21.940679 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:21.940610 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e98c531-00d6-4bf7-b75a-21ecf44f0c15-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "8e98c531-00d6-4bf7-b75a-21ecf44f0c15" (UID: "8e98c531-00d6-4bf7-b75a-21ecf44f0c15"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 00:07:21.940780 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:21.940763 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8e98c531-00d6-4bf7-b75a-21ecf44f0c15-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "8e98c531-00d6-4bf7-b75a-21ecf44f0c15" (UID: "8e98c531-00d6-4bf7-b75a-21ecf44f0c15"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 16 00:07:21.940823 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:21.940781 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e98c531-00d6-4bf7-b75a-21ecf44f0c15-kube-api-access-cgswv" (OuterVolumeSpecName: "kube-api-access-cgswv") pod "8e98c531-00d6-4bf7-b75a-21ecf44f0c15" (UID: "8e98c531-00d6-4bf7-b75a-21ecf44f0c15"). InnerVolumeSpecName "kube-api-access-cgswv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 00:07:22.040216 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:22.040191 2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e98c531-00d6-4bf7-b75a-21ecf44f0c15-credential-socket\") on node \"ip-10-0-128-197.ec2.internal\" DevicePath \"\"" Jul 16 00:07:22.040216 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:22.040217 2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e98c531-00d6-4bf7-b75a-21ecf44f0c15-istiod-ca-cert\") on node \"ip-10-0-128-197.ec2.internal\" DevicePath \"\"" Jul 16 00:07:22.040403 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:22.040229 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cgswv\" (UniqueName: \"kubernetes.io/projected/8e98c531-00d6-4bf7-b75a-21ecf44f0c15-kube-api-access-cgswv\") on node \"ip-10-0-128-197.ec2.internal\" DevicePath \"\"" Jul 16 00:07:22.040403 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:22.040242 2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e98c531-00d6-4bf7-b75a-21ecf44f0c15-istio-token\") on node \"ip-10-0-128-197.ec2.internal\" DevicePath \"\"" Jul 16 00:07:22.040403 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:22.040254 2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e98c531-00d6-4bf7-b75a-21ecf44f0c15-istio-data\") on node \"ip-10-0-128-197.ec2.internal\" DevicePath \"\"" Jul 16 00:07:22.040403 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:22.040263 2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e98c531-00d6-4bf7-b75a-21ecf44f0c15-istio-podinfo\") on node \"ip-10-0-128-197.ec2.internal\" DevicePath \"\"" Jul 16 00:07:22.040403 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:22.040274 2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e98c531-00d6-4bf7-b75a-21ecf44f0c15-istio-envoy\") on node \"ip-10-0-128-197.ec2.internal\" DevicePath \"\"" Jul 16 00:07:22.264432 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:22.264393 2563 generic.go:356] "Generic (PLEG): container finished" podID="8e98c531-00d6-4bf7-b75a-21ecf44f0c15" containerID="6a73a7f6b2a9866a786dbb35989027cc8f9ee0624ce18915cf2e3dc9dd8e07d2" exitCode=0 Jul 16 00:07:22.264618 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:22.264454 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-a22338-openshift-default-75b5ff7965-zlgpp" Jul 16 00:07:22.264618 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:22.264473 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a22338-openshift-default-75b5ff7965-zlgpp" event={"ID":"8e98c531-00d6-4bf7-b75a-21ecf44f0c15","Type":"ContainerDied","Data":"6a73a7f6b2a9866a786dbb35989027cc8f9ee0624ce18915cf2e3dc9dd8e07d2"} Jul 16 00:07:22.264618 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:22.264536 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a22338-openshift-default-75b5ff7965-zlgpp" event={"ID":"8e98c531-00d6-4bf7-b75a-21ecf44f0c15","Type":"ContainerDied","Data":"c2518e71746a72a0e1f74886c99408d759de78f460e716faa77fc63b06bed126"} Jul 16 00:07:22.264618 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:22.264565 2563 scope.go:117] "RemoveContainer" containerID="6a73a7f6b2a9866a786dbb35989027cc8f9ee0624ce18915cf2e3dc9dd8e07d2" Jul 16 00:07:22.274356 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:22.274340 2563 scope.go:117] "RemoveContainer" containerID="4729c9d99925d377b4d028db5a1a309db274f5f40b802b8df6b89d49c6085ae1" Jul 16 00:07:22.282158 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:22.282141 2563 scope.go:117] "RemoveContainer" containerID="6a73a7f6b2a9866a786dbb35989027cc8f9ee0624ce18915cf2e3dc9dd8e07d2" Jul 16 00:07:22.282421 ip-10-0-128-197 kubenswrapper[2563]: E0716 00:07:22.282400 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a73a7f6b2a9866a786dbb35989027cc8f9ee0624ce18915cf2e3dc9dd8e07d2\": container with ID starting with 6a73a7f6b2a9866a786dbb35989027cc8f9ee0624ce18915cf2e3dc9dd8e07d2 not found: ID does not exist" containerID="6a73a7f6b2a9866a786dbb35989027cc8f9ee0624ce18915cf2e3dc9dd8e07d2" Jul 16 00:07:22.282495 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:22.282427 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a73a7f6b2a9866a786dbb35989027cc8f9ee0624ce18915cf2e3dc9dd8e07d2"} err="failed to get container status \"6a73a7f6b2a9866a786dbb35989027cc8f9ee0624ce18915cf2e3dc9dd8e07d2\": rpc error: code = NotFound desc = could not find container \"6a73a7f6b2a9866a786dbb35989027cc8f9ee0624ce18915cf2e3dc9dd8e07d2\": container with ID starting with 6a73a7f6b2a9866a786dbb35989027cc8f9ee0624ce18915cf2e3dc9dd8e07d2 not found: ID does not exist" Jul 16 00:07:22.282495 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:22.282445 2563 scope.go:117] "RemoveContainer" containerID="4729c9d99925d377b4d028db5a1a309db274f5f40b802b8df6b89d49c6085ae1" Jul 16 00:07:22.282663 ip-10-0-128-197 kubenswrapper[2563]: E0716 00:07:22.282648 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4729c9d99925d377b4d028db5a1a309db274f5f40b802b8df6b89d49c6085ae1\": container with ID starting with 4729c9d99925d377b4d028db5a1a309db274f5f40b802b8df6b89d49c6085ae1 not found: ID does not exist" containerID="4729c9d99925d377b4d028db5a1a309db274f5f40b802b8df6b89d49c6085ae1" Jul 16 00:07:22.282705 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:22.282667 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4729c9d99925d377b4d028db5a1a309db274f5f40b802b8df6b89d49c6085ae1"} err="failed to get container status \"4729c9d99925d377b4d028db5a1a309db274f5f40b802b8df6b89d49c6085ae1\": rpc error: code = NotFound desc = could not find container \"4729c9d99925d377b4d028db5a1a309db274f5f40b802b8df6b89d49c6085ae1\": container with ID starting with 4729c9d99925d377b4d028db5a1a309db274f5f40b802b8df6b89d49c6085ae1 not found: ID does not exist" Jul 16 00:07:22.285765 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:22.285744 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-a22338-openshift-default-75b5ff7965-zlgpp"] Jul 16 00:07:22.289182 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:22.289163 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-a22338-openshift-default-75b5ff7965-zlgpp"] Jul 16 00:07:23.112513 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:23.112481 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:07:23.169383 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:23.169351 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8e98c531-00d6-4bf7-b75a-21ecf44f0c15" path="/var/lib/kubelet/pods/8e98c531-00d6-4bf7-b75a-21ecf44f0c15/volumes" Jul 16 00:07:28.746563 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:28.746529 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:07:28.807758 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:28.807674 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:07:29.002810 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:29.002734 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-eae7a4-openshift-default-655765b455-87hdc"] Jul 16 00:07:29.002988 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:29.002952 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-eae7a4-openshift-default-655765b455-87hdc" podUID="7f53994d-00f8-49b1-9743-427275ddf86e" containerName="istio-proxy" containerID="cri-o://f2a8521df85b5518f4e602cb42c2132fb4b502a0e4860789d88535187ad78735" gracePeriod=30 Jul 16 00:07:29.018062 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:29.018015 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:07:29.234499 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:29.234464 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:07:29.585137 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:29.585097 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:07:29.880599 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:29.880521 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-ef3a1a-openshift-default-6b6bf9786c-bwf8h"] Jul 16 00:07:29.880954 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:29.880939 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e98c531-00d6-4bf7-b75a-21ecf44f0c15" containerName="istio-proxy" Jul 16 00:07:29.881027 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:29.880955 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e98c531-00d6-4bf7-b75a-21ecf44f0c15" containerName="istio-proxy" Jul 16 00:07:29.881027 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:29.880983 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e98c531-00d6-4bf7-b75a-21ecf44f0c15" containerName="istio-proxy" Jul 16 00:07:29.881027 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:29.880991 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e98c531-00d6-4bf7-b75a-21ecf44f0c15" containerName="istio-proxy" Jul 16 00:07:29.881131 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:29.881076 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="8e98c531-00d6-4bf7-b75a-21ecf44f0c15" containerName="istio-proxy" Jul 16 00:07:29.881131 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:29.881084 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="8e98c531-00d6-4bf7-b75a-21ecf44f0c15" containerName="istio-proxy" Jul 16 00:07:29.883220 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:29.883201 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-ef3a1a-openshift-default-6b6bf9786c-bwf8h" Jul 16 00:07:29.885167 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:29.885149 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-ef3a1a-openshift-default-dockercfg-gcnnf\"" Jul 16 00:07:29.894461 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:29.894442 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-ef3a1a-openshift-default-6b6bf9786c-bwf8h"] Jul 16 00:07:29.908022 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:29.907962 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:07:30.014222 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:30.014191 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74652b67-44c7-4d41-b07b-c9b83a3952d8-istio-token\") pod \"e2e-shared-a-ef3a1a-openshift-default-6b6bf9786c-bwf8h\" (UID: \"74652b67-44c7-4d41-b07b-c9b83a3952d8\") " pod="openshift-ingress/e2e-shared-a-ef3a1a-openshift-default-6b6bf9786c-bwf8h" Jul 16 00:07:30.014396 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:30.014239 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74652b67-44c7-4d41-b07b-c9b83a3952d8-credential-socket\") pod \"e2e-shared-a-ef3a1a-openshift-default-6b6bf9786c-bwf8h\" (UID: \"74652b67-44c7-4d41-b07b-c9b83a3952d8\") " pod="openshift-ingress/e2e-shared-a-ef3a1a-openshift-default-6b6bf9786c-bwf8h" Jul 16 00:07:30.014396 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:30.014300 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74652b67-44c7-4d41-b07b-c9b83a3952d8-istio-podinfo\") pod \"e2e-shared-a-ef3a1a-openshift-default-6b6bf9786c-bwf8h\" (UID: \"74652b67-44c7-4d41-b07b-c9b83a3952d8\") " pod="openshift-ingress/e2e-shared-a-ef3a1a-openshift-default-6b6bf9786c-bwf8h" Jul 16 00:07:30.014396 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:30.014342 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5bc4\" (UniqueName: \"kubernetes.io/projected/74652b67-44c7-4d41-b07b-c9b83a3952d8-kube-api-access-c5bc4\") pod \"e2e-shared-a-ef3a1a-openshift-default-6b6bf9786c-bwf8h\" (UID: \"74652b67-44c7-4d41-b07b-c9b83a3952d8\") " pod="openshift-ingress/e2e-shared-a-ef3a1a-openshift-default-6b6bf9786c-bwf8h" Jul 16 00:07:30.014396 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:30.014365 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74652b67-44c7-4d41-b07b-c9b83a3952d8-workload-certs\") pod \"e2e-shared-a-ef3a1a-openshift-default-6b6bf9786c-bwf8h\" (UID: \"74652b67-44c7-4d41-b07b-c9b83a3952d8\") " pod="openshift-ingress/e2e-shared-a-ef3a1a-openshift-default-6b6bf9786c-bwf8h" Jul 16 00:07:30.014396 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:30.014388 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74652b67-44c7-4d41-b07b-c9b83a3952d8-istio-data\") pod \"e2e-shared-a-ef3a1a-openshift-default-6b6bf9786c-bwf8h\" (UID: \"74652b67-44c7-4d41-b07b-c9b83a3952d8\") " pod="openshift-ingress/e2e-shared-a-ef3a1a-openshift-default-6b6bf9786c-bwf8h" Jul 16 00:07:30.014564 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:30.014437 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74652b67-44c7-4d41-b07b-c9b83a3952d8-istiod-ca-cert\") pod \"e2e-shared-a-ef3a1a-openshift-default-6b6bf9786c-bwf8h\" (UID: \"74652b67-44c7-4d41-b07b-c9b83a3952d8\") " pod="openshift-ingress/e2e-shared-a-ef3a1a-openshift-default-6b6bf9786c-bwf8h" Jul 16 00:07:30.014564 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:30.014487 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74652b67-44c7-4d41-b07b-c9b83a3952d8-istio-envoy\") pod \"e2e-shared-a-ef3a1a-openshift-default-6b6bf9786c-bwf8h\" (UID: \"74652b67-44c7-4d41-b07b-c9b83a3952d8\") " pod="openshift-ingress/e2e-shared-a-ef3a1a-openshift-default-6b6bf9786c-bwf8h" Jul 16 00:07:30.014564 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:30.014519 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74652b67-44c7-4d41-b07b-c9b83a3952d8-workload-socket\") pod \"e2e-shared-a-ef3a1a-openshift-default-6b6bf9786c-bwf8h\" (UID: \"74652b67-44c7-4d41-b07b-c9b83a3952d8\") " pod="openshift-ingress/e2e-shared-a-ef3a1a-openshift-default-6b6bf9786c-bwf8h" Jul 16 00:07:30.035368 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:30.035335 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:07:30.115215 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:30.115187 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74652b67-44c7-4d41-b07b-c9b83a3952d8-workload-certs\") pod \"e2e-shared-a-ef3a1a-openshift-default-6b6bf9786c-bwf8h\" (UID: \"74652b67-44c7-4d41-b07b-c9b83a3952d8\") " pod="openshift-ingress/e2e-shared-a-ef3a1a-openshift-default-6b6bf9786c-bwf8h" Jul 16 00:07:30.115215 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:30.115219 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74652b67-44c7-4d41-b07b-c9b83a3952d8-istio-data\") pod \"e2e-shared-a-ef3a1a-openshift-default-6b6bf9786c-bwf8h\" (UID: \"74652b67-44c7-4d41-b07b-c9b83a3952d8\") " pod="openshift-ingress/e2e-shared-a-ef3a1a-openshift-default-6b6bf9786c-bwf8h" Jul 16 00:07:30.115475 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:30.115275 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74652b67-44c7-4d41-b07b-c9b83a3952d8-istiod-ca-cert\") pod \"e2e-shared-a-ef3a1a-openshift-default-6b6bf9786c-bwf8h\" (UID: \"74652b67-44c7-4d41-b07b-c9b83a3952d8\") " pod="openshift-ingress/e2e-shared-a-ef3a1a-openshift-default-6b6bf9786c-bwf8h" Jul 16 00:07:30.115475 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:30.115317 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74652b67-44c7-4d41-b07b-c9b83a3952d8-istio-envoy\") pod \"e2e-shared-a-ef3a1a-openshift-default-6b6bf9786c-bwf8h\" (UID: \"74652b67-44c7-4d41-b07b-c9b83a3952d8\") " pod="openshift-ingress/e2e-shared-a-ef3a1a-openshift-default-6b6bf9786c-bwf8h" Jul 16 00:07:30.115475 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:30.115359 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74652b67-44c7-4d41-b07b-c9b83a3952d8-workload-socket\") pod \"e2e-shared-a-ef3a1a-openshift-default-6b6bf9786c-bwf8h\" (UID: \"74652b67-44c7-4d41-b07b-c9b83a3952d8\") " pod="openshift-ingress/e2e-shared-a-ef3a1a-openshift-default-6b6bf9786c-bwf8h" Jul 16 00:07:30.115475 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:30.115428 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74652b67-44c7-4d41-b07b-c9b83a3952d8-istio-token\") pod \"e2e-shared-a-ef3a1a-openshift-default-6b6bf9786c-bwf8h\" (UID: \"74652b67-44c7-4d41-b07b-c9b83a3952d8\") " pod="openshift-ingress/e2e-shared-a-ef3a1a-openshift-default-6b6bf9786c-bwf8h" Jul 16 00:07:30.115475 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:30.115474 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74652b67-44c7-4d41-b07b-c9b83a3952d8-credential-socket\") pod \"e2e-shared-a-ef3a1a-openshift-default-6b6bf9786c-bwf8h\" (UID: \"74652b67-44c7-4d41-b07b-c9b83a3952d8\") " pod="openshift-ingress/e2e-shared-a-ef3a1a-openshift-default-6b6bf9786c-bwf8h" Jul 16 00:07:30.115724 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:30.115503 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74652b67-44c7-4d41-b07b-c9b83a3952d8-istio-podinfo\") pod \"e2e-shared-a-ef3a1a-openshift-default-6b6bf9786c-bwf8h\" (UID: \"74652b67-44c7-4d41-b07b-c9b83a3952d8\") " pod="openshift-ingress/e2e-shared-a-ef3a1a-openshift-default-6b6bf9786c-bwf8h" Jul 16 00:07:30.115724 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:30.115544 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5bc4\" (UniqueName: \"kubernetes.io/projected/74652b67-44c7-4d41-b07b-c9b83a3952d8-kube-api-access-c5bc4\") pod \"e2e-shared-a-ef3a1a-openshift-default-6b6bf9786c-bwf8h\" (UID: \"74652b67-44c7-4d41-b07b-c9b83a3952d8\") " pod="openshift-ingress/e2e-shared-a-ef3a1a-openshift-default-6b6bf9786c-bwf8h" Jul 16 00:07:30.115724 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:30.115629 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74652b67-44c7-4d41-b07b-c9b83a3952d8-workload-certs\") pod \"e2e-shared-a-ef3a1a-openshift-default-6b6bf9786c-bwf8h\" (UID: \"74652b67-44c7-4d41-b07b-c9b83a3952d8\") " pod="openshift-ingress/e2e-shared-a-ef3a1a-openshift-default-6b6bf9786c-bwf8h" Jul 16 00:07:30.116060 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:30.116034 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74652b67-44c7-4d41-b07b-c9b83a3952d8-istiod-ca-cert\") pod \"e2e-shared-a-ef3a1a-openshift-default-6b6bf9786c-bwf8h\" (UID: \"74652b67-44c7-4d41-b07b-c9b83a3952d8\") " pod="openshift-ingress/e2e-shared-a-ef3a1a-openshift-default-6b6bf9786c-bwf8h" Jul 16 00:07:30.116164 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:30.116094 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74652b67-44c7-4d41-b07b-c9b83a3952d8-workload-socket\") pod \"e2e-shared-a-ef3a1a-openshift-default-6b6bf9786c-bwf8h\" (UID: \"74652b67-44c7-4d41-b07b-c9b83a3952d8\") " pod="openshift-ingress/e2e-shared-a-ef3a1a-openshift-default-6b6bf9786c-bwf8h" Jul 16 00:07:30.116164 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:30.116122 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74652b67-44c7-4d41-b07b-c9b83a3952d8-istio-data\") pod \"e2e-shared-a-ef3a1a-openshift-default-6b6bf9786c-bwf8h\" (UID: \"74652b67-44c7-4d41-b07b-c9b83a3952d8\") " pod="openshift-ingress/e2e-shared-a-ef3a1a-openshift-default-6b6bf9786c-bwf8h" Jul 16 00:07:30.116164 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:30.116140 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74652b67-44c7-4d41-b07b-c9b83a3952d8-credential-socket\") pod \"e2e-shared-a-ef3a1a-openshift-default-6b6bf9786c-bwf8h\" (UID: \"74652b67-44c7-4d41-b07b-c9b83a3952d8\") " pod="openshift-ingress/e2e-shared-a-ef3a1a-openshift-default-6b6bf9786c-bwf8h" Jul 16 00:07:30.117596 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:30.117578 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74652b67-44c7-4d41-b07b-c9b83a3952d8-istio-envoy\") pod \"e2e-shared-a-ef3a1a-openshift-default-6b6bf9786c-bwf8h\" (UID: \"74652b67-44c7-4d41-b07b-c9b83a3952d8\") " pod="openshift-ingress/e2e-shared-a-ef3a1a-openshift-default-6b6bf9786c-bwf8h" Jul 16 00:07:30.117922 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:30.117903 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74652b67-44c7-4d41-b07b-c9b83a3952d8-istio-podinfo\") pod \"e2e-shared-a-ef3a1a-openshift-default-6b6bf9786c-bwf8h\" (UID: \"74652b67-44c7-4d41-b07b-c9b83a3952d8\") " pod="openshift-ingress/e2e-shared-a-ef3a1a-openshift-default-6b6bf9786c-bwf8h" Jul 16 00:07:30.123316 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:30.123286 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74652b67-44c7-4d41-b07b-c9b83a3952d8-istio-token\") pod \"e2e-shared-a-ef3a1a-openshift-default-6b6bf9786c-bwf8h\" (UID: \"74652b67-44c7-4d41-b07b-c9b83a3952d8\") " pod="openshift-ingress/e2e-shared-a-ef3a1a-openshift-default-6b6bf9786c-bwf8h" Jul 16 00:07:30.123703 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:30.123686 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5bc4\" (UniqueName: \"kubernetes.io/projected/74652b67-44c7-4d41-b07b-c9b83a3952d8-kube-api-access-c5bc4\") pod \"e2e-shared-a-ef3a1a-openshift-default-6b6bf9786c-bwf8h\" (UID: \"74652b67-44c7-4d41-b07b-c9b83a3952d8\") " pod="openshift-ingress/e2e-shared-a-ef3a1a-openshift-default-6b6bf9786c-bwf8h" Jul 16 00:07:30.193675 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:30.193606 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-ef3a1a-openshift-default-6b6bf9786c-bwf8h" Jul 16 00:07:30.330314 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:30.330284 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-ef3a1a-openshift-default-6b6bf9786c-bwf8h"] Jul 16 00:07:30.332455 ip-10-0-128-197 kubenswrapper[2563]: W0716 00:07:30.332431 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74652b67_44c7_4d41_b07b_c9b83a3952d8.slice/crio-6ba3a9f962c9349ab39ff619557ac37989b53d09b29969d22ff8b16235a89fa8 WatchSource:0}: Error finding container 6ba3a9f962c9349ab39ff619557ac37989b53d09b29969d22ff8b16235a89fa8: Status 404 returned error can't find the container with id 6ba3a9f962c9349ab39ff619557ac37989b53d09b29969d22ff8b16235a89fa8 Jul 16 00:07:30.334544 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:30.334513 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 16 00:07:30.334635 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:30.334596 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 16 00:07:30.334696 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:30.334640 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 16 00:07:31.313398 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:31.313363 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-ef3a1a-openshift-default-6b6bf9786c-bwf8h" event={"ID":"74652b67-44c7-4d41-b07b-c9b83a3952d8","Type":"ContainerStarted","Data":"cf5d38dcaf7b79026a3c0d34e43ea0720c9a41d3e1a613c3d858c1416a9c73ab"} Jul 16 00:07:31.313783 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:31.313422 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-ef3a1a-openshift-default-6b6bf9786c-bwf8h" event={"ID":"74652b67-44c7-4d41-b07b-c9b83a3952d8","Type":"ContainerStarted","Data":"6ba3a9f962c9349ab39ff619557ac37989b53d09b29969d22ff8b16235a89fa8"} Jul 16 00:07:31.332496 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:31.332437 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-ef3a1a-openshift-default-6b6bf9786c-bwf8h" podStartSLOduration=2.332416002 podStartE2EDuration="2.332416002s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 00:07:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 00:07:31.33020041 +0000 UTC m=+1866.703658398" watchObservedRunningTime="2026-07-16 00:07:31.332416002 +0000 UTC m=+1866.705873967" Jul 16 00:07:32.194127 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:32.194096 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-ef3a1a-openshift-default-6b6bf9786c-bwf8h" Jul 16 00:07:32.198636 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:32.198610 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-ef3a1a-openshift-default-6b6bf9786c-bwf8h" Jul 16 00:07:32.212350 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:32.212327 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:07:32.284333 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:32.284296 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:07:32.318769 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:32.318742 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-ef3a1a-openshift-default-6b6bf9786c-bwf8h" Jul 16 00:07:32.319827 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:32.319807 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-ef3a1a-openshift-default-6b6bf9786c-bwf8h" Jul 16 00:07:32.349476 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:32.349445 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:07:32.494641 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:32.494606 2563 patch_prober.go:28] interesting pod/e2e-api-b-eae7a4-openshift-default-655765b455-87hdc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 16 00:07:32.494819 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:32.494668 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-eae7a4-openshift-default-655765b455-87hdc" podUID="7f53994d-00f8-49b1-9743-427275ddf86e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 16 00:07:32.646713 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:32.646677 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:07:32.902652 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:32.902566 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:07:33.004342 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:33.004311 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:07:34.260840 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:34.260818 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-eae7a4-openshift-default-655765b455-87hdc" Jul 16 00:07:34.329489 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:34.329419 2563 generic.go:356] "Generic (PLEG): container finished" podID="7f53994d-00f8-49b1-9743-427275ddf86e" containerID="f2a8521df85b5518f4e602cb42c2132fb4b502a0e4860789d88535187ad78735" exitCode=0 Jul 16 00:07:34.329489 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:34.329461 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-eae7a4-openshift-default-655765b455-87hdc" Jul 16 00:07:34.329646 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:34.329496 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-eae7a4-openshift-default-655765b455-87hdc" event={"ID":"7f53994d-00f8-49b1-9743-427275ddf86e","Type":"ContainerDied","Data":"f2a8521df85b5518f4e602cb42c2132fb4b502a0e4860789d88535187ad78735"} Jul 16 00:07:34.329646 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:34.329547 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-eae7a4-openshift-default-655765b455-87hdc" event={"ID":"7f53994d-00f8-49b1-9743-427275ddf86e","Type":"ContainerDied","Data":"5d4402cd7f37ecb29e20607a00d46e68cd5cfb38d9577cc10e5ce1723433a5df"} Jul 16 00:07:34.329646 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:34.329571 2563 scope.go:117] "RemoveContainer" containerID="f2a8521df85b5518f4e602cb42c2132fb4b502a0e4860789d88535187ad78735" Jul 16 00:07:34.338122 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:34.338107 2563 scope.go:117] "RemoveContainer" containerID="7ca382af7d76f4f278711fbe5c54c7c9343b443368b3ec7ec3ffc2bd3d83e734" Jul 16 00:07:34.345673 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:34.345657 2563 scope.go:117] "RemoveContainer" containerID="f2a8521df85b5518f4e602cb42c2132fb4b502a0e4860789d88535187ad78735" Jul 16 00:07:34.345906 ip-10-0-128-197 kubenswrapper[2563]: E0716 00:07:34.345886 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2a8521df85b5518f4e602cb42c2132fb4b502a0e4860789d88535187ad78735\": container with ID starting with f2a8521df85b5518f4e602cb42c2132fb4b502a0e4860789d88535187ad78735 not found: ID does not exist" containerID="f2a8521df85b5518f4e602cb42c2132fb4b502a0e4860789d88535187ad78735" Jul 16 00:07:34.345952 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:34.345915 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2a8521df85b5518f4e602cb42c2132fb4b502a0e4860789d88535187ad78735"} err="failed to get container status \"f2a8521df85b5518f4e602cb42c2132fb4b502a0e4860789d88535187ad78735\": rpc error: code = NotFound desc = could not find container \"f2a8521df85b5518f4e602cb42c2132fb4b502a0e4860789d88535187ad78735\": container with ID starting with f2a8521df85b5518f4e602cb42c2132fb4b502a0e4860789d88535187ad78735 not found: ID does not exist" Jul 16 00:07:34.345952 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:34.345931 2563 scope.go:117] "RemoveContainer" containerID="7ca382af7d76f4f278711fbe5c54c7c9343b443368b3ec7ec3ffc2bd3d83e734" Jul 16 00:07:34.346177 ip-10-0-128-197 kubenswrapper[2563]: E0716 00:07:34.346162 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ca382af7d76f4f278711fbe5c54c7c9343b443368b3ec7ec3ffc2bd3d83e734\": container with ID starting with 7ca382af7d76f4f278711fbe5c54c7c9343b443368b3ec7ec3ffc2bd3d83e734 not found: ID does not exist" containerID="7ca382af7d76f4f278711fbe5c54c7c9343b443368b3ec7ec3ffc2bd3d83e734" Jul 16 00:07:34.346220 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:34.346182 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ca382af7d76f4f278711fbe5c54c7c9343b443368b3ec7ec3ffc2bd3d83e734"} err="failed to get container status \"7ca382af7d76f4f278711fbe5c54c7c9343b443368b3ec7ec3ffc2bd3d83e734\": rpc error: code = NotFound desc = could not find container \"7ca382af7d76f4f278711fbe5c54c7c9343b443368b3ec7ec3ffc2bd3d83e734\": container with ID starting with 7ca382af7d76f4f278711fbe5c54c7c9343b443368b3ec7ec3ffc2bd3d83e734 not found: ID does not exist" Jul 16 00:07:34.361639 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:34.361622 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f53994d-00f8-49b1-9743-427275ddf86e-istio-token\") pod \"7f53994d-00f8-49b1-9743-427275ddf86e\" (UID: \"7f53994d-00f8-49b1-9743-427275ddf86e\") " Jul 16 00:07:34.361705 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:34.361649 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f53994d-00f8-49b1-9743-427275ddf86e-istio-data\") pod \"7f53994d-00f8-49b1-9743-427275ddf86e\" (UID: \"7f53994d-00f8-49b1-9743-427275ddf86e\") " Jul 16 00:07:34.361705 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:34.361690 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-db5kd\" (UniqueName: \"kubernetes.io/projected/7f53994d-00f8-49b1-9743-427275ddf86e-kube-api-access-db5kd\") pod \"7f53994d-00f8-49b1-9743-427275ddf86e\" (UID: \"7f53994d-00f8-49b1-9743-427275ddf86e\") " Jul 16 00:07:34.361781 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:34.361709 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f53994d-00f8-49b1-9743-427275ddf86e-istio-envoy\") pod \"7f53994d-00f8-49b1-9743-427275ddf86e\" (UID: \"7f53994d-00f8-49b1-9743-427275ddf86e\") " Jul 16 00:07:34.361781 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:34.361760 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f53994d-00f8-49b1-9743-427275ddf86e-istio-podinfo\") pod \"7f53994d-00f8-49b1-9743-427275ddf86e\" (UID: \"7f53994d-00f8-49b1-9743-427275ddf86e\") " Jul 16 00:07:34.361879 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:34.361792 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f53994d-00f8-49b1-9743-427275ddf86e-workload-certs\") pod \"7f53994d-00f8-49b1-9743-427275ddf86e\" (UID: \"7f53994d-00f8-49b1-9743-427275ddf86e\") " Jul 16 00:07:34.361879 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:34.361818 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f53994d-00f8-49b1-9743-427275ddf86e-credential-socket\") pod \"7f53994d-00f8-49b1-9743-427275ddf86e\" (UID: \"7f53994d-00f8-49b1-9743-427275ddf86e\") " Jul 16 00:07:34.361879 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:34.361838 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f53994d-00f8-49b1-9743-427275ddf86e-workload-socket\") pod \"7f53994d-00f8-49b1-9743-427275ddf86e\" (UID: \"7f53994d-00f8-49b1-9743-427275ddf86e\") " Jul 16 00:07:34.362049 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:34.361905 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f53994d-00f8-49b1-9743-427275ddf86e-istiod-ca-cert\") pod \"7f53994d-00f8-49b1-9743-427275ddf86e\" (UID: \"7f53994d-00f8-49b1-9743-427275ddf86e\") " Jul 16 00:07:34.362266 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:34.362236 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f53994d-00f8-49b1-9743-427275ddf86e-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "7f53994d-00f8-49b1-9743-427275ddf86e" (UID: "7f53994d-00f8-49b1-9743-427275ddf86e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 00:07:34.362430 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:34.362410 2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f53994d-00f8-49b1-9743-427275ddf86e-workload-certs\") on node \"ip-10-0-128-197.ec2.internal\" DevicePath \"\"" Jul 16 00:07:34.362501 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:34.362470 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f53994d-00f8-49b1-9743-427275ddf86e-istio-data" (OuterVolumeSpecName: "istio-data") pod "7f53994d-00f8-49b1-9743-427275ddf86e" (UID: "7f53994d-00f8-49b1-9743-427275ddf86e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 00:07:34.362667 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:34.362643 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f53994d-00f8-49b1-9743-427275ddf86e-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "7f53994d-00f8-49b1-9743-427275ddf86e" (UID: "7f53994d-00f8-49b1-9743-427275ddf86e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 00:07:34.362743 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:34.362652 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f53994d-00f8-49b1-9743-427275ddf86e-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "7f53994d-00f8-49b1-9743-427275ddf86e" (UID: "7f53994d-00f8-49b1-9743-427275ddf86e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 00:07:34.362997 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:34.362960 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f53994d-00f8-49b1-9743-427275ddf86e-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "7f53994d-00f8-49b1-9743-427275ddf86e" (UID: "7f53994d-00f8-49b1-9743-427275ddf86e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 00:07:34.363928 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:34.363901 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f53994d-00f8-49b1-9743-427275ddf86e-istio-token" (OuterVolumeSpecName: "istio-token") pod "7f53994d-00f8-49b1-9743-427275ddf86e" (UID: "7f53994d-00f8-49b1-9743-427275ddf86e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 00:07:34.364080 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:34.364031 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f53994d-00f8-49b1-9743-427275ddf86e-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "7f53994d-00f8-49b1-9743-427275ddf86e" (UID: "7f53994d-00f8-49b1-9743-427275ddf86e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 00:07:34.364264 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:34.364243 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7f53994d-00f8-49b1-9743-427275ddf86e-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "7f53994d-00f8-49b1-9743-427275ddf86e" (UID: "7f53994d-00f8-49b1-9743-427275ddf86e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 16 00:07:34.364372 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:34.364314 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f53994d-00f8-49b1-9743-427275ddf86e-kube-api-access-db5kd" (OuterVolumeSpecName: "kube-api-access-db5kd") pod "7f53994d-00f8-49b1-9743-427275ddf86e" (UID: "7f53994d-00f8-49b1-9743-427275ddf86e"). InnerVolumeSpecName "kube-api-access-db5kd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 00:07:34.463851 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:34.463814 2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f53994d-00f8-49b1-9743-427275ddf86e-istio-podinfo\") on node \"ip-10-0-128-197.ec2.internal\" DevicePath \"\"" Jul 16 00:07:34.463851 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:34.463846 2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f53994d-00f8-49b1-9743-427275ddf86e-credential-socket\") on node \"ip-10-0-128-197.ec2.internal\" DevicePath \"\"" Jul 16 00:07:34.463851 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:34.463858 2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f53994d-00f8-49b1-9743-427275ddf86e-workload-socket\") on node \"ip-10-0-128-197.ec2.internal\" DevicePath \"\"" Jul 16 00:07:34.464103 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:34.463867 2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f53994d-00f8-49b1-9743-427275ddf86e-istiod-ca-cert\") on node \"ip-10-0-128-197.ec2.internal\" DevicePath \"\"" Jul 16 00:07:34.464103 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:34.463876 2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f53994d-00f8-49b1-9743-427275ddf86e-istio-token\") on node \"ip-10-0-128-197.ec2.internal\" DevicePath \"\"" Jul 16 00:07:34.464103 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:34.463884 2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f53994d-00f8-49b1-9743-427275ddf86e-istio-data\") on node \"ip-10-0-128-197.ec2.internal\" DevicePath \"\"" Jul 16 00:07:34.464103 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:34.463891 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-db5kd\" (UniqueName: \"kubernetes.io/projected/7f53994d-00f8-49b1-9743-427275ddf86e-kube-api-access-db5kd\") on node \"ip-10-0-128-197.ec2.internal\" DevicePath \"\"" Jul 16 00:07:34.464103 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:34.463902 2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f53994d-00f8-49b1-9743-427275ddf86e-istio-envoy\") on node \"ip-10-0-128-197.ec2.internal\" DevicePath \"\"" Jul 16 00:07:34.651182 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:34.651152 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-eae7a4-openshift-default-655765b455-87hdc"] Jul 16 00:07:34.654990 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:34.654957 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-eae7a4-openshift-default-655765b455-87hdc"] Jul 16 00:07:35.169837 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:35.169808 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7f53994d-00f8-49b1-9743-427275ddf86e" path="/var/lib/kubelet/pods/7f53994d-00f8-49b1-9743-427275ddf86e/volumes" Jul 16 00:07:37.749913 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:37.749881 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:07:38.100488 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:38.100382 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:07:38.372768 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:38.372689 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-e94b3e-openshift-default-598bcf5c59-nnggv"] Jul 16 00:07:38.373236 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:38.373214 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f53994d-00f8-49b1-9743-427275ddf86e" containerName="istio-proxy" Jul 16 00:07:38.373236 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:38.373237 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f53994d-00f8-49b1-9743-427275ddf86e" containerName="istio-proxy" Jul 16 00:07:38.373391 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:38.373250 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f53994d-00f8-49b1-9743-427275ddf86e" containerName="istio-proxy" Jul 16 00:07:38.373391 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:38.373258 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f53994d-00f8-49b1-9743-427275ddf86e" containerName="istio-proxy" Jul 16 00:07:38.373391 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:38.373340 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="7f53994d-00f8-49b1-9743-427275ddf86e" containerName="istio-proxy" Jul 16 00:07:38.373391 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:38.373354 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="7f53994d-00f8-49b1-9743-427275ddf86e" containerName="istio-proxy" Jul 16 00:07:38.376159 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:38.376136 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-e94b3e-openshift-default-598bcf5c59-nnggv" Jul 16 00:07:38.378557 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:38.378538 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-e94b3e-openshift-default-dockercfg-rq6nm\"" Jul 16 00:07:38.399025 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:38.390845 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-e94b3e-openshift-default-598bcf5c59-nnggv"] Jul 16 00:07:38.406212 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:38.405863 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:07:38.511566 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:38.511523 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a1cb0d5-2afd-42a0-a18f-ac401bfbd9bb-istio-podinfo\") pod \"e2e-shared-b-e94b3e-openshift-default-598bcf5c59-nnggv\" (UID: \"9a1cb0d5-2afd-42a0-a18f-ac401bfbd9bb\") " pod="openshift-ingress/e2e-shared-b-e94b3e-openshift-default-598bcf5c59-nnggv" Jul 16 00:07:38.511796 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:38.511775 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a1cb0d5-2afd-42a0-a18f-ac401bfbd9bb-istio-data\") pod \"e2e-shared-b-e94b3e-openshift-default-598bcf5c59-nnggv\" (UID: \"9a1cb0d5-2afd-42a0-a18f-ac401bfbd9bb\") " pod="openshift-ingress/e2e-shared-b-e94b3e-openshift-default-598bcf5c59-nnggv" Jul 16 00:07:38.511916 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:38.511899 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vs42f\" (UniqueName: \"kubernetes.io/projected/9a1cb0d5-2afd-42a0-a18f-ac401bfbd9bb-kube-api-access-vs42f\") pod \"e2e-shared-b-e94b3e-openshift-default-598bcf5c59-nnggv\" (UID: \"9a1cb0d5-2afd-42a0-a18f-ac401bfbd9bb\") " pod="openshift-ingress/e2e-shared-b-e94b3e-openshift-default-598bcf5c59-nnggv" Jul 16 00:07:38.512091 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:38.512074 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a1cb0d5-2afd-42a0-a18f-ac401bfbd9bb-istio-token\") pod \"e2e-shared-b-e94b3e-openshift-default-598bcf5c59-nnggv\" (UID: \"9a1cb0d5-2afd-42a0-a18f-ac401bfbd9bb\") " pod="openshift-ingress/e2e-shared-b-e94b3e-openshift-default-598bcf5c59-nnggv" Jul 16 00:07:38.512257 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:38.512241 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a1cb0d5-2afd-42a0-a18f-ac401bfbd9bb-workload-certs\") pod \"e2e-shared-b-e94b3e-openshift-default-598bcf5c59-nnggv\" (UID: \"9a1cb0d5-2afd-42a0-a18f-ac401bfbd9bb\") " pod="openshift-ingress/e2e-shared-b-e94b3e-openshift-default-598bcf5c59-nnggv" Jul 16 00:07:38.512359 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:38.512345 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a1cb0d5-2afd-42a0-a18f-ac401bfbd9bb-istio-envoy\") pod \"e2e-shared-b-e94b3e-openshift-default-598bcf5c59-nnggv\" (UID: \"9a1cb0d5-2afd-42a0-a18f-ac401bfbd9bb\") " pod="openshift-ingress/e2e-shared-b-e94b3e-openshift-default-598bcf5c59-nnggv" Jul 16 00:07:38.512484 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:38.512469 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a1cb0d5-2afd-42a0-a18f-ac401bfbd9bb-istiod-ca-cert\") pod \"e2e-shared-b-e94b3e-openshift-default-598bcf5c59-nnggv\" (UID: \"9a1cb0d5-2afd-42a0-a18f-ac401bfbd9bb\") " pod="openshift-ingress/e2e-shared-b-e94b3e-openshift-default-598bcf5c59-nnggv" Jul 16 00:07:38.512624 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:38.512610 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a1cb0d5-2afd-42a0-a18f-ac401bfbd9bb-workload-socket\") pod \"e2e-shared-b-e94b3e-openshift-default-598bcf5c59-nnggv\" (UID: \"9a1cb0d5-2afd-42a0-a18f-ac401bfbd9bb\") " pod="openshift-ingress/e2e-shared-b-e94b3e-openshift-default-598bcf5c59-nnggv" Jul 16 00:07:38.512736 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:38.512722 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a1cb0d5-2afd-42a0-a18f-ac401bfbd9bb-credential-socket\") pod \"e2e-shared-b-e94b3e-openshift-default-598bcf5c59-nnggv\" (UID: \"9a1cb0d5-2afd-42a0-a18f-ac401bfbd9bb\") " pod="openshift-ingress/e2e-shared-b-e94b3e-openshift-default-598bcf5c59-nnggv" Jul 16 00:07:38.614389 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:38.614350 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a1cb0d5-2afd-42a0-a18f-ac401bfbd9bb-istio-podinfo\") pod \"e2e-shared-b-e94b3e-openshift-default-598bcf5c59-nnggv\" (UID: \"9a1cb0d5-2afd-42a0-a18f-ac401bfbd9bb\") " pod="openshift-ingress/e2e-shared-b-e94b3e-openshift-default-598bcf5c59-nnggv" Jul 16 00:07:38.614708 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:38.614684 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a1cb0d5-2afd-42a0-a18f-ac401bfbd9bb-istio-data\") pod \"e2e-shared-b-e94b3e-openshift-default-598bcf5c59-nnggv\" (UID: \"9a1cb0d5-2afd-42a0-a18f-ac401bfbd9bb\") " pod="openshift-ingress/e2e-shared-b-e94b3e-openshift-default-598bcf5c59-nnggv" Jul 16 00:07:38.614847 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:38.614822 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vs42f\" (UniqueName: \"kubernetes.io/projected/9a1cb0d5-2afd-42a0-a18f-ac401bfbd9bb-kube-api-access-vs42f\") pod \"e2e-shared-b-e94b3e-openshift-default-598bcf5c59-nnggv\" (UID: \"9a1cb0d5-2afd-42a0-a18f-ac401bfbd9bb\") " pod="openshift-ingress/e2e-shared-b-e94b3e-openshift-default-598bcf5c59-nnggv" Jul 16 00:07:38.615025 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:38.614874 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a1cb0d5-2afd-42a0-a18f-ac401bfbd9bb-istio-token\") pod \"e2e-shared-b-e94b3e-openshift-default-598bcf5c59-nnggv\" (UID: \"9a1cb0d5-2afd-42a0-a18f-ac401bfbd9bb\") " pod="openshift-ingress/e2e-shared-b-e94b3e-openshift-default-598bcf5c59-nnggv" Jul 16 00:07:38.615025 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:38.614950 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a1cb0d5-2afd-42a0-a18f-ac401bfbd9bb-workload-certs\") pod \"e2e-shared-b-e94b3e-openshift-default-598bcf5c59-nnggv\" (UID: \"9a1cb0d5-2afd-42a0-a18f-ac401bfbd9bb\") " pod="openshift-ingress/e2e-shared-b-e94b3e-openshift-default-598bcf5c59-nnggv" Jul 16 00:07:38.615025 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:38.614997 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a1cb0d5-2afd-42a0-a18f-ac401bfbd9bb-istio-envoy\") pod \"e2e-shared-b-e94b3e-openshift-default-598bcf5c59-nnggv\" (UID: \"9a1cb0d5-2afd-42a0-a18f-ac401bfbd9bb\") " pod="openshift-ingress/e2e-shared-b-e94b3e-openshift-default-598bcf5c59-nnggv" Jul 16 00:07:38.615217 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:38.615049 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a1cb0d5-2afd-42a0-a18f-ac401bfbd9bb-istiod-ca-cert\") pod \"e2e-shared-b-e94b3e-openshift-default-598bcf5c59-nnggv\" (UID: \"9a1cb0d5-2afd-42a0-a18f-ac401bfbd9bb\") " pod="openshift-ingress/e2e-shared-b-e94b3e-openshift-default-598bcf5c59-nnggv" Jul 16 00:07:38.615217 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:38.615097 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a1cb0d5-2afd-42a0-a18f-ac401bfbd9bb-workload-socket\") pod \"e2e-shared-b-e94b3e-openshift-default-598bcf5c59-nnggv\" (UID: \"9a1cb0d5-2afd-42a0-a18f-ac401bfbd9bb\") " pod="openshift-ingress/e2e-shared-b-e94b3e-openshift-default-598bcf5c59-nnggv" Jul 16 00:07:38.615217 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:38.615100 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a1cb0d5-2afd-42a0-a18f-ac401bfbd9bb-istio-data\") pod \"e2e-shared-b-e94b3e-openshift-default-598bcf5c59-nnggv\" (UID: \"9a1cb0d5-2afd-42a0-a18f-ac401bfbd9bb\") " pod="openshift-ingress/e2e-shared-b-e94b3e-openshift-default-598bcf5c59-nnggv" Jul 16 00:07:38.615217 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:38.615135 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a1cb0d5-2afd-42a0-a18f-ac401bfbd9bb-credential-socket\") pod \"e2e-shared-b-e94b3e-openshift-default-598bcf5c59-nnggv\" (UID: \"9a1cb0d5-2afd-42a0-a18f-ac401bfbd9bb\") " pod="openshift-ingress/e2e-shared-b-e94b3e-openshift-default-598bcf5c59-nnggv" Jul 16 00:07:38.615443 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:38.615413 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a1cb0d5-2afd-42a0-a18f-ac401bfbd9bb-workload-certs\") pod \"e2e-shared-b-e94b3e-openshift-default-598bcf5c59-nnggv\" (UID: \"9a1cb0d5-2afd-42a0-a18f-ac401bfbd9bb\") " pod="openshift-ingress/e2e-shared-b-e94b3e-openshift-default-598bcf5c59-nnggv" Jul 16 00:07:38.615547 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:38.615519 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a1cb0d5-2afd-42a0-a18f-ac401bfbd9bb-workload-socket\") pod \"e2e-shared-b-e94b3e-openshift-default-598bcf5c59-nnggv\" (UID: \"9a1cb0d5-2afd-42a0-a18f-ac401bfbd9bb\") " pod="openshift-ingress/e2e-shared-b-e94b3e-openshift-default-598bcf5c59-nnggv" Jul 16 00:07:38.615635 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:38.615542 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a1cb0d5-2afd-42a0-a18f-ac401bfbd9bb-credential-socket\") pod \"e2e-shared-b-e94b3e-openshift-default-598bcf5c59-nnggv\" (UID: \"9a1cb0d5-2afd-42a0-a18f-ac401bfbd9bb\") " pod="openshift-ingress/e2e-shared-b-e94b3e-openshift-default-598bcf5c59-nnggv" Jul 16 00:07:38.615865 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:38.615841 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a1cb0d5-2afd-42a0-a18f-ac401bfbd9bb-istiod-ca-cert\") pod \"e2e-shared-b-e94b3e-openshift-default-598bcf5c59-nnggv\" (UID: \"9a1cb0d5-2afd-42a0-a18f-ac401bfbd9bb\") " pod="openshift-ingress/e2e-shared-b-e94b3e-openshift-default-598bcf5c59-nnggv" Jul 16 00:07:38.617387 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:38.617354 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a1cb0d5-2afd-42a0-a18f-ac401bfbd9bb-istio-podinfo\") pod \"e2e-shared-b-e94b3e-openshift-default-598bcf5c59-nnggv\" (UID: \"9a1cb0d5-2afd-42a0-a18f-ac401bfbd9bb\") " pod="openshift-ingress/e2e-shared-b-e94b3e-openshift-default-598bcf5c59-nnggv" Jul 16 00:07:38.617540 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:38.617522 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a1cb0d5-2afd-42a0-a18f-ac401bfbd9bb-istio-envoy\") pod \"e2e-shared-b-e94b3e-openshift-default-598bcf5c59-nnggv\" (UID: \"9a1cb0d5-2afd-42a0-a18f-ac401bfbd9bb\") " pod="openshift-ingress/e2e-shared-b-e94b3e-openshift-default-598bcf5c59-nnggv" Jul 16 00:07:38.622336 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:38.622311 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a1cb0d5-2afd-42a0-a18f-ac401bfbd9bb-istio-token\") pod \"e2e-shared-b-e94b3e-openshift-default-598bcf5c59-nnggv\" (UID: \"9a1cb0d5-2afd-42a0-a18f-ac401bfbd9bb\") " pod="openshift-ingress/e2e-shared-b-e94b3e-openshift-default-598bcf5c59-nnggv" Jul 16 00:07:38.622495 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:38.622479 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vs42f\" (UniqueName: \"kubernetes.io/projected/9a1cb0d5-2afd-42a0-a18f-ac401bfbd9bb-kube-api-access-vs42f\") pod \"e2e-shared-b-e94b3e-openshift-default-598bcf5c59-nnggv\" (UID: \"9a1cb0d5-2afd-42a0-a18f-ac401bfbd9bb\") " pod="openshift-ingress/e2e-shared-b-e94b3e-openshift-default-598bcf5c59-nnggv" Jul 16 00:07:38.686596 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:38.686522 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-e94b3e-openshift-default-598bcf5c59-nnggv" Jul 16 00:07:38.868456 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:38.868431 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-e94b3e-openshift-default-598bcf5c59-nnggv"] Jul 16 00:07:38.870265 ip-10-0-128-197 kubenswrapper[2563]: W0716 00:07:38.870231 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a1cb0d5_2afd_42a0_a18f_ac401bfbd9bb.slice/crio-fa0ffb13dd604629926600e652dc98c39242fc82cbe42d44807df5d71848848e WatchSource:0}: Error finding container fa0ffb13dd604629926600e652dc98c39242fc82cbe42d44807df5d71848848e: Status 404 returned error can't find the container with id fa0ffb13dd604629926600e652dc98c39242fc82cbe42d44807df5d71848848e Jul 16 00:07:38.872524 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:38.872484 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 16 00:07:38.872624 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:38.872549 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 16 00:07:38.872624 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:38.872576 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 16 00:07:39.356520 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:39.356485 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-e94b3e-openshift-default-598bcf5c59-nnggv" event={"ID":"9a1cb0d5-2afd-42a0-a18f-ac401bfbd9bb","Type":"ContainerStarted","Data":"598f000f65f19d07eb07a936c9912532d2fe0f17712f61ba65535520c9da6b4a"} Jul 16 00:07:39.356699 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:39.356540 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-e94b3e-openshift-default-598bcf5c59-nnggv" event={"ID":"9a1cb0d5-2afd-42a0-a18f-ac401bfbd9bb","Type":"ContainerStarted","Data":"fa0ffb13dd604629926600e652dc98c39242fc82cbe42d44807df5d71848848e"} Jul 16 00:07:39.377890 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:39.377828 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-e94b3e-openshift-default-598bcf5c59-nnggv" podStartSLOduration=1.377807898 podStartE2EDuration="1.377807898s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 00:07:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 00:07:39.374876937 +0000 UTC m=+1874.748334926" watchObservedRunningTime="2026-07-16 00:07:39.377807898 +0000 UTC m=+1874.751265860" Jul 16 00:07:39.687097 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:39.687007 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-e94b3e-openshift-default-598bcf5c59-nnggv" Jul 16 00:07:39.699062 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:39.699022 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-e94b3e-openshift-default-598bcf5c59-nnggv" Jul 16 00:07:39.875311 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:39.875281 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:07:40.251626 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:40.251588 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:07:40.361671 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:40.361642 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-e94b3e-openshift-default-598bcf5c59-nnggv" Jul 16 00:07:40.362669 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:40.362646 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-e94b3e-openshift-default-598bcf5c59-nnggv" Jul 16 00:07:40.427434 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:40.427243 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:07:40.522182 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:40.522101 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:07:40.813593 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:40.813450 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:07:40.960002 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:40.959949 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:07:41.061987 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:41.061937 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:07:44.384658 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:44.384624 2563 generic.go:356] "Generic (PLEG): container finished" podID="74652b67-44c7-4d41-b07b-c9b83a3952d8" containerID="cf5d38dcaf7b79026a3c0d34e43ea0720c9a41d3e1a613c3d858c1416a9c73ab" exitCode=0 Jul 16 00:07:44.385111 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:44.384670 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-ef3a1a-openshift-default-6b6bf9786c-bwf8h" event={"ID":"74652b67-44c7-4d41-b07b-c9b83a3952d8","Type":"ContainerDied","Data":"cf5d38dcaf7b79026a3c0d34e43ea0720c9a41d3e1a613c3d858c1416a9c73ab"} Jul 16 00:07:44.385157 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:44.385140 2563 scope.go:117] "RemoveContainer" containerID="cf5d38dcaf7b79026a3c0d34e43ea0720c9a41d3e1a613c3d858c1416a9c73ab" Jul 16 00:07:44.386175 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:44.386147 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 16 00:07:44.386252 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:44.386215 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 16 00:07:44.386252 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:44.386248 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 16 00:07:45.394918 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:45.394877 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-ef3a1a-openshift-default-6b6bf9786c-bwf8h" event={"ID":"74652b67-44c7-4d41-b07b-c9b83a3952d8","Type":"ContainerStarted","Data":"0bd041ebb579213b15eda22949bc3cbe396e4560de27b09b90fe060fbf7d74bb"} Jul 16 00:07:45.923424 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:45.923369 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:07:46.194752 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:46.194653 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-ef3a1a-openshift-default-6b6bf9786c-bwf8h" Jul 16 00:07:46.195874 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:46.195845 2563 patch_prober.go:28] interesting pod/e2e-shared-a-ef3a1a-openshift-default-6b6bf9786c-bwf8h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Jul 16 00:07:46.196102 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:46.196078 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-ef3a1a-openshift-default-6b6bf9786c-bwf8h" podUID="74652b67-44c7-4d41-b07b-c9b83a3952d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Jul 16 00:07:47.195065 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:47.194956 2563 patch_prober.go:28] interesting pod/e2e-shared-a-ef3a1a-openshift-default-6b6bf9786c-bwf8h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Jul 16 00:07:47.195065 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:47.195046 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-ef3a1a-openshift-default-6b6bf9786c-bwf8h" podUID="74652b67-44c7-4d41-b07b-c9b83a3952d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Jul 16 00:07:47.319227 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:47.319186 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-ef3a1a-openshift-default-6b6bf9786c-bwf8h" Jul 16 00:07:48.195100 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:48.195058 2563 patch_prober.go:28] interesting pod/e2e-shared-a-ef3a1a-openshift-default-6b6bf9786c-bwf8h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Jul 16 00:07:48.195605 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:48.195129 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-ef3a1a-openshift-default-6b6bf9786c-bwf8h" podUID="74652b67-44c7-4d41-b07b-c9b83a3952d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Jul 16 00:07:49.194731 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:49.194692 2563 patch_prober.go:28] interesting pod/e2e-shared-a-ef3a1a-openshift-default-6b6bf9786c-bwf8h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Jul 16 00:07:49.195060 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:49.195026 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-ef3a1a-openshift-default-6b6bf9786c-bwf8h" podUID="74652b67-44c7-4d41-b07b-c9b83a3952d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Jul 16 00:07:50.198994 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:50.198951 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-ef3a1a-openshift-default-6b6bf9786c-bwf8h" Jul 16 00:07:50.200140 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:50.200112 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-ef3a1a-openshift-default-6b6bf9786c-bwf8h" Jul 16 00:07:53.455918 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:53.455883 2563 generic.go:356] "Generic (PLEG): container finished" podID="9a1cb0d5-2afd-42a0-a18f-ac401bfbd9bb" containerID="598f000f65f19d07eb07a936c9912532d2fe0f17712f61ba65535520c9da6b4a" exitCode=0 Jul 16 00:07:53.456400 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:53.455936 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-e94b3e-openshift-default-598bcf5c59-nnggv" event={"ID":"9a1cb0d5-2afd-42a0-a18f-ac401bfbd9bb","Type":"ContainerDied","Data":"598f000f65f19d07eb07a936c9912532d2fe0f17712f61ba65535520c9da6b4a"} Jul 16 00:07:53.456443 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:53.456425 2563 scope.go:117] "RemoveContainer" containerID="598f000f65f19d07eb07a936c9912532d2fe0f17712f61ba65535520c9da6b4a" Jul 16 00:07:53.457479 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:53.457448 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 16 00:07:53.457553 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:53.457535 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 16 00:07:53.457591 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:53.457579 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 16 00:07:53.833189 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:53.833151 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:07:53.946360 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:53.946329 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:07:54.462960 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:54.462929 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-e94b3e-openshift-default-598bcf5c59-nnggv" event={"ID":"9a1cb0d5-2afd-42a0-a18f-ac401bfbd9bb","Type":"ContainerStarted","Data":"98c8b91e3f8e33851fb32d62b991af5f0327782a78eae2c64d08798d1486cf7a"} Jul 16 00:07:54.686723 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:54.686692 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-e94b3e-openshift-default-598bcf5c59-nnggv" Jul 16 00:07:54.687563 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:54.687539 2563 patch_prober.go:28] interesting pod/e2e-shared-b-e94b3e-openshift-default-598bcf5c59-nnggv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Jul 16 00:07:54.687664 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:54.687585 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-e94b3e-openshift-default-598bcf5c59-nnggv" podUID="9a1cb0d5-2afd-42a0-a18f-ac401bfbd9bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Jul 16 00:07:55.362512 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:55.362468 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-e94b3e-openshift-default-598bcf5c59-nnggv" Jul 16 00:07:55.687075 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:55.686991 2563 patch_prober.go:28] interesting pod/e2e-shared-b-e94b3e-openshift-default-598bcf5c59-nnggv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Jul 16 00:07:55.687075 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:55.687048 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-e94b3e-openshift-default-598bcf5c59-nnggv" podUID="9a1cb0d5-2afd-42a0-a18f-ac401bfbd9bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Jul 16 00:07:56.349193 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:56.349160 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:07:56.687008 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:56.686895 2563 patch_prober.go:28] interesting pod/e2e-shared-b-e94b3e-openshift-default-598bcf5c59-nnggv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Jul 16 00:07:56.687008 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:56.686951 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-e94b3e-openshift-default-598bcf5c59-nnggv" podUID="9a1cb0d5-2afd-42a0-a18f-ac401bfbd9bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Jul 16 00:07:57.686912 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:57.686874 2563 patch_prober.go:28] interesting pod/e2e-shared-b-e94b3e-openshift-default-598bcf5c59-nnggv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Jul 16 00:07:57.687110 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:57.686927 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-e94b3e-openshift-default-598bcf5c59-nnggv" podUID="9a1cb0d5-2afd-42a0-a18f-ac401bfbd9bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Jul 16 00:07:58.690602 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:58.690575 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-e94b3e-openshift-default-598bcf5c59-nnggv" Jul 16 00:07:58.691603 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:58.691583 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-e94b3e-openshift-default-598bcf5c59-nnggv" Jul 16 00:07:58.924152 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:07:58.924122 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:08:01.747448 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:08:01.747411 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:08:01.850954 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:08:01.850922 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:08:34.992918 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:08:34.992887 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-e94b3e/auth-test-model-e94b3e-kserve-fb99fbddc-k9bpj"] Jul 16 00:08:34.995578 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:08:34.995559 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-e94b3e/auth-test-model-e94b3e-kserve-fb99fbddc-k9bpj" Jul 16 00:08:34.998818 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:08:34.998797 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-e94b3e\"/\"kube-root-ca.crt\"" Jul 16 00:08:34.998945 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:08:34.998840 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-e94b3e\"/\"default-dockercfg-ldkbh\"" Jul 16 00:08:34.998945 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:08:34.998841 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-e94b3e\"/\"auth-test-model-e94b3e-kserve-self-signed-certs\"" Jul 16 00:08:34.998945 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:08:34.998818 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-e94b3e\"/\"openshift-service-ca.crt\"" Jul 16 00:08:35.003925 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:08:35.003904 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-e94b3e/auth-test-model-e94b3e-kserve-fb99fbddc-k9bpj"] Jul 16 00:08:35.019693 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:08:35.019642 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:08:35.182665 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:08:35.182636 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/62dd5805-49b3-41eb-a56f-c461ecde3195-dshm\") pod \"auth-test-model-e94b3e-kserve-fb99fbddc-k9bpj\" (UID: \"62dd5805-49b3-41eb-a56f-c461ecde3195\") " pod="ai-tenant-e2e-shared-b-e94b3e/auth-test-model-e94b3e-kserve-fb99fbddc-k9bpj" Jul 16 00:08:35.182829 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:08:35.182682 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/62dd5805-49b3-41eb-a56f-c461ecde3195-home\") pod \"auth-test-model-e94b3e-kserve-fb99fbddc-k9bpj\" (UID: \"62dd5805-49b3-41eb-a56f-c461ecde3195\") " pod="ai-tenant-e2e-shared-b-e94b3e/auth-test-model-e94b3e-kserve-fb99fbddc-k9bpj" Jul 16 00:08:35.182829 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:08:35.182735 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/62dd5805-49b3-41eb-a56f-c461ecde3195-model-cache\") pod \"auth-test-model-e94b3e-kserve-fb99fbddc-k9bpj\" (UID: \"62dd5805-49b3-41eb-a56f-c461ecde3195\") " pod="ai-tenant-e2e-shared-b-e94b3e/auth-test-model-e94b3e-kserve-fb99fbddc-k9bpj" Jul 16 00:08:35.182829 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:08:35.182781 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8ch7\" (UniqueName: \"kubernetes.io/projected/62dd5805-49b3-41eb-a56f-c461ecde3195-kube-api-access-d8ch7\") pod \"auth-test-model-e94b3e-kserve-fb99fbddc-k9bpj\" (UID: \"62dd5805-49b3-41eb-a56f-c461ecde3195\") " pod="ai-tenant-e2e-shared-b-e94b3e/auth-test-model-e94b3e-kserve-fb99fbddc-k9bpj" Jul 16 00:08:35.182829 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:08:35.182817 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62dd5805-49b3-41eb-a56f-c461ecde3195-tls-certs\") pod \"auth-test-model-e94b3e-kserve-fb99fbddc-k9bpj\" (UID: \"62dd5805-49b3-41eb-a56f-c461ecde3195\") " pod="ai-tenant-e2e-shared-b-e94b3e/auth-test-model-e94b3e-kserve-fb99fbddc-k9bpj" Jul 16 00:08:35.182993 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:08:35.182895 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62dd5805-49b3-41eb-a56f-c461ecde3195-tmp-dir\") pod \"auth-test-model-e94b3e-kserve-fb99fbddc-k9bpj\" (UID: \"62dd5805-49b3-41eb-a56f-c461ecde3195\") " pod="ai-tenant-e2e-shared-b-e94b3e/auth-test-model-e94b3e-kserve-fb99fbddc-k9bpj" Jul 16 00:08:35.283824 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:08:35.283744 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8ch7\" (UniqueName: \"kubernetes.io/projected/62dd5805-49b3-41eb-a56f-c461ecde3195-kube-api-access-d8ch7\") pod \"auth-test-model-e94b3e-kserve-fb99fbddc-k9bpj\" (UID: \"62dd5805-49b3-41eb-a56f-c461ecde3195\") " pod="ai-tenant-e2e-shared-b-e94b3e/auth-test-model-e94b3e-kserve-fb99fbddc-k9bpj" Jul 16 00:08:35.283824 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:08:35.283797 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62dd5805-49b3-41eb-a56f-c461ecde3195-tls-certs\") pod \"auth-test-model-e94b3e-kserve-fb99fbddc-k9bpj\" (UID: \"62dd5805-49b3-41eb-a56f-c461ecde3195\") " pod="ai-tenant-e2e-shared-b-e94b3e/auth-test-model-e94b3e-kserve-fb99fbddc-k9bpj" Jul 16 00:08:35.284057 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:08:35.283842 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62dd5805-49b3-41eb-a56f-c461ecde3195-tmp-dir\") pod \"auth-test-model-e94b3e-kserve-fb99fbddc-k9bpj\" (UID: \"62dd5805-49b3-41eb-a56f-c461ecde3195\") " pod="ai-tenant-e2e-shared-b-e94b3e/auth-test-model-e94b3e-kserve-fb99fbddc-k9bpj" Jul 16 00:08:35.284057 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:08:35.283893 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/62dd5805-49b3-41eb-a56f-c461ecde3195-dshm\") pod \"auth-test-model-e94b3e-kserve-fb99fbddc-k9bpj\" (UID: \"62dd5805-49b3-41eb-a56f-c461ecde3195\") " pod="ai-tenant-e2e-shared-b-e94b3e/auth-test-model-e94b3e-kserve-fb99fbddc-k9bpj" Jul 16 00:08:35.284057 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:08:35.283927 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/62dd5805-49b3-41eb-a56f-c461ecde3195-home\") pod \"auth-test-model-e94b3e-kserve-fb99fbddc-k9bpj\" (UID: \"62dd5805-49b3-41eb-a56f-c461ecde3195\") " pod="ai-tenant-e2e-shared-b-e94b3e/auth-test-model-e94b3e-kserve-fb99fbddc-k9bpj" Jul 16 00:08:35.284057 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:08:35.283998 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/62dd5805-49b3-41eb-a56f-c461ecde3195-model-cache\") pod \"auth-test-model-e94b3e-kserve-fb99fbddc-k9bpj\" (UID: \"62dd5805-49b3-41eb-a56f-c461ecde3195\") " pod="ai-tenant-e2e-shared-b-e94b3e/auth-test-model-e94b3e-kserve-fb99fbddc-k9bpj" Jul 16 00:08:35.284303 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:08:35.284280 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62dd5805-49b3-41eb-a56f-c461ecde3195-tmp-dir\") pod \"auth-test-model-e94b3e-kserve-fb99fbddc-k9bpj\" (UID: \"62dd5805-49b3-41eb-a56f-c461ecde3195\") " pod="ai-tenant-e2e-shared-b-e94b3e/auth-test-model-e94b3e-kserve-fb99fbddc-k9bpj" Jul 16 00:08:35.284354 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:08:35.284337 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/62dd5805-49b3-41eb-a56f-c461ecde3195-home\") pod \"auth-test-model-e94b3e-kserve-fb99fbddc-k9bpj\" (UID: \"62dd5805-49b3-41eb-a56f-c461ecde3195\") " pod="ai-tenant-e2e-shared-b-e94b3e/auth-test-model-e94b3e-kserve-fb99fbddc-k9bpj" Jul 16 00:08:35.284406 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:08:35.284387 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/62dd5805-49b3-41eb-a56f-c461ecde3195-model-cache\") pod \"auth-test-model-e94b3e-kserve-fb99fbddc-k9bpj\" (UID: \"62dd5805-49b3-41eb-a56f-c461ecde3195\") " pod="ai-tenant-e2e-shared-b-e94b3e/auth-test-model-e94b3e-kserve-fb99fbddc-k9bpj" Jul 16 00:08:35.286087 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:08:35.286066 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/62dd5805-49b3-41eb-a56f-c461ecde3195-dshm\") pod \"auth-test-model-e94b3e-kserve-fb99fbddc-k9bpj\" (UID: \"62dd5805-49b3-41eb-a56f-c461ecde3195\") " pod="ai-tenant-e2e-shared-b-e94b3e/auth-test-model-e94b3e-kserve-fb99fbddc-k9bpj" Jul 16 00:08:35.286319 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:08:35.286303 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62dd5805-49b3-41eb-a56f-c461ecde3195-tls-certs\") pod \"auth-test-model-e94b3e-kserve-fb99fbddc-k9bpj\" (UID: \"62dd5805-49b3-41eb-a56f-c461ecde3195\") " pod="ai-tenant-e2e-shared-b-e94b3e/auth-test-model-e94b3e-kserve-fb99fbddc-k9bpj" Jul 16 00:08:35.290794 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:08:35.290771 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8ch7\" (UniqueName: \"kubernetes.io/projected/62dd5805-49b3-41eb-a56f-c461ecde3195-kube-api-access-d8ch7\") pod \"auth-test-model-e94b3e-kserve-fb99fbddc-k9bpj\" (UID: \"62dd5805-49b3-41eb-a56f-c461ecde3195\") " pod="ai-tenant-e2e-shared-b-e94b3e/auth-test-model-e94b3e-kserve-fb99fbddc-k9bpj" Jul 16 00:08:35.305791 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:08:35.305768 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-e94b3e/auth-test-model-e94b3e-kserve-fb99fbddc-k9bpj" Jul 16 00:08:35.433911 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:08:35.433887 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-e94b3e/auth-test-model-e94b3e-kserve-fb99fbddc-k9bpj"] Jul 16 00:08:35.435533 ip-10-0-128-197 kubenswrapper[2563]: W0716 00:08:35.435506 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod62dd5805_49b3_41eb_a56f_c461ecde3195.slice/crio-85270056592e295c29445e6b190fa6f7e70aee8e676f85196b6539c83d56fb8f WatchSource:0}: Error finding container 85270056592e295c29445e6b190fa6f7e70aee8e676f85196b6539c83d56fb8f: Status 404 returned error can't find the container with id 85270056592e295c29445e6b190fa6f7e70aee8e676f85196b6539c83d56fb8f Jul 16 00:08:35.669997 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:08:35.669899 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-e94b3e/auth-test-model-e94b3e-kserve-fb99fbddc-k9bpj" event={"ID":"62dd5805-49b3-41eb-a56f-c461ecde3195","Type":"ContainerStarted","Data":"05ae868d49676880123c93994b7fc75d9b66a8b77893801938214c2c7c5de9b8"} Jul 16 00:08:35.669997 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:08:35.669941 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-e94b3e/auth-test-model-e94b3e-kserve-fb99fbddc-k9bpj" event={"ID":"62dd5805-49b3-41eb-a56f-c461ecde3195","Type":"ContainerStarted","Data":"85270056592e295c29445e6b190fa6f7e70aee8e676f85196b6539c83d56fb8f"} Jul 16 00:08:35.686893 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:08:35.686843 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-e94b3e/auth-test-model-e94b3e-kserve-fb99fbddc-k9bpj" podStartSLOduration=1.6868272549999999 podStartE2EDuration="1.686827255s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 00:08:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 00:08:35.685719236 +0000 UTC m=+1931.059177233" watchObservedRunningTime="2026-07-16 00:08:35.686827255 +0000 UTC m=+1931.060285235" Jul 16 00:08:45.306608 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:08:45.306575 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-e94b3e/auth-test-model-e94b3e-kserve-fb99fbddc-k9bpj" Jul 16 00:08:45.306608 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:08:45.306614 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-e94b3e/auth-test-model-e94b3e-kserve-fb99fbddc-k9bpj" Jul 16 00:08:45.319212 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:08:45.319190 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-e94b3e/auth-test-model-e94b3e-kserve-fb99fbddc-k9bpj" Jul 16 00:08:45.732148 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:08:45.732123 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-e94b3e/auth-test-model-e94b3e-kserve-fb99fbddc-k9bpj" Jul 16 00:08:48.374649 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:08:48.374614 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:08:48.474435 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:08:48.474401 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:09:35.363690 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:09:35.363657 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:09:35.464983 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:09:35.464935 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:10:36.055738 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:10:36.055702 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:10:59.519173 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:10:59.519093 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:11:00.471258 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:11:00.471223 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-e94b3e/auth-test-model-e94b3e-kserve-fb99fbddc-k9bpj"] Jul 16 00:11:00.504052 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:11:00.504019 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:11:15.491463 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:11:15.491418 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-e94b3e/auth-test-model-e94b3e-kserve-fb99fbddc-k9bpj" podUID="62dd5805-49b3-41eb-a56f-c461ecde3195" containerName="main" containerID="cri-o://05ae868d49676880123c93994b7fc75d9b66a8b77893801938214c2c7c5de9b8" gracePeriod=45 Jul 16 00:11:15.738095 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:11:15.738071 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-e94b3e/auth-test-model-e94b3e-kserve-fb99fbddc-k9bpj" Jul 16 00:11:15.839242 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:11:15.839162 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/62dd5805-49b3-41eb-a56f-c461ecde3195-dshm\") pod \"62dd5805-49b3-41eb-a56f-c461ecde3195\" (UID: \"62dd5805-49b3-41eb-a56f-c461ecde3195\") " Jul 16 00:11:15.839387 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:11:15.839259 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d8ch7\" (UniqueName: \"kubernetes.io/projected/62dd5805-49b3-41eb-a56f-c461ecde3195-kube-api-access-d8ch7\") pod \"62dd5805-49b3-41eb-a56f-c461ecde3195\" (UID: \"62dd5805-49b3-41eb-a56f-c461ecde3195\") " Jul 16 00:11:15.839387 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:11:15.839287 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/62dd5805-49b3-41eb-a56f-c461ecde3195-home\") pod \"62dd5805-49b3-41eb-a56f-c461ecde3195\" (UID: \"62dd5805-49b3-41eb-a56f-c461ecde3195\") " Jul 16 00:11:15.839387 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:11:15.839302 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62dd5805-49b3-41eb-a56f-c461ecde3195-tmp-dir\") pod \"62dd5805-49b3-41eb-a56f-c461ecde3195\" (UID: \"62dd5805-49b3-41eb-a56f-c461ecde3195\") " Jul 16 00:11:15.839387 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:11:15.839330 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/62dd5805-49b3-41eb-a56f-c461ecde3195-model-cache\") pod \"62dd5805-49b3-41eb-a56f-c461ecde3195\" (UID: \"62dd5805-49b3-41eb-a56f-c461ecde3195\") " Jul 16 00:11:15.839569 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:11:15.839405 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62dd5805-49b3-41eb-a56f-c461ecde3195-tls-certs\") pod \"62dd5805-49b3-41eb-a56f-c461ecde3195\" (UID: \"62dd5805-49b3-41eb-a56f-c461ecde3195\") " Jul 16 00:11:15.839628 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:11:15.839574 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62dd5805-49b3-41eb-a56f-c461ecde3195-model-cache" (OuterVolumeSpecName: "model-cache") pod "62dd5805-49b3-41eb-a56f-c461ecde3195" (UID: "62dd5805-49b3-41eb-a56f-c461ecde3195"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 00:11:15.839628 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:11:15.839590 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62dd5805-49b3-41eb-a56f-c461ecde3195-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "62dd5805-49b3-41eb-a56f-c461ecde3195" (UID: "62dd5805-49b3-41eb-a56f-c461ecde3195"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 00:11:15.839628 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:11:15.839597 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62dd5805-49b3-41eb-a56f-c461ecde3195-home" (OuterVolumeSpecName: "home") pod "62dd5805-49b3-41eb-a56f-c461ecde3195" (UID: "62dd5805-49b3-41eb-a56f-c461ecde3195"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 00:11:15.839852 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:11:15.839829 2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/62dd5805-49b3-41eb-a56f-c461ecde3195-model-cache\") on node \"ip-10-0-128-197.ec2.internal\" DevicePath \"\"" Jul 16 00:11:15.839895 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:11:15.839858 2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/62dd5805-49b3-41eb-a56f-c461ecde3195-home\") on node \"ip-10-0-128-197.ec2.internal\" DevicePath \"\"" Jul 16 00:11:15.839895 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:11:15.839871 2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62dd5805-49b3-41eb-a56f-c461ecde3195-tmp-dir\") on node \"ip-10-0-128-197.ec2.internal\" DevicePath \"\"" Jul 16 00:11:15.841495 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:11:15.841468 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62dd5805-49b3-41eb-a56f-c461ecde3195-dshm" (OuterVolumeSpecName: "dshm") pod "62dd5805-49b3-41eb-a56f-c461ecde3195" (UID: "62dd5805-49b3-41eb-a56f-c461ecde3195"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 00:11:15.841495 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:11:15.841476 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62dd5805-49b3-41eb-a56f-c461ecde3195-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "62dd5805-49b3-41eb-a56f-c461ecde3195" (UID: "62dd5805-49b3-41eb-a56f-c461ecde3195"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 00:11:15.841644 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:11:15.841513 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62dd5805-49b3-41eb-a56f-c461ecde3195-kube-api-access-d8ch7" (OuterVolumeSpecName: "kube-api-access-d8ch7") pod "62dd5805-49b3-41eb-a56f-c461ecde3195" (UID: "62dd5805-49b3-41eb-a56f-c461ecde3195"). InnerVolumeSpecName "kube-api-access-d8ch7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 00:11:15.940898 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:11:15.940870 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d8ch7\" (UniqueName: \"kubernetes.io/projected/62dd5805-49b3-41eb-a56f-c461ecde3195-kube-api-access-d8ch7\") on node \"ip-10-0-128-197.ec2.internal\" DevicePath \"\"" Jul 16 00:11:15.940898 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:11:15.940894 2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62dd5805-49b3-41eb-a56f-c461ecde3195-tls-certs\") on node \"ip-10-0-128-197.ec2.internal\" DevicePath \"\"" Jul 16 00:11:15.941087 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:11:15.940907 2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/62dd5805-49b3-41eb-a56f-c461ecde3195-dshm\") on node \"ip-10-0-128-197.ec2.internal\" DevicePath \"\"" Jul 16 00:11:16.481564 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:11:16.481535 2563 generic.go:356] "Generic (PLEG): container finished" podID="62dd5805-49b3-41eb-a56f-c461ecde3195" containerID="05ae868d49676880123c93994b7fc75d9b66a8b77893801938214c2c7c5de9b8" exitCode=0 Jul 16 00:11:16.481740 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:11:16.481580 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-e94b3e/auth-test-model-e94b3e-kserve-fb99fbddc-k9bpj" Jul 16 00:11:16.481740 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:11:16.481581 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-e94b3e/auth-test-model-e94b3e-kserve-fb99fbddc-k9bpj" event={"ID":"62dd5805-49b3-41eb-a56f-c461ecde3195","Type":"ContainerDied","Data":"05ae868d49676880123c93994b7fc75d9b66a8b77893801938214c2c7c5de9b8"} Jul 16 00:11:16.481740 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:11:16.481703 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-e94b3e/auth-test-model-e94b3e-kserve-fb99fbddc-k9bpj" event={"ID":"62dd5805-49b3-41eb-a56f-c461ecde3195","Type":"ContainerDied","Data":"85270056592e295c29445e6b190fa6f7e70aee8e676f85196b6539c83d56fb8f"} Jul 16 00:11:16.481740 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:11:16.481733 2563 scope.go:117] "RemoveContainer" containerID="05ae868d49676880123c93994b7fc75d9b66a8b77893801938214c2c7c5de9b8" Jul 16 00:11:16.490785 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:11:16.490769 2563 scope.go:117] "RemoveContainer" containerID="05ae868d49676880123c93994b7fc75d9b66a8b77893801938214c2c7c5de9b8" Jul 16 00:11:16.491032 ip-10-0-128-197 kubenswrapper[2563]: E0716 00:11:16.491018 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"05ae868d49676880123c93994b7fc75d9b66a8b77893801938214c2c7c5de9b8\": container with ID starting with 05ae868d49676880123c93994b7fc75d9b66a8b77893801938214c2c7c5de9b8 not found: ID does not exist" containerID="05ae868d49676880123c93994b7fc75d9b66a8b77893801938214c2c7c5de9b8" Jul 16 00:11:16.491106 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:11:16.491039 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05ae868d49676880123c93994b7fc75d9b66a8b77893801938214c2c7c5de9b8"} err="failed to get container status \"05ae868d49676880123c93994b7fc75d9b66a8b77893801938214c2c7c5de9b8\": rpc error: code = NotFound desc = could not find container \"05ae868d49676880123c93994b7fc75d9b66a8b77893801938214c2c7c5de9b8\": container with ID starting with 05ae868d49676880123c93994b7fc75d9b66a8b77893801938214c2c7c5de9b8 not found: ID does not exist" Jul 16 00:11:16.501767 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:11:16.501742 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-e94b3e/auth-test-model-e94b3e-kserve-fb99fbddc-k9bpj"] Jul 16 00:11:16.504013 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:11:16.503992 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-e94b3e/auth-test-model-e94b3e-kserve-fb99fbddc-k9bpj"] Jul 16 00:11:17.169624 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:11:17.169590 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="62dd5805-49b3-41eb-a56f-c461ecde3195" path="/var/lib/kubelet/pods/62dd5805-49b3-41eb-a56f-c461ecde3195/volumes" Jul 16 00:11:20.721899 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:11:20.721854 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-e94b3e/auth-test-model-e94b3e-kserve-fb99fbddc-k9bpj" podUID="62dd5805-49b3-41eb-a56f-c461ecde3195" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded" Jul 16 00:11:23.982673 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:11:23.982638 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:11:25.186509 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:11:25.186479 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d8f6k_2f64cab3-104a-44f6-b125-ba458f54f058/ovn-acl-logging/0.log" Jul 16 00:11:25.190859 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:11:25.190838 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d8f6k_2f64cab3-104a-44f6-b125-ba458f54f058/ovn-acl-logging/0.log" Jul 16 00:11:27.060663 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:11:27.060630 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:11:27.109517 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:11:27.109483 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:11:57.494255 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:11:57.494221 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-e94b3e/sub-test-model-e94b3e-kserve-65587fcb6c-89jmm"] Jul 16 00:11:57.494868 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:11:57.494852 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62dd5805-49b3-41eb-a56f-c461ecde3195" containerName="main" Jul 16 00:11:57.494914 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:11:57.494872 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="62dd5805-49b3-41eb-a56f-c461ecde3195" containerName="main" Jul 16 00:11:57.494997 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:11:57.494985 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="62dd5805-49b3-41eb-a56f-c461ecde3195" containerName="main" Jul 16 00:11:57.497345 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:11:57.497324 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-e94b3e/sub-test-model-e94b3e-kserve-65587fcb6c-89jmm" Jul 16 00:11:57.499663 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:11:57.499637 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-e94b3e\"/\"kube-root-ca.crt\"" Jul 16 00:11:57.499815 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:11:57.499674 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-e94b3e\"/\"openshift-service-ca.crt\"" Jul 16 00:11:57.500619 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:11:57.500559 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-e94b3e\"/\"default-dockercfg-ldkbh\"" Jul 16 00:11:57.500619 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:11:57.500591 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-e94b3e\"/\"sub-test-model-e94b3e-kserve-self-signed-certs\"" Jul 16 00:11:57.504110 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:11:57.504089 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-e94b3e/sub-test-model-e94b3e-kserve-65587fcb6c-89jmm"] Jul 16 00:11:57.517990 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:11:57.517956 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:11:57.536652 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:11:57.536631 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13c7137c-f37b-4614-9ea5-d354274532ec-tls-certs\") pod \"sub-test-model-e94b3e-kserve-65587fcb6c-89jmm\" (UID: \"13c7137c-f37b-4614-9ea5-d354274532ec\") " pod="ai-tenant-e2e-shared-b-e94b3e/sub-test-model-e94b3e-kserve-65587fcb6c-89jmm" Jul 16 00:11:57.536757 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:11:57.536666 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nnn4d\" (UniqueName: \"kubernetes.io/projected/13c7137c-f37b-4614-9ea5-d354274532ec-kube-api-access-nnn4d\") pod \"sub-test-model-e94b3e-kserve-65587fcb6c-89jmm\" (UID: \"13c7137c-f37b-4614-9ea5-d354274532ec\") " pod="ai-tenant-e2e-shared-b-e94b3e/sub-test-model-e94b3e-kserve-65587fcb6c-89jmm" Jul 16 00:11:57.536757 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:11:57.536745 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13c7137c-f37b-4614-9ea5-d354274532ec-model-cache\") pod \"sub-test-model-e94b3e-kserve-65587fcb6c-89jmm\" (UID: \"13c7137c-f37b-4614-9ea5-d354274532ec\") " pod="ai-tenant-e2e-shared-b-e94b3e/sub-test-model-e94b3e-kserve-65587fcb6c-89jmm" Jul 16 00:11:57.536829 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:11:57.536808 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/13c7137c-f37b-4614-9ea5-d354274532ec-dshm\") pod \"sub-test-model-e94b3e-kserve-65587fcb6c-89jmm\" (UID: \"13c7137c-f37b-4614-9ea5-d354274532ec\") " pod="ai-tenant-e2e-shared-b-e94b3e/sub-test-model-e94b3e-kserve-65587fcb6c-89jmm" Jul 16 00:11:57.536867 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:11:57.536826 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/13c7137c-f37b-4614-9ea5-d354274532ec-home\") pod \"sub-test-model-e94b3e-kserve-65587fcb6c-89jmm\" (UID: \"13c7137c-f37b-4614-9ea5-d354274532ec\") " pod="ai-tenant-e2e-shared-b-e94b3e/sub-test-model-e94b3e-kserve-65587fcb6c-89jmm" Jul 16 00:11:57.536867 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:11:57.536861 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13c7137c-f37b-4614-9ea5-d354274532ec-tmp-dir\") pod \"sub-test-model-e94b3e-kserve-65587fcb6c-89jmm\" (UID: \"13c7137c-f37b-4614-9ea5-d354274532ec\") " pod="ai-tenant-e2e-shared-b-e94b3e/sub-test-model-e94b3e-kserve-65587fcb6c-89jmm" Jul 16 00:11:57.637774 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:11:57.637740 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/13c7137c-f37b-4614-9ea5-d354274532ec-home\") pod \"sub-test-model-e94b3e-kserve-65587fcb6c-89jmm\" (UID: \"13c7137c-f37b-4614-9ea5-d354274532ec\") " pod="ai-tenant-e2e-shared-b-e94b3e/sub-test-model-e94b3e-kserve-65587fcb6c-89jmm" Jul 16 00:11:57.637942 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:11:57.637784 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13c7137c-f37b-4614-9ea5-d354274532ec-tmp-dir\") pod \"sub-test-model-e94b3e-kserve-65587fcb6c-89jmm\" (UID: \"13c7137c-f37b-4614-9ea5-d354274532ec\") " pod="ai-tenant-e2e-shared-b-e94b3e/sub-test-model-e94b3e-kserve-65587fcb6c-89jmm" Jul 16 00:11:57.637942 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:11:57.637824 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13c7137c-f37b-4614-9ea5-d354274532ec-tls-certs\") pod \"sub-test-model-e94b3e-kserve-65587fcb6c-89jmm\" (UID: \"13c7137c-f37b-4614-9ea5-d354274532ec\") " pod="ai-tenant-e2e-shared-b-e94b3e/sub-test-model-e94b3e-kserve-65587fcb6c-89jmm" Jul 16 00:11:57.637942 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:11:57.637858 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnn4d\" (UniqueName: \"kubernetes.io/projected/13c7137c-f37b-4614-9ea5-d354274532ec-kube-api-access-nnn4d\") pod \"sub-test-model-e94b3e-kserve-65587fcb6c-89jmm\" (UID: \"13c7137c-f37b-4614-9ea5-d354274532ec\") " pod="ai-tenant-e2e-shared-b-e94b3e/sub-test-model-e94b3e-kserve-65587fcb6c-89jmm" Jul 16 00:11:57.637942 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:11:57.637934 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13c7137c-f37b-4614-9ea5-d354274532ec-model-cache\") pod \"sub-test-model-e94b3e-kserve-65587fcb6c-89jmm\" (UID: \"13c7137c-f37b-4614-9ea5-d354274532ec\") " pod="ai-tenant-e2e-shared-b-e94b3e/sub-test-model-e94b3e-kserve-65587fcb6c-89jmm" Jul 16 00:11:57.638211 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:11:57.638059 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/13c7137c-f37b-4614-9ea5-d354274532ec-dshm\") pod \"sub-test-model-e94b3e-kserve-65587fcb6c-89jmm\" (UID: \"13c7137c-f37b-4614-9ea5-d354274532ec\") " pod="ai-tenant-e2e-shared-b-e94b3e/sub-test-model-e94b3e-kserve-65587fcb6c-89jmm" Jul 16 00:11:57.638211 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:11:57.638184 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/13c7137c-f37b-4614-9ea5-d354274532ec-home\") pod \"sub-test-model-e94b3e-kserve-65587fcb6c-89jmm\" (UID: \"13c7137c-f37b-4614-9ea5-d354274532ec\") " pod="ai-tenant-e2e-shared-b-e94b3e/sub-test-model-e94b3e-kserve-65587fcb6c-89jmm" Jul 16 00:11:57.638342 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:11:57.638235 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13c7137c-f37b-4614-9ea5-d354274532ec-tmp-dir\") pod \"sub-test-model-e94b3e-kserve-65587fcb6c-89jmm\" (UID: \"13c7137c-f37b-4614-9ea5-d354274532ec\") " pod="ai-tenant-e2e-shared-b-e94b3e/sub-test-model-e94b3e-kserve-65587fcb6c-89jmm" Jul 16 00:11:57.638402 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:11:57.638351 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13c7137c-f37b-4614-9ea5-d354274532ec-model-cache\") pod \"sub-test-model-e94b3e-kserve-65587fcb6c-89jmm\" (UID: \"13c7137c-f37b-4614-9ea5-d354274532ec\") " pod="ai-tenant-e2e-shared-b-e94b3e/sub-test-model-e94b3e-kserve-65587fcb6c-89jmm" Jul 16 00:11:57.640277 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:11:57.640250 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/13c7137c-f37b-4614-9ea5-d354274532ec-dshm\") pod \"sub-test-model-e94b3e-kserve-65587fcb6c-89jmm\" (UID: \"13c7137c-f37b-4614-9ea5-d354274532ec\") " pod="ai-tenant-e2e-shared-b-e94b3e/sub-test-model-e94b3e-kserve-65587fcb6c-89jmm" Jul 16 00:11:57.640424 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:11:57.640408 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13c7137c-f37b-4614-9ea5-d354274532ec-tls-certs\") pod \"sub-test-model-e94b3e-kserve-65587fcb6c-89jmm\" (UID: \"13c7137c-f37b-4614-9ea5-d354274532ec\") " pod="ai-tenant-e2e-shared-b-e94b3e/sub-test-model-e94b3e-kserve-65587fcb6c-89jmm" Jul 16 00:11:57.644838 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:11:57.644821 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nnn4d\" (UniqueName: \"kubernetes.io/projected/13c7137c-f37b-4614-9ea5-d354274532ec-kube-api-access-nnn4d\") pod \"sub-test-model-e94b3e-kserve-65587fcb6c-89jmm\" (UID: \"13c7137c-f37b-4614-9ea5-d354274532ec\") " pod="ai-tenant-e2e-shared-b-e94b3e/sub-test-model-e94b3e-kserve-65587fcb6c-89jmm" Jul 16 00:11:57.808445 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:11:57.808359 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-e94b3e/sub-test-model-e94b3e-kserve-65587fcb6c-89jmm" Jul 16 00:11:57.937357 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:11:57.937287 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-e94b3e/sub-test-model-e94b3e-kserve-65587fcb6c-89jmm"] Jul 16 00:11:57.938906 ip-10-0-128-197 kubenswrapper[2563]: W0716 00:11:57.938880 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod13c7137c_f37b_4614_9ea5_d354274532ec.slice/crio-40264d0197f8b1ad06de256e171a74f27c918efdccc6e78bf882a2221fef53ab WatchSource:0}: Error finding container 40264d0197f8b1ad06de256e171a74f27c918efdccc6e78bf882a2221fef53ab: Status 404 returned error can't find the container with id 40264d0197f8b1ad06de256e171a74f27c918efdccc6e78bf882a2221fef53ab Jul 16 00:11:58.707047 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:11:58.707012 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-e94b3e/sub-test-model-e94b3e-kserve-65587fcb6c-89jmm" event={"ID":"13c7137c-f37b-4614-9ea5-d354274532ec","Type":"ContainerStarted","Data":"10341e3c4f6264dcf3a29b28642ba6357188143a6e96fa43bf0ccb1e7fd28d9c"} Jul 16 00:11:58.707047 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:11:58.707052 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-e94b3e/sub-test-model-e94b3e-kserve-65587fcb6c-89jmm" event={"ID":"13c7137c-f37b-4614-9ea5-d354274532ec","Type":"ContainerStarted","Data":"40264d0197f8b1ad06de256e171a74f27c918efdccc6e78bf882a2221fef53ab"} Jul 16 00:11:58.722397 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:11:58.722357 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-e94b3e/sub-test-model-e94b3e-kserve-65587fcb6c-89jmm" podStartSLOduration=1.722340859 podStartE2EDuration="1.722340859s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 00:11:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 00:11:58.721122894 +0000 UTC m=+2134.094580861" watchObservedRunningTime="2026-07-16 00:11:58.722340859 +0000 UTC m=+2134.095798821" Jul 16 00:12:07.808576 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:12:07.808545 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-e94b3e/sub-test-model-e94b3e-kserve-65587fcb6c-89jmm" Jul 16 00:12:07.809036 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:12:07.808606 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-e94b3e/sub-test-model-e94b3e-kserve-65587fcb6c-89jmm" Jul 16 00:12:07.820916 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:12:07.820893 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-e94b3e/sub-test-model-e94b3e-kserve-65587fcb6c-89jmm" Jul 16 00:12:08.771685 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:12:08.771656 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-e94b3e/sub-test-model-e94b3e-kserve-65587fcb6c-89jmm" Jul 16 00:12:39.115518 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:12:39.115480 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:12:39.995941 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:12:39.995915 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-e94b3e/sub-test-model-e94b3e-kserve-65587fcb6c-89jmm"] Jul 16 00:12:40.016728 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:12:40.016699 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:12:54.563285 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:12:54.563245 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:12:54.659562 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:12:54.659535 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:12:55.016307 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:12:55.016270 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-e94b3e/sub-test-model-e94b3e-kserve-65587fcb6c-89jmm" podUID="13c7137c-f37b-4614-9ea5-d354274532ec" containerName="main" containerID="cri-o://10341e3c4f6264dcf3a29b28642ba6357188143a6e96fa43bf0ccb1e7fd28d9c" gracePeriod=45 Jul 16 00:12:55.261140 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:12:55.261116 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-e94b3e/sub-test-model-e94b3e-kserve-65587fcb6c-89jmm" Jul 16 00:12:55.300433 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:12:55.300359 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13c7137c-f37b-4614-9ea5-d354274532ec-model-cache\") pod \"13c7137c-f37b-4614-9ea5-d354274532ec\" (UID: \"13c7137c-f37b-4614-9ea5-d354274532ec\") " Jul 16 00:12:55.300558 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:12:55.300430 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13c7137c-f37b-4614-9ea5-d354274532ec-tmp-dir\") pod \"13c7137c-f37b-4614-9ea5-d354274532ec\" (UID: \"13c7137c-f37b-4614-9ea5-d354274532ec\") " Jul 16 00:12:55.300558 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:12:55.300534 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13c7137c-f37b-4614-9ea5-d354274532ec-tls-certs\") pod \"13c7137c-f37b-4614-9ea5-d354274532ec\" (UID: \"13c7137c-f37b-4614-9ea5-d354274532ec\") " Jul 16 00:12:55.300649 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:12:55.300561 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nnn4d\" (UniqueName: \"kubernetes.io/projected/13c7137c-f37b-4614-9ea5-d354274532ec-kube-api-access-nnn4d\") pod \"13c7137c-f37b-4614-9ea5-d354274532ec\" (UID: \"13c7137c-f37b-4614-9ea5-d354274532ec\") " Jul 16 00:12:55.300649 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:12:55.300592 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/13c7137c-f37b-4614-9ea5-d354274532ec-home\") pod \"13c7137c-f37b-4614-9ea5-d354274532ec\" (UID: \"13c7137c-f37b-4614-9ea5-d354274532ec\") " Jul 16 00:12:55.300649 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:12:55.300632 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13c7137c-f37b-4614-9ea5-d354274532ec-model-cache" (OuterVolumeSpecName: "model-cache") pod "13c7137c-f37b-4614-9ea5-d354274532ec" (UID: "13c7137c-f37b-4614-9ea5-d354274532ec"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 00:12:55.300814 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:12:55.300655 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13c7137c-f37b-4614-9ea5-d354274532ec-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "13c7137c-f37b-4614-9ea5-d354274532ec" (UID: "13c7137c-f37b-4614-9ea5-d354274532ec"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 00:12:55.300923 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:12:55.300897 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13c7137c-f37b-4614-9ea5-d354274532ec-home" (OuterVolumeSpecName: "home") pod "13c7137c-f37b-4614-9ea5-d354274532ec" (UID: "13c7137c-f37b-4614-9ea5-d354274532ec"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 00:12:55.300997 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:12:55.300922 2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13c7137c-f37b-4614-9ea5-d354274532ec-model-cache\") on node \"ip-10-0-128-197.ec2.internal\" DevicePath \"\"" Jul 16 00:12:55.300997 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:12:55.300939 2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13c7137c-f37b-4614-9ea5-d354274532ec-tmp-dir\") on node \"ip-10-0-128-197.ec2.internal\" DevicePath \"\"" Jul 16 00:12:55.302590 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:12:55.302569 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13c7137c-f37b-4614-9ea5-d354274532ec-kube-api-access-nnn4d" (OuterVolumeSpecName: "kube-api-access-nnn4d") pod "13c7137c-f37b-4614-9ea5-d354274532ec" (UID: "13c7137c-f37b-4614-9ea5-d354274532ec"). InnerVolumeSpecName "kube-api-access-nnn4d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 00:12:55.302676 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:12:55.302593 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/13c7137c-f37b-4614-9ea5-d354274532ec-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "13c7137c-f37b-4614-9ea5-d354274532ec" (UID: "13c7137c-f37b-4614-9ea5-d354274532ec"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 00:12:55.402136 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:12:55.402101 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/13c7137c-f37b-4614-9ea5-d354274532ec-dshm\") pod \"13c7137c-f37b-4614-9ea5-d354274532ec\" (UID: \"13c7137c-f37b-4614-9ea5-d354274532ec\") " Jul 16 00:12:55.402362 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:12:55.402347 2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13c7137c-f37b-4614-9ea5-d354274532ec-tls-certs\") on node \"ip-10-0-128-197.ec2.internal\" DevicePath \"\"" Jul 16 00:12:55.402362 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:12:55.402362 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nnn4d\" (UniqueName: \"kubernetes.io/projected/13c7137c-f37b-4614-9ea5-d354274532ec-kube-api-access-nnn4d\") on node \"ip-10-0-128-197.ec2.internal\" DevicePath \"\"" Jul 16 00:12:55.402484 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:12:55.402372 2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/13c7137c-f37b-4614-9ea5-d354274532ec-home\") on node \"ip-10-0-128-197.ec2.internal\" DevicePath \"\"" Jul 16 00:12:55.404042 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:12:55.404021 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13c7137c-f37b-4614-9ea5-d354274532ec-dshm" (OuterVolumeSpecName: "dshm") pod "13c7137c-f37b-4614-9ea5-d354274532ec" (UID: "13c7137c-f37b-4614-9ea5-d354274532ec"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 00:12:55.502889 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:12:55.502859 2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/13c7137c-f37b-4614-9ea5-d354274532ec-dshm\") on node \"ip-10-0-128-197.ec2.internal\" DevicePath \"\"" Jul 16 00:12:56.018046 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:12:56.018008 2563 generic.go:356] "Generic (PLEG): container finished" podID="13c7137c-f37b-4614-9ea5-d354274532ec" containerID="10341e3c4f6264dcf3a29b28642ba6357188143a6e96fa43bf0ccb1e7fd28d9c" exitCode=0 Jul 16 00:12:56.018511 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:12:56.018053 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-e94b3e/sub-test-model-e94b3e-kserve-65587fcb6c-89jmm" event={"ID":"13c7137c-f37b-4614-9ea5-d354274532ec","Type":"ContainerDied","Data":"10341e3c4f6264dcf3a29b28642ba6357188143a6e96fa43bf0ccb1e7fd28d9c"} Jul 16 00:12:56.018511 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:12:56.018084 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-e94b3e/sub-test-model-e94b3e-kserve-65587fcb6c-89jmm" event={"ID":"13c7137c-f37b-4614-9ea5-d354274532ec","Type":"ContainerDied","Data":"40264d0197f8b1ad06de256e171a74f27c918efdccc6e78bf882a2221fef53ab"} Jul 16 00:12:56.018511 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:12:56.018103 2563 scope.go:117] "RemoveContainer" containerID="10341e3c4f6264dcf3a29b28642ba6357188143a6e96fa43bf0ccb1e7fd28d9c" Jul 16 00:12:56.018511 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:12:56.018057 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-e94b3e/sub-test-model-e94b3e-kserve-65587fcb6c-89jmm" Jul 16 00:12:56.027197 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:12:56.027181 2563 scope.go:117] "RemoveContainer" containerID="10341e3c4f6264dcf3a29b28642ba6357188143a6e96fa43bf0ccb1e7fd28d9c" Jul 16 00:12:56.027448 ip-10-0-128-197 kubenswrapper[2563]: E0716 00:12:56.027431 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10341e3c4f6264dcf3a29b28642ba6357188143a6e96fa43bf0ccb1e7fd28d9c\": container with ID starting with 10341e3c4f6264dcf3a29b28642ba6357188143a6e96fa43bf0ccb1e7fd28d9c not found: ID does not exist" containerID="10341e3c4f6264dcf3a29b28642ba6357188143a6e96fa43bf0ccb1e7fd28d9c" Jul 16 00:12:56.027515 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:12:56.027460 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10341e3c4f6264dcf3a29b28642ba6357188143a6e96fa43bf0ccb1e7fd28d9c"} err="failed to get container status \"10341e3c4f6264dcf3a29b28642ba6357188143a6e96fa43bf0ccb1e7fd28d9c\": rpc error: code = NotFound desc = could not find container \"10341e3c4f6264dcf3a29b28642ba6357188143a6e96fa43bf0ccb1e7fd28d9c\": container with ID starting with 10341e3c4f6264dcf3a29b28642ba6357188143a6e96fa43bf0ccb1e7fd28d9c not found: ID does not exist" Jul 16 00:12:56.038871 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:12:56.038839 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-e94b3e/sub-test-model-e94b3e-kserve-65587fcb6c-89jmm"] Jul 16 00:12:56.042085 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:12:56.042065 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-e94b3e/sub-test-model-e94b3e-kserve-65587fcb6c-89jmm"] Jul 16 00:12:57.170076 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:12:57.170043 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="13c7137c-f37b-4614-9ea5-d354274532ec" path="/var/lib/kubelet/pods/13c7137c-f37b-4614-9ea5-d354274532ec/volumes" Jul 16 00:13:02.443017 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:13:02.442982 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:13:02.540871 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:13:02.540837 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:13:08.264728 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:13:08.264699 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:13:08.334489 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:13:08.334457 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:13:34.710656 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:13:34.710624 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-e94b3e/rate-test-model-e94b3e-kserve-d87fb57dc-r2fzd"] Jul 16 00:13:34.711135 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:13:34.711062 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="13c7137c-f37b-4614-9ea5-d354274532ec" containerName="main" Jul 16 00:13:34.711135 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:13:34.711075 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="13c7137c-f37b-4614-9ea5-d354274532ec" containerName="main" Jul 16 00:13:34.711135 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:13:34.711130 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="13c7137c-f37b-4614-9ea5-d354274532ec" containerName="main" Jul 16 00:13:34.714151 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:13:34.714133 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-e94b3e/rate-test-model-e94b3e-kserve-d87fb57dc-r2fzd" Jul 16 00:13:34.717384 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:13:34.717358 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-e94b3e\"/\"openshift-service-ca.crt\"" Jul 16 00:13:34.717977 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:13:34.717953 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-e94b3e\"/\"default-dockercfg-ldkbh\"" Jul 16 00:13:34.721917 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:13:34.721900 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-e94b3e\"/\"kube-root-ca.crt\"" Jul 16 00:13:34.723842 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:13:34.723822 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-e94b3e\"/\"rate-test-model-e94b3e-kserve-self-signed-certs\"" Jul 16 00:13:34.735167 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:13:34.735148 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-e94b3e/rate-test-model-e94b3e-kserve-d87fb57dc-r2fzd"] Jul 16 00:13:34.777731 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:13:34.777705 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:13:34.785035 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:13:34.785013 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7dad212c-899f-4c3a-babd-039fd94b4aad-home\") pod \"rate-test-model-e94b3e-kserve-d87fb57dc-r2fzd\" (UID: \"7dad212c-899f-4c3a-babd-039fd94b4aad\") " pod="ai-tenant-e2e-shared-b-e94b3e/rate-test-model-e94b3e-kserve-d87fb57dc-r2fzd" Jul 16 00:13:34.785181 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:13:34.785052 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7dad212c-899f-4c3a-babd-039fd94b4aad-tls-certs\") pod \"rate-test-model-e94b3e-kserve-d87fb57dc-r2fzd\" (UID: \"7dad212c-899f-4c3a-babd-039fd94b4aad\") " pod="ai-tenant-e2e-shared-b-e94b3e/rate-test-model-e94b3e-kserve-d87fb57dc-r2fzd" Jul 16 00:13:34.785181 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:13:34.785077 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7dad212c-899f-4c3a-babd-039fd94b4aad-model-cache\") pod \"rate-test-model-e94b3e-kserve-d87fb57dc-r2fzd\" (UID: \"7dad212c-899f-4c3a-babd-039fd94b4aad\") " pod="ai-tenant-e2e-shared-b-e94b3e/rate-test-model-e94b3e-kserve-d87fb57dc-r2fzd" Jul 16 00:13:34.785181 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:13:34.785153 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gnsdc\" (UniqueName: \"kubernetes.io/projected/7dad212c-899f-4c3a-babd-039fd94b4aad-kube-api-access-gnsdc\") pod \"rate-test-model-e94b3e-kserve-d87fb57dc-r2fzd\" (UID: \"7dad212c-899f-4c3a-babd-039fd94b4aad\") " pod="ai-tenant-e2e-shared-b-e94b3e/rate-test-model-e94b3e-kserve-d87fb57dc-r2fzd" Jul 16 00:13:34.785346 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:13:34.785256 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7dad212c-899f-4c3a-babd-039fd94b4aad-dshm\") pod \"rate-test-model-e94b3e-kserve-d87fb57dc-r2fzd\" (UID: \"7dad212c-899f-4c3a-babd-039fd94b4aad\") " pod="ai-tenant-e2e-shared-b-e94b3e/rate-test-model-e94b3e-kserve-d87fb57dc-r2fzd" Jul 16 00:13:34.785346 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:13:34.785308 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7dad212c-899f-4c3a-babd-039fd94b4aad-tmp-dir\") pod \"rate-test-model-e94b3e-kserve-d87fb57dc-r2fzd\" (UID: \"7dad212c-899f-4c3a-babd-039fd94b4aad\") " pod="ai-tenant-e2e-shared-b-e94b3e/rate-test-model-e94b3e-kserve-d87fb57dc-r2fzd" Jul 16 00:13:34.886730 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:13:34.886703 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7dad212c-899f-4c3a-babd-039fd94b4aad-home\") pod \"rate-test-model-e94b3e-kserve-d87fb57dc-r2fzd\" (UID: \"7dad212c-899f-4c3a-babd-039fd94b4aad\") " pod="ai-tenant-e2e-shared-b-e94b3e/rate-test-model-e94b3e-kserve-d87fb57dc-r2fzd" Jul 16 00:13:34.886873 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:13:34.886742 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7dad212c-899f-4c3a-babd-039fd94b4aad-tls-certs\") pod \"rate-test-model-e94b3e-kserve-d87fb57dc-r2fzd\" (UID: \"7dad212c-899f-4c3a-babd-039fd94b4aad\") " pod="ai-tenant-e2e-shared-b-e94b3e/rate-test-model-e94b3e-kserve-d87fb57dc-r2fzd" Jul 16 00:13:34.886873 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:13:34.886773 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7dad212c-899f-4c3a-babd-039fd94b4aad-model-cache\") pod \"rate-test-model-e94b3e-kserve-d87fb57dc-r2fzd\" (UID: \"7dad212c-899f-4c3a-babd-039fd94b4aad\") " pod="ai-tenant-e2e-shared-b-e94b3e/rate-test-model-e94b3e-kserve-d87fb57dc-r2fzd" Jul 16 00:13:34.886873 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:13:34.886816 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnsdc\" (UniqueName: \"kubernetes.io/projected/7dad212c-899f-4c3a-babd-039fd94b4aad-kube-api-access-gnsdc\") pod \"rate-test-model-e94b3e-kserve-d87fb57dc-r2fzd\" (UID: \"7dad212c-899f-4c3a-babd-039fd94b4aad\") " pod="ai-tenant-e2e-shared-b-e94b3e/rate-test-model-e94b3e-kserve-d87fb57dc-r2fzd" Jul 16 00:13:34.887088 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:13:34.886900 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7dad212c-899f-4c3a-babd-039fd94b4aad-dshm\") pod \"rate-test-model-e94b3e-kserve-d87fb57dc-r2fzd\" (UID: \"7dad212c-899f-4c3a-babd-039fd94b4aad\") " pod="ai-tenant-e2e-shared-b-e94b3e/rate-test-model-e94b3e-kserve-d87fb57dc-r2fzd" Jul 16 00:13:34.887088 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:13:34.886939 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7dad212c-899f-4c3a-babd-039fd94b4aad-tmp-dir\") pod \"rate-test-model-e94b3e-kserve-d87fb57dc-r2fzd\" (UID: \"7dad212c-899f-4c3a-babd-039fd94b4aad\") " pod="ai-tenant-e2e-shared-b-e94b3e/rate-test-model-e94b3e-kserve-d87fb57dc-r2fzd" Jul 16 00:13:34.887195 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:13:34.887180 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7dad212c-899f-4c3a-babd-039fd94b4aad-home\") pod \"rate-test-model-e94b3e-kserve-d87fb57dc-r2fzd\" (UID: \"7dad212c-899f-4c3a-babd-039fd94b4aad\") " pod="ai-tenant-e2e-shared-b-e94b3e/rate-test-model-e94b3e-kserve-d87fb57dc-r2fzd" Jul 16 00:13:34.887250 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:13:34.887195 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7dad212c-899f-4c3a-babd-039fd94b4aad-model-cache\") pod \"rate-test-model-e94b3e-kserve-d87fb57dc-r2fzd\" (UID: \"7dad212c-899f-4c3a-babd-039fd94b4aad\") " pod="ai-tenant-e2e-shared-b-e94b3e/rate-test-model-e94b3e-kserve-d87fb57dc-r2fzd" Jul 16 00:13:34.887327 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:13:34.887303 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7dad212c-899f-4c3a-babd-039fd94b4aad-tmp-dir\") pod \"rate-test-model-e94b3e-kserve-d87fb57dc-r2fzd\" (UID: \"7dad212c-899f-4c3a-babd-039fd94b4aad\") " pod="ai-tenant-e2e-shared-b-e94b3e/rate-test-model-e94b3e-kserve-d87fb57dc-r2fzd" Jul 16 00:13:34.889161 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:13:34.889136 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7dad212c-899f-4c3a-babd-039fd94b4aad-dshm\") pod \"rate-test-model-e94b3e-kserve-d87fb57dc-r2fzd\" (UID: \"7dad212c-899f-4c3a-babd-039fd94b4aad\") " pod="ai-tenant-e2e-shared-b-e94b3e/rate-test-model-e94b3e-kserve-d87fb57dc-r2fzd" Jul 16 00:13:34.889340 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:13:34.889322 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7dad212c-899f-4c3a-babd-039fd94b4aad-tls-certs\") pod \"rate-test-model-e94b3e-kserve-d87fb57dc-r2fzd\" (UID: \"7dad212c-899f-4c3a-babd-039fd94b4aad\") " pod="ai-tenant-e2e-shared-b-e94b3e/rate-test-model-e94b3e-kserve-d87fb57dc-r2fzd" Jul 16 00:13:34.893774 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:13:34.893754 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnsdc\" (UniqueName: \"kubernetes.io/projected/7dad212c-899f-4c3a-babd-039fd94b4aad-kube-api-access-gnsdc\") pod \"rate-test-model-e94b3e-kserve-d87fb57dc-r2fzd\" (UID: \"7dad212c-899f-4c3a-babd-039fd94b4aad\") " pod="ai-tenant-e2e-shared-b-e94b3e/rate-test-model-e94b3e-kserve-d87fb57dc-r2fzd" Jul 16 00:13:35.023503 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:13:35.023473 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-e94b3e/rate-test-model-e94b3e-kserve-d87fb57dc-r2fzd" Jul 16 00:13:35.158204 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:13:35.158181 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-e94b3e/rate-test-model-e94b3e-kserve-d87fb57dc-r2fzd"] Jul 16 00:13:35.160562 ip-10-0-128-197 kubenswrapper[2563]: W0716 00:13:35.160535 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7dad212c_899f_4c3a_babd_039fd94b4aad.slice/crio-ed77948626003f84cce7de64ec5087434a6bc764cab6635d5cad771273522e2a WatchSource:0}: Error finding container ed77948626003f84cce7de64ec5087434a6bc764cab6635d5cad771273522e2a: Status 404 returned error can't find the container with id ed77948626003f84cce7de64ec5087434a6bc764cab6635d5cad771273522e2a Jul 16 00:13:35.219554 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:13:35.219524 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-e94b3e/rate-test-model-e94b3e-kserve-d87fb57dc-r2fzd" event={"ID":"7dad212c-899f-4c3a-babd-039fd94b4aad","Type":"ContainerStarted","Data":"ed77948626003f84cce7de64ec5087434a6bc764cab6635d5cad771273522e2a"} Jul 16 00:13:36.225629 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:13:36.225596 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-e94b3e/rate-test-model-e94b3e-kserve-d87fb57dc-r2fzd" event={"ID":"7dad212c-899f-4c3a-babd-039fd94b4aad","Type":"ContainerStarted","Data":"de9e8e5d9814688902180509b5e69bcb9f733eb966872f2069bdfec1647fefe6"} Jul 16 00:13:36.242215 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:13:36.242169 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-e94b3e/rate-test-model-e94b3e-kserve-d87fb57dc-r2fzd" podStartSLOduration=2.242153493 podStartE2EDuration="2.242153493s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 00:13:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 00:13:36.240990464 +0000 UTC m=+2231.614448428" watchObservedRunningTime="2026-07-16 00:13:36.242153493 +0000 UTC m=+2231.615611456" Jul 16 00:13:45.023761 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:13:45.023724 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-e94b3e/rate-test-model-e94b3e-kserve-d87fb57dc-r2fzd" Jul 16 00:13:45.024243 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:13:45.023852 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-e94b3e/rate-test-model-e94b3e-kserve-d87fb57dc-r2fzd" Jul 16 00:13:45.036690 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:13:45.036669 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-e94b3e/rate-test-model-e94b3e-kserve-d87fb57dc-r2fzd" Jul 16 00:13:45.285217 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:13:45.285136 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-e94b3e/rate-test-model-e94b3e-kserve-d87fb57dc-r2fzd" Jul 16 00:13:49.112520 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:13:49.112245 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:13:50.360408 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:13:50.360375 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:14:09.683378 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:14:09.683344 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:14:33.229107 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:14:33.229078 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:14:34.100979 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:14:34.100943 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-e94b3e/rate-test-model-e94b3e-kserve-d87fb57dc-r2fzd"] Jul 16 00:14:34.119402 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:14:34.119373 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:14:37.711787 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:14:37.711749 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7742-140237448057392"] Jul 16 00:14:37.715357 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:14:37.715333 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7742-140237448057392" Jul 16 00:14:37.717714 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:14:37.717695 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-6557m\"" Jul 16 00:14:37.721594 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:14:37.721573 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7742-140237448057392"] Jul 16 00:14:37.802066 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:14:37.802038 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pkr85\" (UniqueName: \"kubernetes.io/projected/713d0be0-a7fc-4351-a373-80feb720276b-kube-api-access-pkr85\") pod \"test-curl-7742-140237448057392\" (UID: \"713d0be0-a7fc-4351-a373-80feb720276b\") " pod="opendatahub/test-curl-7742-140237448057392" Jul 16 00:14:37.903569 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:14:37.903538 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pkr85\" (UniqueName: \"kubernetes.io/projected/713d0be0-a7fc-4351-a373-80feb720276b-kube-api-access-pkr85\") pod \"test-curl-7742-140237448057392\" (UID: \"713d0be0-a7fc-4351-a373-80feb720276b\") " pod="opendatahub/test-curl-7742-140237448057392" Jul 16 00:14:37.913000 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:14:37.912956 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pkr85\" (UniqueName: \"kubernetes.io/projected/713d0be0-a7fc-4351-a373-80feb720276b-kube-api-access-pkr85\") pod \"test-curl-7742-140237448057392\" (UID: \"713d0be0-a7fc-4351-a373-80feb720276b\") " pod="opendatahub/test-curl-7742-140237448057392" Jul 16 00:14:38.025819 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:14:38.025790 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7742-140237448057392" Jul 16 00:14:38.359510 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:14:38.359489 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7742-140237448057392"] Jul 16 00:14:38.361689 ip-10-0-128-197 kubenswrapper[2563]: W0716 00:14:38.361660 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod713d0be0_a7fc_4351_a373_80feb720276b.slice/crio-708826496e2580b9764220a9a1c7832f6e017978a6d3808574b07ff7d98cb1d9 WatchSource:0}: Error finding container 708826496e2580b9764220a9a1c7832f6e017978a6d3808574b07ff7d98cb1d9: Status 404 returned error can't find the container with id 708826496e2580b9764220a9a1c7832f6e017978a6d3808574b07ff7d98cb1d9 Jul 16 00:14:38.363156 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:14:38.363139 2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 16 00:14:38.555476 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:14:38.555445 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7742-140237448057392" event={"ID":"713d0be0-a7fc-4351-a373-80feb720276b","Type":"ContainerStarted","Data":"708826496e2580b9764220a9a1c7832f6e017978a6d3808574b07ff7d98cb1d9"} Jul 16 00:14:39.562270 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:14:39.562241 2563 generic.go:356] "Generic (PLEG): container finished" podID="713d0be0-a7fc-4351-a373-80feb720276b" containerID="148d1825f5fa94ab775619525b5746f1a5abb7f738ffad6ee4267441cb791a54" exitCode=0 Jul 16 00:14:39.562735 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:14:39.562307 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7742-140237448057392" event={"ID":"713d0be0-a7fc-4351-a373-80feb720276b","Type":"ContainerDied","Data":"148d1825f5fa94ab775619525b5746f1a5abb7f738ffad6ee4267441cb791a54"} Jul 16 00:14:40.695363 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:14:40.695341 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7742-140237448057392" Jul 16 00:14:40.713600 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:14:40.713577 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7742-140237448057392_713d0be0-a7fc-4351-a373-80feb720276b/test-curl-7742-140237448057392/0.log" Jul 16 00:14:40.732210 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:14:40.732186 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pkr85\" (UniqueName: \"kubernetes.io/projected/713d0be0-a7fc-4351-a373-80feb720276b-kube-api-access-pkr85\") pod \"713d0be0-a7fc-4351-a373-80feb720276b\" (UID: \"713d0be0-a7fc-4351-a373-80feb720276b\") " Jul 16 00:14:40.734417 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:14:40.734398 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7742-140237448057392"] Jul 16 00:14:40.734507 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:14:40.734410 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/713d0be0-a7fc-4351-a373-80feb720276b-kube-api-access-pkr85" (OuterVolumeSpecName: "kube-api-access-pkr85") pod "713d0be0-a7fc-4351-a373-80feb720276b" (UID: "713d0be0-a7fc-4351-a373-80feb720276b"). InnerVolumeSpecName "kube-api-access-pkr85". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 00:14:40.739483 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:14:40.739458 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7742-140237448057392"] Jul 16 00:14:40.834325 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:14:40.834251 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pkr85\" (UniqueName: \"kubernetes.io/projected/713d0be0-a7fc-4351-a373-80feb720276b-kube-api-access-pkr85\") on node \"ip-10-0-128-197.ec2.internal\" DevicePath \"\"" Jul 16 00:14:41.169905 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:14:41.169823 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="713d0be0-a7fc-4351-a373-80feb720276b" path="/var/lib/kubelet/pods/713d0be0-a7fc-4351-a373-80feb720276b/volumes" Jul 16 00:14:41.574095 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:14:41.574071 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7742-140237448057392" Jul 16 00:14:41.574095 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:14:41.574090 2563 scope.go:117] "RemoveContainer" containerID="148d1825f5fa94ab775619525b5746f1a5abb7f738ffad6ee4267441cb791a54" Jul 16 00:14:46.746980 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:14:46.746933 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7742-140237449396560"] Jul 16 00:14:46.747424 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:14:46.747409 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="713d0be0-a7fc-4351-a373-80feb720276b" containerName="test-curl-7742-140237448057392" Jul 16 00:14:46.747424 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:14:46.747425 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="713d0be0-a7fc-4351-a373-80feb720276b" containerName="test-curl-7742-140237448057392" Jul 16 00:14:46.747529 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:14:46.747516 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="713d0be0-a7fc-4351-a373-80feb720276b" containerName="test-curl-7742-140237448057392" Jul 16 00:14:46.752431 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:14:46.752400 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7742-140237449396560" Jul 16 00:14:46.754939 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:14:46.754913 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-6557m\"" Jul 16 00:14:46.755526 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:14:46.755503 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7742-140237449396560"] Jul 16 00:14:46.889191 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:14:46.889156 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qdvq9\" (UniqueName: \"kubernetes.io/projected/ddf61030-1eee-4631-ab68-bfde214a13c6-kube-api-access-qdvq9\") pod \"test-curl-7742-140237449396560\" (UID: \"ddf61030-1eee-4631-ab68-bfde214a13c6\") " pod="opendatahub/test-curl-7742-140237449396560" Jul 16 00:14:46.990203 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:14:46.990171 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdvq9\" (UniqueName: \"kubernetes.io/projected/ddf61030-1eee-4631-ab68-bfde214a13c6-kube-api-access-qdvq9\") pod \"test-curl-7742-140237449396560\" (UID: \"ddf61030-1eee-4631-ab68-bfde214a13c6\") " pod="opendatahub/test-curl-7742-140237449396560" Jul 16 00:14:46.999946 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:14:46.999873 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qdvq9\" (UniqueName: \"kubernetes.io/projected/ddf61030-1eee-4631-ab68-bfde214a13c6-kube-api-access-qdvq9\") pod \"test-curl-7742-140237449396560\" (UID: \"ddf61030-1eee-4631-ab68-bfde214a13c6\") " pod="opendatahub/test-curl-7742-140237449396560" Jul 16 00:14:47.063017 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:14:47.062992 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7742-140237449396560" Jul 16 00:14:47.198062 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:14:47.198039 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7742-140237449396560"] Jul 16 00:14:47.200246 ip-10-0-128-197 kubenswrapper[2563]: W0716 00:14:47.200219 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podddf61030_1eee_4631_ab68_bfde214a13c6.slice/crio-0852eb92a3e0dd5d4d6aa614a98f9e6a57876a8fe776b137497a8dc5eb9fd118 WatchSource:0}: Error finding container 0852eb92a3e0dd5d4d6aa614a98f9e6a57876a8fe776b137497a8dc5eb9fd118: Status 404 returned error can't find the container with id 0852eb92a3e0dd5d4d6aa614a98f9e6a57876a8fe776b137497a8dc5eb9fd118 Jul 16 00:14:47.611402 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:14:47.611369 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7742-140237449396560" event={"ID":"ddf61030-1eee-4631-ab68-bfde214a13c6","Type":"ContainerStarted","Data":"787855e8cf9256fc2b92c4f4f20553d0b7eeb924a25351a81571d6b44440b927"} Jul 16 00:14:47.611543 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:14:47.611418 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7742-140237449396560" event={"ID":"ddf61030-1eee-4631-ab68-bfde214a13c6","Type":"ContainerStarted","Data":"0852eb92a3e0dd5d4d6aa614a98f9e6a57876a8fe776b137497a8dc5eb9fd118"} Jul 16 00:14:47.626744 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:14:47.626702 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7742-140237449396560" podStartSLOduration=1.3157273950000001 podStartE2EDuration="1.626685087s" totalImagesPullingTime="310.957692ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 00:14:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 00:14:47.625049216 +0000 UTC m=+2302.998507198" watchObservedRunningTime="2026-07-16 00:14:47.626685087 +0000 UTC m=+2303.000143049" Jul 16 00:14:47.726413 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:14:47.726383 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7742-140237449396560_ddf61030-1eee-4631-ab68-bfde214a13c6/test-curl-7742-140237449396560/0.log" Jul 16 00:14:48.617768 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:14:48.617732 2563 generic.go:356] "Generic (PLEG): container finished" podID="ddf61030-1eee-4631-ab68-bfde214a13c6" containerID="787855e8cf9256fc2b92c4f4f20553d0b7eeb924a25351a81571d6b44440b927" exitCode=0 Jul 16 00:14:48.618217 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:14:48.617819 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7742-140237449396560" event={"ID":"ddf61030-1eee-4631-ab68-bfde214a13c6","Type":"ContainerDied","Data":"787855e8cf9256fc2b92c4f4f20553d0b7eeb924a25351a81571d6b44440b927"} Jul 16 00:14:49.121581 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:14:49.121544 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-e94b3e/rate-test-model-e94b3e-kserve-d87fb57dc-r2fzd" podUID="7dad212c-899f-4c3a-babd-039fd94b4aad" containerName="main" containerID="cri-o://de9e8e5d9814688902180509b5e69bcb9f733eb966872f2069bdfec1647fefe6" gracePeriod=45 Jul 16 00:14:49.361448 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:14:49.361425 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-e94b3e/rate-test-model-e94b3e-kserve-d87fb57dc-r2fzd" Jul 16 00:14:49.415155 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:14:49.415082 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7dad212c-899f-4c3a-babd-039fd94b4aad-model-cache\") pod \"7dad212c-899f-4c3a-babd-039fd94b4aad\" (UID: \"7dad212c-899f-4c3a-babd-039fd94b4aad\") " Jul 16 00:14:49.415155 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:14:49.415113 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7dad212c-899f-4c3a-babd-039fd94b4aad-dshm\") pod \"7dad212c-899f-4c3a-babd-039fd94b4aad\" (UID: \"7dad212c-899f-4c3a-babd-039fd94b4aad\") " Jul 16 00:14:49.415370 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:14:49.415163 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gnsdc\" (UniqueName: \"kubernetes.io/projected/7dad212c-899f-4c3a-babd-039fd94b4aad-kube-api-access-gnsdc\") pod \"7dad212c-899f-4c3a-babd-039fd94b4aad\" (UID: \"7dad212c-899f-4c3a-babd-039fd94b4aad\") " Jul 16 00:14:49.415370 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:14:49.415186 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7dad212c-899f-4c3a-babd-039fd94b4aad-home\") pod \"7dad212c-899f-4c3a-babd-039fd94b4aad\" (UID: \"7dad212c-899f-4c3a-babd-039fd94b4aad\") " Jul 16 00:14:49.415370 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:14:49.415259 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7dad212c-899f-4c3a-babd-039fd94b4aad-tls-certs\") pod \"7dad212c-899f-4c3a-babd-039fd94b4aad\" (UID: \"7dad212c-899f-4c3a-babd-039fd94b4aad\") " Jul 16 00:14:49.415370 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:14:49.415287 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7dad212c-899f-4c3a-babd-039fd94b4aad-tmp-dir\") pod \"7dad212c-899f-4c3a-babd-039fd94b4aad\" (UID: \"7dad212c-899f-4c3a-babd-039fd94b4aad\") " Jul 16 00:14:49.415370 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:14:49.415344 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7dad212c-899f-4c3a-babd-039fd94b4aad-model-cache" (OuterVolumeSpecName: "model-cache") pod "7dad212c-899f-4c3a-babd-039fd94b4aad" (UID: "7dad212c-899f-4c3a-babd-039fd94b4aad"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 00:14:49.415642 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:14:49.415534 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7dad212c-899f-4c3a-babd-039fd94b4aad-home" (OuterVolumeSpecName: "home") pod "7dad212c-899f-4c3a-babd-039fd94b4aad" (UID: "7dad212c-899f-4c3a-babd-039fd94b4aad"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 00:14:49.415642 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:14:49.415626 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7dad212c-899f-4c3a-babd-039fd94b4aad-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7dad212c-899f-4c3a-babd-039fd94b4aad" (UID: "7dad212c-899f-4c3a-babd-039fd94b4aad"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 00:14:49.415802 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:14:49.415783 2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7dad212c-899f-4c3a-babd-039fd94b4aad-tmp-dir\") on node \"ip-10-0-128-197.ec2.internal\" DevicePath \"\"" Jul 16 00:14:49.415841 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:14:49.415809 2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7dad212c-899f-4c3a-babd-039fd94b4aad-model-cache\") on node \"ip-10-0-128-197.ec2.internal\" DevicePath \"\"" Jul 16 00:14:49.415841 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:14:49.415823 2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7dad212c-899f-4c3a-babd-039fd94b4aad-home\") on node \"ip-10-0-128-197.ec2.internal\" DevicePath \"\"" Jul 16 00:14:49.417273 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:14:49.417245 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7dad212c-899f-4c3a-babd-039fd94b4aad-kube-api-access-gnsdc" (OuterVolumeSpecName: "kube-api-access-gnsdc") pod "7dad212c-899f-4c3a-babd-039fd94b4aad" (UID: "7dad212c-899f-4c3a-babd-039fd94b4aad"). InnerVolumeSpecName "kube-api-access-gnsdc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 00:14:49.417273 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:14:49.417265 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7dad212c-899f-4c3a-babd-039fd94b4aad-dshm" (OuterVolumeSpecName: "dshm") pod "7dad212c-899f-4c3a-babd-039fd94b4aad" (UID: "7dad212c-899f-4c3a-babd-039fd94b4aad"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 00:14:49.417401 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:14:49.417340 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7dad212c-899f-4c3a-babd-039fd94b4aad-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7dad212c-899f-4c3a-babd-039fd94b4aad" (UID: "7dad212c-899f-4c3a-babd-039fd94b4aad"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 00:14:49.517058 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:14:49.517031 2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7dad212c-899f-4c3a-babd-039fd94b4aad-dshm\") on node \"ip-10-0-128-197.ec2.internal\" DevicePath \"\"" Jul 16 00:14:49.517058 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:14:49.517054 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gnsdc\" (UniqueName: \"kubernetes.io/projected/7dad212c-899f-4c3a-babd-039fd94b4aad-kube-api-access-gnsdc\") on node \"ip-10-0-128-197.ec2.internal\" DevicePath \"\"" Jul 16 00:14:49.517222 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:14:49.517065 2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7dad212c-899f-4c3a-babd-039fd94b4aad-tls-certs\") on node \"ip-10-0-128-197.ec2.internal\" DevicePath \"\"" Jul 16 00:14:49.624097 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:14:49.624064 2563 generic.go:356] "Generic (PLEG): container finished" podID="7dad212c-899f-4c3a-babd-039fd94b4aad" containerID="de9e8e5d9814688902180509b5e69bcb9f733eb966872f2069bdfec1647fefe6" exitCode=0 Jul 16 00:14:49.624520 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:14:49.624108 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-e94b3e/rate-test-model-e94b3e-kserve-d87fb57dc-r2fzd" Jul 16 00:14:49.624520 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:14:49.624149 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-e94b3e/rate-test-model-e94b3e-kserve-d87fb57dc-r2fzd" event={"ID":"7dad212c-899f-4c3a-babd-039fd94b4aad","Type":"ContainerDied","Data":"de9e8e5d9814688902180509b5e69bcb9f733eb966872f2069bdfec1647fefe6"} Jul 16 00:14:49.624520 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:14:49.624197 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-e94b3e/rate-test-model-e94b3e-kserve-d87fb57dc-r2fzd" event={"ID":"7dad212c-899f-4c3a-babd-039fd94b4aad","Type":"ContainerDied","Data":"ed77948626003f84cce7de64ec5087434a6bc764cab6635d5cad771273522e2a"} Jul 16 00:14:49.624520 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:14:49.624228 2563 scope.go:117] "RemoveContainer" containerID="de9e8e5d9814688902180509b5e69bcb9f733eb966872f2069bdfec1647fefe6" Jul 16 00:14:49.633955 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:14:49.633936 2563 scope.go:117] "RemoveContainer" containerID="de9e8e5d9814688902180509b5e69bcb9f733eb966872f2069bdfec1647fefe6" Jul 16 00:14:49.634260 ip-10-0-128-197 kubenswrapper[2563]: E0716 00:14:49.634244 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de9e8e5d9814688902180509b5e69bcb9f733eb966872f2069bdfec1647fefe6\": container with ID starting with de9e8e5d9814688902180509b5e69bcb9f733eb966872f2069bdfec1647fefe6 not found: ID does not exist" containerID="de9e8e5d9814688902180509b5e69bcb9f733eb966872f2069bdfec1647fefe6" Jul 16 00:14:49.634341 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:14:49.634268 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de9e8e5d9814688902180509b5e69bcb9f733eb966872f2069bdfec1647fefe6"} err="failed to get container status \"de9e8e5d9814688902180509b5e69bcb9f733eb966872f2069bdfec1647fefe6\": rpc error: code = NotFound desc = could not find container \"de9e8e5d9814688902180509b5e69bcb9f733eb966872f2069bdfec1647fefe6\": container with ID starting with de9e8e5d9814688902180509b5e69bcb9f733eb966872f2069bdfec1647fefe6 not found: ID does not exist" Jul 16 00:14:49.645890 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:14:49.645862 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-e94b3e/rate-test-model-e94b3e-kserve-d87fb57dc-r2fzd"] Jul 16 00:14:49.647740 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:14:49.647722 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-e94b3e/rate-test-model-e94b3e-kserve-d87fb57dc-r2fzd"] Jul 16 00:14:49.747210 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:14:49.747186 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7742-140237449396560" Jul 16 00:14:49.782265 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:14:49.782239 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7742-140237449396560"] Jul 16 00:14:49.785623 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:14:49.785603 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7742-140237449396560"] Jul 16 00:14:49.819925 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:14:49.819904 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qdvq9\" (UniqueName: \"kubernetes.io/projected/ddf61030-1eee-4631-ab68-bfde214a13c6-kube-api-access-qdvq9\") pod \"ddf61030-1eee-4631-ab68-bfde214a13c6\" (UID: \"ddf61030-1eee-4631-ab68-bfde214a13c6\") " Jul 16 00:14:49.821847 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:14:49.821826 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ddf61030-1eee-4631-ab68-bfde214a13c6-kube-api-access-qdvq9" (OuterVolumeSpecName: "kube-api-access-qdvq9") pod "ddf61030-1eee-4631-ab68-bfde214a13c6" (UID: "ddf61030-1eee-4631-ab68-bfde214a13c6"). InnerVolumeSpecName "kube-api-access-qdvq9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 00:14:49.901519 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:14:49.901495 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7742-140237449396560"] Jul 16 00:14:49.901873 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:14:49.901861 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ddf61030-1eee-4631-ab68-bfde214a13c6" containerName="test-curl-7742-140237449396560" Jul 16 00:14:49.901914 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:14:49.901875 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="ddf61030-1eee-4631-ab68-bfde214a13c6" containerName="test-curl-7742-140237449396560" Jul 16 00:14:49.901914 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:14:49.901903 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7dad212c-899f-4c3a-babd-039fd94b4aad" containerName="main" Jul 16 00:14:49.901914 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:14:49.901909 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="7dad212c-899f-4c3a-babd-039fd94b4aad" containerName="main" Jul 16 00:14:49.902039 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:14:49.901979 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="ddf61030-1eee-4631-ab68-bfde214a13c6" containerName="test-curl-7742-140237449396560" Jul 16 00:14:49.902039 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:14:49.901991 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="7dad212c-899f-4c3a-babd-039fd94b4aad" containerName="main" Jul 16 00:14:49.904959 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:14:49.904945 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7742-140237449396560" Jul 16 00:14:49.909684 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:14:49.909664 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7742-140237449396560"] Jul 16 00:14:49.921397 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:14:49.921378 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qdvq9\" (UniqueName: \"kubernetes.io/projected/ddf61030-1eee-4631-ab68-bfde214a13c6-kube-api-access-qdvq9\") on node \"ip-10-0-128-197.ec2.internal\" DevicePath \"\"" Jul 16 00:14:50.022350 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:14:50.022323 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjqxx\" (UniqueName: \"kubernetes.io/projected/c7e6582c-b72d-4bd6-be40-b4cba03496a8-kube-api-access-tjqxx\") pod \"test-curl-7742-140237449396560\" (UID: \"c7e6582c-b72d-4bd6-be40-b4cba03496a8\") " pod="opendatahub/test-curl-7742-140237449396560" Jul 16 00:14:50.123778 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:14:50.123746 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjqxx\" (UniqueName: \"kubernetes.io/projected/c7e6582c-b72d-4bd6-be40-b4cba03496a8-kube-api-access-tjqxx\") pod \"test-curl-7742-140237449396560\" (UID: \"c7e6582c-b72d-4bd6-be40-b4cba03496a8\") " pod="opendatahub/test-curl-7742-140237449396560" Jul 16 00:14:50.131758 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:14:50.131732 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjqxx\" (UniqueName: \"kubernetes.io/projected/c7e6582c-b72d-4bd6-be40-b4cba03496a8-kube-api-access-tjqxx\") pod \"test-curl-7742-140237449396560\" (UID: \"c7e6582c-b72d-4bd6-be40-b4cba03496a8\") " pod="opendatahub/test-curl-7742-140237449396560" Jul 16 00:14:50.214863 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:14:50.214834 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7742-140237449396560" Jul 16 00:14:50.343962 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:14:50.343919 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7742-140237449396560"] Jul 16 00:14:50.345457 ip-10-0-128-197 kubenswrapper[2563]: W0716 00:14:50.345427 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc7e6582c_b72d_4bd6_be40_b4cba03496a8.slice/crio-fabfe3c078c9782c19e4fdeab858b7fe4015336ab569acf1f612c457cf67c84e WatchSource:0}: Error finding container fabfe3c078c9782c19e4fdeab858b7fe4015336ab569acf1f612c457cf67c84e: Status 404 returned error can't find the container with id fabfe3c078c9782c19e4fdeab858b7fe4015336ab569acf1f612c457cf67c84e Jul 16 00:14:50.636309 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:14:50.636223 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7742-140237449396560" Jul 16 00:14:50.636309 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:14:50.636253 2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0852eb92a3e0dd5d4d6aa614a98f9e6a57876a8fe776b137497a8dc5eb9fd118" Jul 16 00:14:50.637778 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:14:50.637748 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7742-140237449396560" event={"ID":"c7e6582c-b72d-4bd6-be40-b4cba03496a8","Type":"ContainerStarted","Data":"fabfe3c078c9782c19e4fdeab858b7fe4015336ab569acf1f612c457cf67c84e"} Jul 16 00:14:50.647605 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:14:50.647579 2563 status_manager.go:1027] "Pod was deleted and then recreated, skipping status update" pod="opendatahub/test-curl-7742-140237449396560" oldPodUID="ddf61030-1eee-4631-ab68-bfde214a13c6" podUID="c7e6582c-b72d-4bd6-be40-b4cba03496a8" Jul 16 00:14:51.169173 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:14:51.169143 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7dad212c-899f-4c3a-babd-039fd94b4aad" path="/var/lib/kubelet/pods/7dad212c-899f-4c3a-babd-039fd94b4aad/volumes" Jul 16 00:14:51.169481 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:14:51.169468 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ddf61030-1eee-4631-ab68-bfde214a13c6" path="/var/lib/kubelet/pods/ddf61030-1eee-4631-ab68-bfde214a13c6/volumes" Jul 16 00:14:51.645174 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:14:51.645143 2563 generic.go:356] "Generic (PLEG): container finished" podID="c7e6582c-b72d-4bd6-be40-b4cba03496a8" containerID="694ffba426ba9cee7939cee180379e2e921af07bada38e62326cb2ffd90e74ad" exitCode=0 Jul 16 00:14:51.645570 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:14:51.645196 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7742-140237449396560" event={"ID":"c7e6582c-b72d-4bd6-be40-b4cba03496a8","Type":"ContainerDied","Data":"694ffba426ba9cee7939cee180379e2e921af07bada38e62326cb2ffd90e74ad"} Jul 16 00:14:52.786846 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:14:52.786824 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7742-140237449396560" Jul 16 00:14:52.820664 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:14:52.820619 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7742-140237449396560_c7e6582c-b72d-4bd6-be40-b4cba03496a8/test-curl-7742-140237449396560/0.log" Jul 16 00:14:52.853360 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:14:52.853333 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tjqxx\" (UniqueName: \"kubernetes.io/projected/c7e6582c-b72d-4bd6-be40-b4cba03496a8-kube-api-access-tjqxx\") pod \"c7e6582c-b72d-4bd6-be40-b4cba03496a8\" (UID: \"c7e6582c-b72d-4bd6-be40-b4cba03496a8\") " Jul 16 00:14:52.855267 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:14:52.855236 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7e6582c-b72d-4bd6-be40-b4cba03496a8-kube-api-access-tjqxx" (OuterVolumeSpecName: "kube-api-access-tjqxx") pod "c7e6582c-b72d-4bd6-be40-b4cba03496a8" (UID: "c7e6582c-b72d-4bd6-be40-b4cba03496a8"). InnerVolumeSpecName "kube-api-access-tjqxx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 00:14:52.868853 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:14:52.868828 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7742-140237449396560"] Jul 16 00:14:52.871924 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:14:52.871906 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7742-140237449396560"] Jul 16 00:14:52.954730 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:14:52.954661 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tjqxx\" (UniqueName: \"kubernetes.io/projected/c7e6582c-b72d-4bd6-be40-b4cba03496a8-kube-api-access-tjqxx\") on node \"ip-10-0-128-197.ec2.internal\" DevicePath \"\"" Jul 16 00:14:53.169049 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:14:53.169020 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c7e6582c-b72d-4bd6-be40-b4cba03496a8" path="/var/lib/kubelet/pods/c7e6582c-b72d-4bd6-be40-b4cba03496a8/volumes" Jul 16 00:14:53.398602 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:14:53.398548 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:14:53.657242 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:14:53.657166 2563 scope.go:117] "RemoveContainer" containerID="694ffba426ba9cee7939cee180379e2e921af07bada38e62326cb2ffd90e74ad" Jul 16 00:14:53.657407 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:14:53.657172 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7742-140237449396560" Jul 16 00:14:59.116649 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:14:59.116619 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:14:59.217264 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:14:59.217222 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:14:59.435438 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:14:59.435360 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-ef3a1a-openshift-default-6b6bf9786c-bwf8h"] Jul 16 00:14:59.435606 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:14:59.435561 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-ef3a1a-openshift-default-6b6bf9786c-bwf8h" podUID="74652b67-44c7-4d41-b07b-c9b83a3952d8" containerName="istio-proxy" containerID="cri-o://0bd041ebb579213b15eda22949bc3cbe396e4560de27b09b90fe060fbf7d74bb" gracePeriod=30 Jul 16 00:14:59.450718 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:14:59.450697 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:14:59.613330 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:14:59.613293 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:15:00.089161 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:00.089110 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:15:00.148046 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:00.148013 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-5a03fa09-29736015-wc8vn"] Jul 16 00:15:00.148432 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:00.148419 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c7e6582c-b72d-4bd6-be40-b4cba03496a8" containerName="test-curl-7742-140237449396560" Jul 16 00:15:00.148432 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:00.148433 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="c7e6582c-b72d-4bd6-be40-b4cba03496a8" containerName="test-curl-7742-140237449396560" Jul 16 00:15:00.148543 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:00.148534 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="c7e6582c-b72d-4bd6-be40-b4cba03496a8" containerName="test-curl-7742-140237449396560" Jul 16 00:15:00.153385 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:00.153360 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-df4149da-29736015-ljkbl"] Jul 16 00:15:00.153523 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:00.153470 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-5a03fa09-29736015-wc8vn" Jul 16 00:15:00.155762 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:00.155741 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vnx5t\"" Jul 16 00:15:00.157375 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:00.157357 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d3fd63-29736015-7ftm8"] Jul 16 00:15:00.157496 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:00.157479 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-df4149da-29736015-ljkbl" Jul 16 00:15:00.160769 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:00.160754 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-a22338-29736015-k7f4f"] Jul 16 00:15:00.160928 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:00.160913 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d3fd63-29736015-7ftm8" Jul 16 00:15:00.164573 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:00.164551 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-eae7a4-29736015-zcf7j"] Jul 16 00:15:00.164688 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:00.164632 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-a22338-29736015-k7f4f" Jul 16 00:15:00.166018 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:00.166000 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-5a03fa09-29736015-wc8vn" Jul 16 00:15:00.175086 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:00.171455 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-5a03fa09-29736015-wc8vn"] Jul 16 00:15:00.175086 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:00.171493 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d3fd63-29736015-7ftm8"] Jul 16 00:15:00.175086 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:00.172236 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-eae7a4-29736015-zcf7j" Jul 16 00:15:00.175464 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:00.175258 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-df4149da-29736015-ljkbl"] Jul 16 00:15:00.178664 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:00.178637 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-eae7a4-29736015-zcf7j"] Jul 16 00:15:00.181110 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:00.181092 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-df4149da-29736015-ljkbl" Jul 16 00:15:00.183496 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:00.183434 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-a22338-29736015-k7f4f"] Jul 16 00:15:00.190483 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:00.189273 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-dd8441-29736015-ktwck"] Jul 16 00:15:00.222608 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:00.222575 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-55cbdb16-29736015-btm49"] Jul 16 00:15:00.223011 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:00.222989 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-dd8441-29736015-ktwck" Jul 16 00:15:00.227930 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:00.227896 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-ef3a1a-29736015-zjjfm"] Jul 16 00:15:00.230592 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:00.230572 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-55cbdb16-29736015-btm49" Jul 16 00:15:00.235591 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:00.235570 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-65d40585-29736015-mkfkx"] Jul 16 00:15:00.235810 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:00.235785 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-ef3a1a-29736015-zjjfm" Jul 16 00:15:00.243587 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:00.243151 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29736015-97t7h"] Jul 16 00:15:00.245120 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:00.245097 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-65d40585-29736015-mkfkx" Jul 16 00:15:00.245831 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:00.245813 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d3fd63-29736015-7ftm8" Jul 16 00:15:00.249670 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:00.249651 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-dd8441-29736015-ktwck"] Jul 16 00:15:00.249841 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:00.249827 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-ef3a1a-29736015-zjjfm"] Jul 16 00:15:00.250152 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:00.250136 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-65d40585-29736015-mkfkx"] Jul 16 00:15:00.250271 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:00.249729 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29736015-97t7h" Jul 16 00:15:00.255607 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:00.255240 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29736015-97t7h"] Jul 16 00:15:00.256312 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:00.256292 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-e94b3e-29736015-265rr"] Jul 16 00:15:00.262365 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:00.260722 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-e94b3e-29736015-265rr" Jul 16 00:15:00.267027 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:00.266437 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-a22338-29736015-k7f4f" Jul 16 00:15:00.268135 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:00.268108 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-e94b3e-29736015-265rr"] Jul 16 00:15:00.279073 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:00.277020 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-55cbdb16-29736015-btm49"] Jul 16 00:15:00.295635 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:00.295470 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-eae7a4-29736015-zcf7j" Jul 16 00:15:00.316532 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:00.316509 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-dd8441-29736015-ktwck" Jul 16 00:15:00.333287 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:00.333119 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-55cbdb16-29736015-btm49" Jul 16 00:15:00.373472 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:00.367140 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-ef3a1a-29736015-zjjfm" Jul 16 00:15:00.384143 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:00.384115 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-65d40585-29736015-mkfkx" Jul 16 00:15:00.386505 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:00.386473 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-5a03fa09-29736015-wc8vn"] Jul 16 00:15:00.402186 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:00.400360 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29736015-97t7h" Jul 16 00:15:00.420306 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:00.420141 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-e94b3e-29736015-265rr" Jul 16 00:15:00.608899 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:00.608860 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d3fd63-29736015-7ftm8"] Jul 16 00:15:00.615297 ip-10-0-128-197 kubenswrapper[2563]: W0716 00:15:00.615239 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac7f3cfe_2835_4253_b11b_2971857d8b1c.slice/crio-17ca5e3f03c17461345f3f919090c1fd926391f465bb3a692bf5eeae4eebb16f WatchSource:0}: Error finding container 17ca5e3f03c17461345f3f919090c1fd926391f465bb3a692bf5eeae4eebb16f: Status 404 returned error can't find the container with id 17ca5e3f03c17461345f3f919090c1fd926391f465bb3a692bf5eeae4eebb16f Jul 16 00:15:00.634828 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:00.634232 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-df4149da-29736015-ljkbl"] Jul 16 00:15:00.699356 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:00.699332 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-dd8441-29736015-ktwck"] Jul 16 00:15:00.700153 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:00.700097 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-5a03fa09-29736015-wc8vn" event={"ID":"afd0e0fa-91e1-4191-8265-c6c319035dba","Type":"ContainerStarted","Data":"f9dbde5029a1249c8df885f4193b6a90c9993cb2115a64514213f7fe924b3786"} Jul 16 00:15:00.701525 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:00.701499 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d3fd63-29736015-7ftm8" event={"ID":"ac7f3cfe-2835-4253-b11b-2971857d8b1c","Type":"ContainerStarted","Data":"17ca5e3f03c17461345f3f919090c1fd926391f465bb3a692bf5eeae4eebb16f"} Jul 16 00:15:00.702183 ip-10-0-128-197 kubenswrapper[2563]: W0716 00:15:00.702153 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc18b0d5b_2e6d_4e4c_9330_f77d2eebe189.slice/crio-83365e24b1f76d5149ef8abfc54493c21ecf685b8638d2803cdf8bd6a5038460 WatchSource:0}: Error finding container 83365e24b1f76d5149ef8abfc54493c21ecf685b8638d2803cdf8bd6a5038460: Status 404 returned error can't find the container with id 83365e24b1f76d5149ef8abfc54493c21ecf685b8638d2803cdf8bd6a5038460 Jul 16 00:15:00.702991 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:00.702913 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-df4149da-29736015-ljkbl" event={"ID":"c5cf1c1b-6e76-4ead-b8db-2f6a300e6bc4","Type":"ContainerStarted","Data":"c55e3a39087bd7b1837e0d74edc7cf4a497b3128c6d466edbe40aec406006de0"} Jul 16 00:15:00.789301 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:00.789264 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-e94b3e-29736015-265rr"] Jul 16 00:15:00.790693 ip-10-0-128-197 kubenswrapper[2563]: W0716 00:15:00.790658 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef5f616c_898d_4d47_b853_336fdcff4d60.slice/crio-236ee4e2e8da8c1fc5d10dea11b12f3f0d4ff0d70c3e63073e52f1c4740a157d WatchSource:0}: Error finding container 236ee4e2e8da8c1fc5d10dea11b12f3f0d4ff0d70c3e63073e52f1c4740a157d: Status 404 returned error can't find the container with id 236ee4e2e8da8c1fc5d10dea11b12f3f0d4ff0d70c3e63073e52f1c4740a157d Jul 16 00:15:00.829746 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:00.829722 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-a22338-29736015-k7f4f"] Jul 16 00:15:00.839131 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:00.839109 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-eae7a4-29736015-zcf7j"] Jul 16 00:15:00.892386 ip-10-0-128-197 kubenswrapper[2563]: W0716 00:15:00.892359 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ce3ce3b_842f_4b98_8263_7cece84002c6.slice/crio-411390218d28c8043a2529bd74379cc6d119381c095d8776315f0da83e10872d WatchSource:0}: Error finding container 411390218d28c8043a2529bd74379cc6d119381c095d8776315f0da83e10872d: Status 404 returned error can't find the container with id 411390218d28c8043a2529bd74379cc6d119381c095d8776315f0da83e10872d Jul 16 00:15:00.929365 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:00.929141 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-65d40585-29736015-mkfkx"] Jul 16 00:15:00.930800 ip-10-0-128-197 kubenswrapper[2563]: W0716 00:15:00.930768 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod890529de_9adc_4f10_9611_43ef685116c0.slice/crio-3db26358c597abd92ad981a4c86cf084c91aa56354ef50607bc2960a302be832 WatchSource:0}: Error finding container 3db26358c597abd92ad981a4c86cf084c91aa56354ef50607bc2960a302be832: Status 404 returned error can't find the container with id 3db26358c597abd92ad981a4c86cf084c91aa56354ef50607bc2960a302be832 Jul 16 00:15:00.932677 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:00.932524 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-55cbdb16-29736015-btm49"] Jul 16 00:15:00.933895 ip-10-0-128-197 kubenswrapper[2563]: W0716 00:15:00.933878 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56b285e9_f9f2_4cfe_af3d_29390185eb3d.slice/crio-5dc8164261141a48bb1d1ec1538043abe0433d9b1635a4616e926b21d9c2c13d WatchSource:0}: Error finding container 5dc8164261141a48bb1d1ec1538043abe0433d9b1635a4616e926b21d9c2c13d: Status 404 returned error can't find the container with id 5dc8164261141a48bb1d1ec1538043abe0433d9b1635a4616e926b21d9c2c13d Jul 16 00:15:00.965588 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:00.965119 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-ef3a1a-29736015-zjjfm"] Jul 16 00:15:00.968090 ip-10-0-128-197 kubenswrapper[2563]: W0716 00:15:00.968066 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb00bc8b_cc11_4d4d_81c7_98f9295e4f20.slice/crio-836512f673bd085b5b1781cbcac213707b642911800817b66d6bf8f4b0bfbabe WatchSource:0}: Error finding container 836512f673bd085b5b1781cbcac213707b642911800817b66d6bf8f4b0bfbabe: Status 404 returned error can't find the container with id 836512f673bd085b5b1781cbcac213707b642911800817b66d6bf8f4b0bfbabe Jul 16 00:15:00.968479 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:00.968458 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29736015-97t7h"] Jul 16 00:15:00.968703 ip-10-0-128-197 kubenswrapper[2563]: W0716 00:15:00.968684 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1e446bd_bc6f_435e_a57b_fb37ad6b2786.slice/crio-bbc6b8ad6ac37855257e35957063f4777d8f00beb033587f4af304293b9e98e8 WatchSource:0}: Error finding container bbc6b8ad6ac37855257e35957063f4777d8f00beb033587f4af304293b9e98e8: Status 404 returned error can't find the container with id bbc6b8ad6ac37855257e35957063f4777d8f00beb033587f4af304293b9e98e8 Jul 16 00:15:01.710850 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:01.710815 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-5a03fa09-29736015-wc8vn" event={"ID":"afd0e0fa-91e1-4191-8265-c6c319035dba","Type":"ContainerStarted","Data":"31f2ac8f172d86bba6f69c717a4fd286fb9a40de7627433d9b8d2e1c89bb24bc"} Jul 16 00:15:01.712367 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:01.712341 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d3fd63-29736015-7ftm8" event={"ID":"ac7f3cfe-2835-4253-b11b-2971857d8b1c","Type":"ContainerStarted","Data":"52e6ff865fe6313d77134838400c5610a98914e69feb93c9cf990e52daffecd5"} Jul 16 00:15:01.713901 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:01.713878 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29736015-97t7h" event={"ID":"f1e446bd-bc6f-435e-a57b-fb37ad6b2786","Type":"ContainerStarted","Data":"d985e394a47e85281216cc50adafe88150bef1dd550fc3df33550c3707c53083"} Jul 16 00:15:01.714030 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:01.713917 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29736015-97t7h" event={"ID":"f1e446bd-bc6f-435e-a57b-fb37ad6b2786","Type":"ContainerStarted","Data":"bbc6b8ad6ac37855257e35957063f4777d8f00beb033587f4af304293b9e98e8"} Jul 16 00:15:01.715379 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:01.715358 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-ef3a1a-29736015-zjjfm" event={"ID":"db00bc8b-cc11-4d4d-81c7-98f9295e4f20","Type":"ContainerStarted","Data":"4e7ac04982a0556f2e14ca7f8ecfa10171080df76667327a5b7b7ec8d600317c"} Jul 16 00:15:01.715469 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:01.715399 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-ef3a1a-29736015-zjjfm" event={"ID":"db00bc8b-cc11-4d4d-81c7-98f9295e4f20","Type":"ContainerStarted","Data":"836512f673bd085b5b1781cbcac213707b642911800817b66d6bf8f4b0bfbabe"} Jul 16 00:15:01.716939 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:01.716912 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-dd8441-29736015-ktwck" event={"ID":"c18b0d5b-2e6d-4e4c-9330-f77d2eebe189","Type":"ContainerStarted","Data":"842008acaa60a09b7355ea26ca45721d645ec1b02e7db1e6a5d74f328db84ac4"} Jul 16 00:15:01.717059 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:01.716953 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-dd8441-29736015-ktwck" event={"ID":"c18b0d5b-2e6d-4e4c-9330-f77d2eebe189","Type":"ContainerStarted","Data":"83365e24b1f76d5149ef8abfc54493c21ecf685b8638d2803cdf8bd6a5038460"} Jul 16 00:15:01.718617 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:01.718524 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-a22338-29736015-k7f4f" event={"ID":"5ce3ce3b-842f-4b98-8263-7cece84002c6","Type":"ContainerStarted","Data":"4d1853264cfe118f68527f300966a484b276361154321d9bc831928aeb39e34e"} Jul 16 00:15:01.718617 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:01.718568 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-a22338-29736015-k7f4f" event={"ID":"5ce3ce3b-842f-4b98-8263-7cece84002c6","Type":"ContainerStarted","Data":"411390218d28c8043a2529bd74379cc6d119381c095d8776315f0da83e10872d"} Jul 16 00:15:01.720088 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:01.720067 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-e94b3e-29736015-265rr" event={"ID":"ef5f616c-898d-4d47-b853-336fdcff4d60","Type":"ContainerStarted","Data":"9b1e142cf4cb68477473c486dd3d3db6336d75e6489cb47f866ad5350b01ccab"} Jul 16 00:15:01.720192 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:01.720103 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-e94b3e-29736015-265rr" event={"ID":"ef5f616c-898d-4d47-b853-336fdcff4d60","Type":"ContainerStarted","Data":"236ee4e2e8da8c1fc5d10dea11b12f3f0d4ff0d70c3e63073e52f1c4740a157d"} Jul 16 00:15:01.721507 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:01.721486 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-55cbdb16-29736015-btm49" event={"ID":"56b285e9-f9f2-4cfe-af3d-29390185eb3d","Type":"ContainerStarted","Data":"5ee84b069783b7f3cdcdbe750fdb995826dc7de1d5b64cfcd3c63c079f832827"} Jul 16 00:15:01.721609 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:01.721522 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-55cbdb16-29736015-btm49" event={"ID":"56b285e9-f9f2-4cfe-af3d-29390185eb3d","Type":"ContainerStarted","Data":"5dc8164261141a48bb1d1ec1538043abe0433d9b1635a4616e926b21d9c2c13d"} Jul 16 00:15:01.722791 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:01.722770 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-eae7a4-29736015-zcf7j" event={"ID":"75696f17-5b1f-4f49-bb12-fd2a041c45e2","Type":"ContainerStarted","Data":"bdfaa03958ed427b37e0e09089182bd984f740ef3a1bec70d284dd34f609bc19"} Jul 16 00:15:01.722879 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:01.722814 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-eae7a4-29736015-zcf7j" event={"ID":"75696f17-5b1f-4f49-bb12-fd2a041c45e2","Type":"ContainerStarted","Data":"bbae5e9eb93182deb314d1b3e806aa158d50d943d38da3db0b98647d7f3fb73d"} Jul 16 00:15:01.724184 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:01.724152 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-df4149da-29736015-ljkbl" event={"ID":"c5cf1c1b-6e76-4ead-b8db-2f6a300e6bc4","Type":"ContainerStarted","Data":"c6bbee67a2daa56e5d99cd292f4c1401a99ec5494b0401ce384988d37fc49d0e"} Jul 16 00:15:01.726073 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:01.726051 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-65d40585-29736015-mkfkx" event={"ID":"890529de-9adc-4f10-9611-43ef685116c0","Type":"ContainerStarted","Data":"fa7426ebb63e3dbd760c82b6f1428e13f6126128dcf11430e8188beb037e07fb"} Jul 16 00:15:01.726175 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:01.726090 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-65d40585-29736015-mkfkx" event={"ID":"890529de-9adc-4f10-9611-43ef685116c0","Type":"ContainerStarted","Data":"3db26358c597abd92ad981a4c86cf084c91aa56354ef50607bc2960a302be832"} Jul 16 00:15:01.726270 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:01.726226 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-5a03fa09-29736015-wc8vn" podStartSLOduration=1.7262085950000001 podStartE2EDuration="1.726208595s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 00:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 00:15:01.724375436 +0000 UTC m=+2317.097833424" watchObservedRunningTime="2026-07-16 00:15:01.726208595 +0000 UTC m=+2317.099666559" Jul 16 00:15:01.738479 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:01.738435 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-df4149da-29736015-ljkbl" podStartSLOduration=1.738423066 podStartE2EDuration="1.738423066s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 00:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 00:15:01.736854978 +0000 UTC m=+2317.110312942" watchObservedRunningTime="2026-07-16 00:15:01.738423066 +0000 UTC m=+2317.111881028" Jul 16 00:15:01.754483 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:01.754369 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d3fd63-29736015-7ftm8" podStartSLOduration=1.754353506 podStartE2EDuration="1.754353506s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 00:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 00:15:01.75433805 +0000 UTC m=+2317.127796013" watchObservedRunningTime="2026-07-16 00:15:01.754353506 +0000 UTC m=+2317.127811470" Jul 16 00:15:01.769156 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:01.769118 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-eae7a4-29736015-zcf7j" podStartSLOduration=1.769105157 podStartE2EDuration="1.769105157s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 00:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 00:15:01.76893551 +0000 UTC m=+2317.142393475" watchObservedRunningTime="2026-07-16 00:15:01.769105157 +0000 UTC m=+2317.142563117" Jul 16 00:15:01.797496 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:01.797444 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-dd8441-29736015-ktwck" podStartSLOduration=1.797426637 podStartE2EDuration="1.797426637s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 00:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 00:15:01.793819194 +0000 UTC m=+2317.167277158" watchObservedRunningTime="2026-07-16 00:15:01.797426637 +0000 UTC m=+2317.170884603" Jul 16 00:15:01.812526 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:01.812481 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-ef3a1a-29736015-zjjfm" podStartSLOduration=1.812466548 podStartE2EDuration="1.812466548s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 00:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 00:15:01.810005428 +0000 UTC m=+2317.183463391" watchObservedRunningTime="2026-07-16 00:15:01.812466548 +0000 UTC m=+2317.185924510" Jul 16 00:15:01.825816 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:01.825763 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-a22338-29736015-k7f4f" podStartSLOduration=1.825745242 podStartE2EDuration="1.825745242s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 00:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 00:15:01.823606948 +0000 UTC m=+2317.197064912" watchObservedRunningTime="2026-07-16 00:15:01.825745242 +0000 UTC m=+2317.199203205" Jul 16 00:15:01.839188 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:01.839149 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29736015-97t7h" podStartSLOduration=1.8391354930000001 podStartE2EDuration="1.839135493s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 00:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 00:15:01.837420572 +0000 UTC m=+2317.210878534" watchObservedRunningTime="2026-07-16 00:15:01.839135493 +0000 UTC m=+2317.212593456" Jul 16 00:15:01.853782 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:01.853737 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-e94b3e-29736015-265rr" podStartSLOduration=1.853720027 podStartE2EDuration="1.853720027s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 00:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 00:15:01.85246143 +0000 UTC m=+2317.225919394" watchObservedRunningTime="2026-07-16 00:15:01.853720027 +0000 UTC m=+2317.227177991" Jul 16 00:15:01.868262 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:01.868221 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-55cbdb16-29736015-btm49" podStartSLOduration=1.868208463 podStartE2EDuration="1.868208463s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 00:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 00:15:01.867107088 +0000 UTC m=+2317.240565052" watchObservedRunningTime="2026-07-16 00:15:01.868208463 +0000 UTC m=+2317.241666425" Jul 16 00:15:01.881756 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:01.881712 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-65d40585-29736015-mkfkx" podStartSLOduration=1.881699215 podStartE2EDuration="1.881699215s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 00:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 00:15:01.880636554 +0000 UTC m=+2317.254094517" watchObservedRunningTime="2026-07-16 00:15:01.881699215 +0000 UTC m=+2317.255157174" Jul 16 00:15:02.741400 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:02.741368 2563 generic.go:356] "Generic (PLEG): container finished" podID="db00bc8b-cc11-4d4d-81c7-98f9295e4f20" containerID="4e7ac04982a0556f2e14ca7f8ecfa10171080df76667327a5b7b7ec8d600317c" exitCode=0 Jul 16 00:15:02.741860 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:02.741452 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-ef3a1a-29736015-zjjfm" event={"ID":"db00bc8b-cc11-4d4d-81c7-98f9295e4f20","Type":"ContainerDied","Data":"4e7ac04982a0556f2e14ca7f8ecfa10171080df76667327a5b7b7ec8d600317c"} Jul 16 00:15:02.743207 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:02.743183 2563 generic.go:356] "Generic (PLEG): container finished" podID="c18b0d5b-2e6d-4e4c-9330-f77d2eebe189" containerID="842008acaa60a09b7355ea26ca45721d645ec1b02e7db1e6a5d74f328db84ac4" exitCode=0 Jul 16 00:15:02.743350 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:02.743248 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-dd8441-29736015-ktwck" event={"ID":"c18b0d5b-2e6d-4e4c-9330-f77d2eebe189","Type":"ContainerDied","Data":"842008acaa60a09b7355ea26ca45721d645ec1b02e7db1e6a5d74f328db84ac4"} Jul 16 00:15:02.744798 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:02.744774 2563 generic.go:356] "Generic (PLEG): container finished" podID="5ce3ce3b-842f-4b98-8263-7cece84002c6" containerID="4d1853264cfe118f68527f300966a484b276361154321d9bc831928aeb39e34e" exitCode=0 Jul 16 00:15:02.744898 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:02.744802 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-a22338-29736015-k7f4f" event={"ID":"5ce3ce3b-842f-4b98-8263-7cece84002c6","Type":"ContainerDied","Data":"4d1853264cfe118f68527f300966a484b276361154321d9bc831928aeb39e34e"} Jul 16 00:15:02.746343 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:02.746324 2563 generic.go:356] "Generic (PLEG): container finished" podID="ef5f616c-898d-4d47-b853-336fdcff4d60" containerID="9b1e142cf4cb68477473c486dd3d3db6336d75e6489cb47f866ad5350b01ccab" exitCode=0 Jul 16 00:15:02.746434 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:02.746398 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-e94b3e-29736015-265rr" event={"ID":"ef5f616c-898d-4d47-b853-336fdcff4d60","Type":"ContainerDied","Data":"9b1e142cf4cb68477473c486dd3d3db6336d75e6489cb47f866ad5350b01ccab"} Jul 16 00:15:02.747721 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:02.747702 2563 generic.go:356] "Generic (PLEG): container finished" podID="56b285e9-f9f2-4cfe-af3d-29390185eb3d" containerID="5ee84b069783b7f3cdcdbe750fdb995826dc7de1d5b64cfcd3c63c079f832827" exitCode=0 Jul 16 00:15:02.747813 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:02.747730 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-55cbdb16-29736015-btm49" event={"ID":"56b285e9-f9f2-4cfe-af3d-29390185eb3d","Type":"ContainerDied","Data":"5ee84b069783b7f3cdcdbe750fdb995826dc7de1d5b64cfcd3c63c079f832827"} Jul 16 00:15:02.749195 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:02.749175 2563 generic.go:356] "Generic (PLEG): container finished" podID="75696f17-5b1f-4f49-bb12-fd2a041c45e2" containerID="bdfaa03958ed427b37e0e09089182bd984f740ef3a1bec70d284dd34f609bc19" exitCode=0 Jul 16 00:15:02.749274 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:02.749253 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-eae7a4-29736015-zcf7j" event={"ID":"75696f17-5b1f-4f49-bb12-fd2a041c45e2","Type":"ContainerDied","Data":"bdfaa03958ed427b37e0e09089182bd984f740ef3a1bec70d284dd34f609bc19"} Jul 16 00:15:02.750608 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:02.750590 2563 generic.go:356] "Generic (PLEG): container finished" podID="c5cf1c1b-6e76-4ead-b8db-2f6a300e6bc4" containerID="c6bbee67a2daa56e5d99cd292f4c1401a99ec5494b0401ce384988d37fc49d0e" exitCode=0 Jul 16 00:15:02.750680 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:02.750645 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-df4149da-29736015-ljkbl" event={"ID":"c5cf1c1b-6e76-4ead-b8db-2f6a300e6bc4","Type":"ContainerDied","Data":"c6bbee67a2daa56e5d99cd292f4c1401a99ec5494b0401ce384988d37fc49d0e"} Jul 16 00:15:02.751887 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:02.751867 2563 generic.go:356] "Generic (PLEG): container finished" podID="890529de-9adc-4f10-9611-43ef685116c0" containerID="fa7426ebb63e3dbd760c82b6f1428e13f6126128dcf11430e8188beb037e07fb" exitCode=0 Jul 16 00:15:02.751989 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:02.751928 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-65d40585-29736015-mkfkx" event={"ID":"890529de-9adc-4f10-9611-43ef685116c0","Type":"ContainerDied","Data":"fa7426ebb63e3dbd760c82b6f1428e13f6126128dcf11430e8188beb037e07fb"} Jul 16 00:15:02.753180 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:02.753161 2563 generic.go:356] "Generic (PLEG): container finished" podID="afd0e0fa-91e1-4191-8265-c6c319035dba" containerID="31f2ac8f172d86bba6f69c717a4fd286fb9a40de7627433d9b8d2e1c89bb24bc" exitCode=0 Jul 16 00:15:02.753271 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:02.753239 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-5a03fa09-29736015-wc8vn" event={"ID":"afd0e0fa-91e1-4191-8265-c6c319035dba","Type":"ContainerDied","Data":"31f2ac8f172d86bba6f69c717a4fd286fb9a40de7627433d9b8d2e1c89bb24bc"} Jul 16 00:15:02.754499 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:02.754477 2563 generic.go:356] "Generic (PLEG): container finished" podID="ac7f3cfe-2835-4253-b11b-2971857d8b1c" containerID="52e6ff865fe6313d77134838400c5610a98914e69feb93c9cf990e52daffecd5" exitCode=0 Jul 16 00:15:02.754583 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:02.754504 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d3fd63-29736015-7ftm8" event={"ID":"ac7f3cfe-2835-4253-b11b-2971857d8b1c","Type":"ContainerDied","Data":"52e6ff865fe6313d77134838400c5610a98914e69feb93c9cf990e52daffecd5"} Jul 16 00:15:02.756299 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:02.756278 2563 generic.go:356] "Generic (PLEG): container finished" podID="f1e446bd-bc6f-435e-a57b-fb37ad6b2786" containerID="d985e394a47e85281216cc50adafe88150bef1dd550fc3df33550c3707c53083" exitCode=0 Jul 16 00:15:02.756377 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:02.756311 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29736015-97t7h" event={"ID":"f1e446bd-bc6f-435e-a57b-fb37ad6b2786","Type":"ContainerDied","Data":"d985e394a47e85281216cc50adafe88150bef1dd550fc3df33550c3707c53083"} Jul 16 00:15:03.993592 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:03.993538 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-65d40585-29736015-mkfkx" Jul 16 00:15:04.049927 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:04.049888 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29736015-97t7h" Jul 16 00:15:04.062180 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:04.058389 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-ef3a1a-29736015-zjjfm" Jul 16 00:15:04.062180 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:04.061084 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-55cbdb16-29736015-btm49" Jul 16 00:15:04.067662 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:04.067585 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-a22338-29736015-k7f4f" Jul 16 00:15:04.114550 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:04.114522 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-eae7a4-29736015-zcf7j" Jul 16 00:15:04.119694 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:04.119673 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-df4149da-29736015-ljkbl" Jul 16 00:15:04.128868 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:04.128844 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-e94b3e-29736015-265rr" Jul 16 00:15:04.132975 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:04.132943 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-5a03fa09-29736015-wc8vn" Jul 16 00:15:04.137448 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:04.137421 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d3fd63-29736015-7ftm8" Jul 16 00:15:04.146370 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:04.146347 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-dd8441-29736015-ktwck" Jul 16 00:15:04.669141 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:04.669120 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-ef3a1a-openshift-default-6b6bf9786c-bwf8h" Jul 16 00:15:04.686155 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:04.686131 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74652b67-44c7-4d41-b07b-c9b83a3952d8-istio-token\") pod \"74652b67-44c7-4d41-b07b-c9b83a3952d8\" (UID: \"74652b67-44c7-4d41-b07b-c9b83a3952d8\") " Jul 16 00:15:04.686239 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:04.686182 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74652b67-44c7-4d41-b07b-c9b83a3952d8-workload-socket\") pod \"74652b67-44c7-4d41-b07b-c9b83a3952d8\" (UID: \"74652b67-44c7-4d41-b07b-c9b83a3952d8\") " Jul 16 00:15:04.686239 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:04.686207 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74652b67-44c7-4d41-b07b-c9b83a3952d8-istiod-ca-cert\") pod \"74652b67-44c7-4d41-b07b-c9b83a3952d8\" (UID: \"74652b67-44c7-4d41-b07b-c9b83a3952d8\") " Jul 16 00:15:04.686239 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:04.686232 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74652b67-44c7-4d41-b07b-c9b83a3952d8-istio-envoy\") pod \"74652b67-44c7-4d41-b07b-c9b83a3952d8\" (UID: \"74652b67-44c7-4d41-b07b-c9b83a3952d8\") " Jul 16 00:15:04.686462 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:04.686254 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74652b67-44c7-4d41-b07b-c9b83a3952d8-istio-data\") pod \"74652b67-44c7-4d41-b07b-c9b83a3952d8\" (UID: \"74652b67-44c7-4d41-b07b-c9b83a3952d8\") " Jul 16 00:15:04.686462 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:04.686307 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74652b67-44c7-4d41-b07b-c9b83a3952d8-workload-certs\") pod \"74652b67-44c7-4d41-b07b-c9b83a3952d8\" (UID: \"74652b67-44c7-4d41-b07b-c9b83a3952d8\") " Jul 16 00:15:04.686462 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:04.686401 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74652b67-44c7-4d41-b07b-c9b83a3952d8-credential-socket\") pod \"74652b67-44c7-4d41-b07b-c9b83a3952d8\" (UID: \"74652b67-44c7-4d41-b07b-c9b83a3952d8\") " Jul 16 00:15:04.686736 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:04.686471 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74652b67-44c7-4d41-b07b-c9b83a3952d8-istio-podinfo\") pod \"74652b67-44c7-4d41-b07b-c9b83a3952d8\" (UID: \"74652b67-44c7-4d41-b07b-c9b83a3952d8\") " Jul 16 00:15:04.686736 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:04.686494 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74652b67-44c7-4d41-b07b-c9b83a3952d8-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "74652b67-44c7-4d41-b07b-c9b83a3952d8" (UID: "74652b67-44c7-4d41-b07b-c9b83a3952d8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 00:15:04.686736 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:04.686519 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c5bc4\" (UniqueName: \"kubernetes.io/projected/74652b67-44c7-4d41-b07b-c9b83a3952d8-kube-api-access-c5bc4\") pod \"74652b67-44c7-4d41-b07b-c9b83a3952d8\" (UID: \"74652b67-44c7-4d41-b07b-c9b83a3952d8\") " Jul 16 00:15:04.686736 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:04.686641 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74652b67-44c7-4d41-b07b-c9b83a3952d8-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "74652b67-44c7-4d41-b07b-c9b83a3952d8" (UID: "74652b67-44c7-4d41-b07b-c9b83a3952d8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 00:15:04.686736 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:04.686643 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74652b67-44c7-4d41-b07b-c9b83a3952d8-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "74652b67-44c7-4d41-b07b-c9b83a3952d8" (UID: "74652b67-44c7-4d41-b07b-c9b83a3952d8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 00:15:04.687128 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:04.686766 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74652b67-44c7-4d41-b07b-c9b83a3952d8-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "74652b67-44c7-4d41-b07b-c9b83a3952d8" (UID: "74652b67-44c7-4d41-b07b-c9b83a3952d8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 00:15:04.687128 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:04.686943 2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74652b67-44c7-4d41-b07b-c9b83a3952d8-workload-socket\") on node \"ip-10-0-128-197.ec2.internal\" DevicePath \"\"" Jul 16 00:15:04.687128 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:04.686962 2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74652b67-44c7-4d41-b07b-c9b83a3952d8-istiod-ca-cert\") on node \"ip-10-0-128-197.ec2.internal\" DevicePath \"\"" Jul 16 00:15:04.687128 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:04.687001 2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74652b67-44c7-4d41-b07b-c9b83a3952d8-workload-certs\") on node \"ip-10-0-128-197.ec2.internal\" DevicePath \"\"" Jul 16 00:15:04.687128 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:04.687015 2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74652b67-44c7-4d41-b07b-c9b83a3952d8-credential-socket\") on node \"ip-10-0-128-197.ec2.internal\" DevicePath \"\"" Jul 16 00:15:04.687538 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:04.687504 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74652b67-44c7-4d41-b07b-c9b83a3952d8-istio-data" (OuterVolumeSpecName: "istio-data") pod "74652b67-44c7-4d41-b07b-c9b83a3952d8" (UID: "74652b67-44c7-4d41-b07b-c9b83a3952d8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 00:15:04.689615 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:04.689578 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74652b67-44c7-4d41-b07b-c9b83a3952d8-kube-api-access-c5bc4" (OuterVolumeSpecName: "kube-api-access-c5bc4") pod "74652b67-44c7-4d41-b07b-c9b83a3952d8" (UID: "74652b67-44c7-4d41-b07b-c9b83a3952d8"). InnerVolumeSpecName "kube-api-access-c5bc4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 00:15:04.689615 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:04.689592 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74652b67-44c7-4d41-b07b-c9b83a3952d8-istio-token" (OuterVolumeSpecName: "istio-token") pod "74652b67-44c7-4d41-b07b-c9b83a3952d8" (UID: "74652b67-44c7-4d41-b07b-c9b83a3952d8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 00:15:04.690159 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:04.690140 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74652b67-44c7-4d41-b07b-c9b83a3952d8-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "74652b67-44c7-4d41-b07b-c9b83a3952d8" (UID: "74652b67-44c7-4d41-b07b-c9b83a3952d8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 00:15:04.690917 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:04.690881 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/74652b67-44c7-4d41-b07b-c9b83a3952d8-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "74652b67-44c7-4d41-b07b-c9b83a3952d8" (UID: "74652b67-44c7-4d41-b07b-c9b83a3952d8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 16 00:15:04.769243 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:04.769217 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-df4149da-29736015-ljkbl" Jul 16 00:15:04.769431 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:04.769217 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-df4149da-29736015-ljkbl" event={"ID":"c5cf1c1b-6e76-4ead-b8db-2f6a300e6bc4","Type":"ContainerDied","Data":"c55e3a39087bd7b1837e0d74edc7cf4a497b3128c6d466edbe40aec406006de0"} Jul 16 00:15:04.769431 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:04.769360 2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c55e3a39087bd7b1837e0d74edc7cf4a497b3128c6d466edbe40aec406006de0" Jul 16 00:15:04.770943 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:04.770911 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-65d40585-29736015-mkfkx" Jul 16 00:15:04.770943 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:04.770926 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-65d40585-29736015-mkfkx" event={"ID":"890529de-9adc-4f10-9611-43ef685116c0","Type":"ContainerDied","Data":"3db26358c597abd92ad981a4c86cf084c91aa56354ef50607bc2960a302be832"} Jul 16 00:15:04.771101 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:04.770995 2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3db26358c597abd92ad981a4c86cf084c91aa56354ef50607bc2960a302be832" Jul 16 00:15:04.781223 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:04.781199 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-5a03fa09-29736015-wc8vn" Jul 16 00:15:04.781364 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:04.781288 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-5a03fa09-29736015-wc8vn" event={"ID":"afd0e0fa-91e1-4191-8265-c6c319035dba","Type":"ContainerDied","Data":"f9dbde5029a1249c8df885f4193b6a90c9993cb2115a64514213f7fe924b3786"} Jul 16 00:15:04.781432 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:04.781364 2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f9dbde5029a1249c8df885f4193b6a90c9993cb2115a64514213f7fe924b3786" Jul 16 00:15:04.782911 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:04.782886 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d3fd63-29736015-7ftm8" event={"ID":"ac7f3cfe-2835-4253-b11b-2971857d8b1c","Type":"ContainerDied","Data":"17ca5e3f03c17461345f3f919090c1fd926391f465bb3a692bf5eeae4eebb16f"} Jul 16 00:15:04.783038 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:04.782929 2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="17ca5e3f03c17461345f3f919090c1fd926391f465bb3a692bf5eeae4eebb16f" Jul 16 00:15:04.783038 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:04.783036 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d3fd63-29736015-7ftm8" Jul 16 00:15:04.785230 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:04.785209 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29736015-97t7h" event={"ID":"f1e446bd-bc6f-435e-a57b-fb37ad6b2786","Type":"ContainerDied","Data":"bbc6b8ad6ac37855257e35957063f4777d8f00beb033587f4af304293b9e98e8"} Jul 16 00:15:04.785326 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:04.785250 2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bbc6b8ad6ac37855257e35957063f4777d8f00beb033587f4af304293b9e98e8" Jul 16 00:15:04.785326 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:04.785320 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29736015-97t7h" Jul 16 00:15:04.787508 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:04.787483 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-ef3a1a-29736015-zjjfm" event={"ID":"db00bc8b-cc11-4d4d-81c7-98f9295e4f20","Type":"ContainerDied","Data":"836512f673bd085b5b1781cbcac213707b642911800817b66d6bf8f4b0bfbabe"} Jul 16 00:15:04.787611 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:04.787527 2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="836512f673bd085b5b1781cbcac213707b642911800817b66d6bf8f4b0bfbabe" Jul 16 00:15:04.787611 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:04.787573 2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74652b67-44c7-4d41-b07b-c9b83a3952d8-istio-envoy\") on node \"ip-10-0-128-197.ec2.internal\" DevicePath \"\"" Jul 16 00:15:04.787611 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:04.787585 2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74652b67-44c7-4d41-b07b-c9b83a3952d8-istio-data\") on node \"ip-10-0-128-197.ec2.internal\" DevicePath \"\"" Jul 16 00:15:04.787611 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:04.787594 2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74652b67-44c7-4d41-b07b-c9b83a3952d8-istio-podinfo\") on node \"ip-10-0-128-197.ec2.internal\" DevicePath \"\"" Jul 16 00:15:04.787611 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:04.787602 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c5bc4\" (UniqueName: \"kubernetes.io/projected/74652b67-44c7-4d41-b07b-c9b83a3952d8-kube-api-access-c5bc4\") on node \"ip-10-0-128-197.ec2.internal\" DevicePath \"\"" Jul 16 00:15:04.787611 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:04.787611 2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74652b67-44c7-4d41-b07b-c9b83a3952d8-istio-token\") on node \"ip-10-0-128-197.ec2.internal\" DevicePath \"\"" Jul 16 00:15:04.787907 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:04.787583 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-ef3a1a-29736015-zjjfm" Jul 16 00:15:04.789671 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:04.789653 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-dd8441-29736015-ktwck" Jul 16 00:15:04.789858 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:04.789656 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-dd8441-29736015-ktwck" event={"ID":"c18b0d5b-2e6d-4e4c-9330-f77d2eebe189","Type":"ContainerDied","Data":"83365e24b1f76d5149ef8abfc54493c21ecf685b8638d2803cdf8bd6a5038460"} Jul 16 00:15:04.789941 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:04.789885 2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="83365e24b1f76d5149ef8abfc54493c21ecf685b8638d2803cdf8bd6a5038460" Jul 16 00:15:04.791354 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:04.791331 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-a22338-29736015-k7f4f" event={"ID":"5ce3ce3b-842f-4b98-8263-7cece84002c6","Type":"ContainerDied","Data":"411390218d28c8043a2529bd74379cc6d119381c095d8776315f0da83e10872d"} Jul 16 00:15:04.791442 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:04.791371 2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="411390218d28c8043a2529bd74379cc6d119381c095d8776315f0da83e10872d" Jul 16 00:15:04.791442 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:04.791429 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-a22338-29736015-k7f4f" Jul 16 00:15:04.793893 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:04.793871 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-e94b3e-29736015-265rr" Jul 16 00:15:04.794032 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:04.793878 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-e94b3e-29736015-265rr" event={"ID":"ef5f616c-898d-4d47-b853-336fdcff4d60","Type":"ContainerDied","Data":"236ee4e2e8da8c1fc5d10dea11b12f3f0d4ff0d70c3e63073e52f1c4740a157d"} Jul 16 00:15:04.794144 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:04.794055 2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="236ee4e2e8da8c1fc5d10dea11b12f3f0d4ff0d70c3e63073e52f1c4740a157d" Jul 16 00:15:04.795823 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:04.795806 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-55cbdb16-29736015-btm49" Jul 16 00:15:04.797108 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:04.797089 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-55cbdb16-29736015-btm49" event={"ID":"56b285e9-f9f2-4cfe-af3d-29390185eb3d","Type":"ContainerDied","Data":"5dc8164261141a48bb1d1ec1538043abe0433d9b1635a4616e926b21d9c2c13d"} Jul 16 00:15:04.797275 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:04.797260 2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5dc8164261141a48bb1d1ec1538043abe0433d9b1635a4616e926b21d9c2c13d" Jul 16 00:15:04.798562 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:04.798541 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-eae7a4-29736015-zcf7j" event={"ID":"75696f17-5b1f-4f49-bb12-fd2a041c45e2","Type":"ContainerDied","Data":"bbae5e9eb93182deb314d1b3e806aa158d50d943d38da3db0b98647d7f3fb73d"} Jul 16 00:15:04.798668 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:04.798579 2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bbae5e9eb93182deb314d1b3e806aa158d50d943d38da3db0b98647d7f3fb73d" Jul 16 00:15:04.798668 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:04.798637 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-eae7a4-29736015-zcf7j" Jul 16 00:15:04.803192 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:04.803144 2563 generic.go:356] "Generic (PLEG): container finished" podID="74652b67-44c7-4d41-b07b-c9b83a3952d8" containerID="0bd041ebb579213b15eda22949bc3cbe396e4560de27b09b90fe060fbf7d74bb" exitCode=0 Jul 16 00:15:04.803192 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:04.803175 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-ef3a1a-openshift-default-6b6bf9786c-bwf8h" event={"ID":"74652b67-44c7-4d41-b07b-c9b83a3952d8","Type":"ContainerDied","Data":"0bd041ebb579213b15eda22949bc3cbe396e4560de27b09b90fe060fbf7d74bb"} Jul 16 00:15:04.803384 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:04.803209 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-ef3a1a-openshift-default-6b6bf9786c-bwf8h" event={"ID":"74652b67-44c7-4d41-b07b-c9b83a3952d8","Type":"ContainerDied","Data":"6ba3a9f962c9349ab39ff619557ac37989b53d09b29969d22ff8b16235a89fa8"} Jul 16 00:15:04.803384 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:04.803238 2563 scope.go:117] "RemoveContainer" containerID="0bd041ebb579213b15eda22949bc3cbe396e4560de27b09b90fe060fbf7d74bb" Jul 16 00:15:04.803488 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:04.803395 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-ef3a1a-openshift-default-6b6bf9786c-bwf8h" Jul 16 00:15:04.816746 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:04.816729 2563 scope.go:117] "RemoveContainer" containerID="cf5d38dcaf7b79026a3c0d34e43ea0720c9a41d3e1a613c3d858c1416a9c73ab" Jul 16 00:15:04.825707 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:04.825692 2563 scope.go:117] "RemoveContainer" containerID="0bd041ebb579213b15eda22949bc3cbe396e4560de27b09b90fe060fbf7d74bb" Jul 16 00:15:04.825907 ip-10-0-128-197 kubenswrapper[2563]: E0716 00:15:04.825891 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0bd041ebb579213b15eda22949bc3cbe396e4560de27b09b90fe060fbf7d74bb\": container with ID starting with 0bd041ebb579213b15eda22949bc3cbe396e4560de27b09b90fe060fbf7d74bb not found: ID does not exist" containerID="0bd041ebb579213b15eda22949bc3cbe396e4560de27b09b90fe060fbf7d74bb" Jul 16 00:15:04.825961 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:04.825914 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0bd041ebb579213b15eda22949bc3cbe396e4560de27b09b90fe060fbf7d74bb"} err="failed to get container status \"0bd041ebb579213b15eda22949bc3cbe396e4560de27b09b90fe060fbf7d74bb\": rpc error: code = NotFound desc = could not find container \"0bd041ebb579213b15eda22949bc3cbe396e4560de27b09b90fe060fbf7d74bb\": container with ID starting with 0bd041ebb579213b15eda22949bc3cbe396e4560de27b09b90fe060fbf7d74bb not found: ID does not exist" Jul 16 00:15:04.825961 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:04.825930 2563 scope.go:117] "RemoveContainer" containerID="cf5d38dcaf7b79026a3c0d34e43ea0720c9a41d3e1a613c3d858c1416a9c73ab" Jul 16 00:15:04.826129 ip-10-0-128-197 kubenswrapper[2563]: E0716 00:15:04.826116 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf5d38dcaf7b79026a3c0d34e43ea0720c9a41d3e1a613c3d858c1416a9c73ab\": container with ID starting with cf5d38dcaf7b79026a3c0d34e43ea0720c9a41d3e1a613c3d858c1416a9c73ab not found: ID does not exist" containerID="cf5d38dcaf7b79026a3c0d34e43ea0720c9a41d3e1a613c3d858c1416a9c73ab" Jul 16 00:15:04.826165 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:04.826132 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf5d38dcaf7b79026a3c0d34e43ea0720c9a41d3e1a613c3d858c1416a9c73ab"} err="failed to get container status \"cf5d38dcaf7b79026a3c0d34e43ea0720c9a41d3e1a613c3d858c1416a9c73ab\": rpc error: code = NotFound desc = could not find container \"cf5d38dcaf7b79026a3c0d34e43ea0720c9a41d3e1a613c3d858c1416a9c73ab\": container with ID starting with cf5d38dcaf7b79026a3c0d34e43ea0720c9a41d3e1a613c3d858c1416a9c73ab not found: ID does not exist" Jul 16 00:15:04.835336 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:04.835316 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-ef3a1a-openshift-default-6b6bf9786c-bwf8h"] Jul 16 00:15:04.839325 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:04.839305 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-ef3a1a-openshift-default-6b6bf9786c-bwf8h"] Jul 16 00:15:05.171732 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:05.171653 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="74652b67-44c7-4d41-b07b-c9b83a3952d8" path="/var/lib/kubelet/pods/74652b67-44c7-4d41-b07b-c9b83a3952d8/volumes" Jul 16 00:15:06.011249 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:06.011223 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:15:06.070231 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:06.070202 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:15:06.288701 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:06.288615 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-e94b3e-openshift-default-598bcf5c59-nnggv"] Jul 16 00:15:06.289156 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:06.288819 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-e94b3e-openshift-default-598bcf5c59-nnggv" podUID="9a1cb0d5-2afd-42a0-a18f-ac401bfbd9bb" containerName="istio-proxy" containerID="cri-o://98c8b91e3f8e33851fb32d62b991af5f0327782a78eae2c64d08798d1486cf7a" gracePeriod=30 Jul 16 00:15:06.298251 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:06.298216 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:15:06.553633 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:06.553553 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:15:06.996561 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:06.996532 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-584bd4579f-v6wzp_2038c4d3-a937-4f11-99ee-a43b6705e985/authorino/0.log" Jul 16 00:15:11.566604 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:11.566546 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-e94b3e-openshift-default-598bcf5c59-nnggv" Jul 16 00:15:11.651814 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:11.651774 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a1cb0d5-2afd-42a0-a18f-ac401bfbd9bb-credential-socket\") pod \"9a1cb0d5-2afd-42a0-a18f-ac401bfbd9bb\" (UID: \"9a1cb0d5-2afd-42a0-a18f-ac401bfbd9bb\") " Jul 16 00:15:11.651814 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:11.651823 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a1cb0d5-2afd-42a0-a18f-ac401bfbd9bb-istio-data\") pod \"9a1cb0d5-2afd-42a0-a18f-ac401bfbd9bb\" (UID: \"9a1cb0d5-2afd-42a0-a18f-ac401bfbd9bb\") " Jul 16 00:15:11.652063 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:11.651868 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a1cb0d5-2afd-42a0-a18f-ac401bfbd9bb-istio-podinfo\") pod \"9a1cb0d5-2afd-42a0-a18f-ac401bfbd9bb\" (UID: \"9a1cb0d5-2afd-42a0-a18f-ac401bfbd9bb\") " Jul 16 00:15:11.652063 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:11.651890 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vs42f\" (UniqueName: \"kubernetes.io/projected/9a1cb0d5-2afd-42a0-a18f-ac401bfbd9bb-kube-api-access-vs42f\") pod \"9a1cb0d5-2afd-42a0-a18f-ac401bfbd9bb\" (UID: \"9a1cb0d5-2afd-42a0-a18f-ac401bfbd9bb\") " Jul 16 00:15:11.652063 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:11.651943 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a1cb0d5-2afd-42a0-a18f-ac401bfbd9bb-workload-certs\") pod \"9a1cb0d5-2afd-42a0-a18f-ac401bfbd9bb\" (UID: \"9a1cb0d5-2afd-42a0-a18f-ac401bfbd9bb\") " Jul 16 00:15:11.652227 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:11.652074 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a1cb0d5-2afd-42a0-a18f-ac401bfbd9bb-istiod-ca-cert\") pod \"9a1cb0d5-2afd-42a0-a18f-ac401bfbd9bb\" (UID: \"9a1cb0d5-2afd-42a0-a18f-ac401bfbd9bb\") " Jul 16 00:15:11.652227 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:11.652116 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a1cb0d5-2afd-42a0-a18f-ac401bfbd9bb-workload-socket\") pod \"9a1cb0d5-2afd-42a0-a18f-ac401bfbd9bb\" (UID: \"9a1cb0d5-2afd-42a0-a18f-ac401bfbd9bb\") " Jul 16 00:15:11.652227 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:11.652109 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a1cb0d5-2afd-42a0-a18f-ac401bfbd9bb-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "9a1cb0d5-2afd-42a0-a18f-ac401bfbd9bb" (UID: "9a1cb0d5-2afd-42a0-a18f-ac401bfbd9bb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 00:15:11.652227 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:11.652146 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a1cb0d5-2afd-42a0-a18f-ac401bfbd9bb-istio-token\") pod \"9a1cb0d5-2afd-42a0-a18f-ac401bfbd9bb\" (UID: \"9a1cb0d5-2afd-42a0-a18f-ac401bfbd9bb\") " Jul 16 00:15:11.652227 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:11.652180 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a1cb0d5-2afd-42a0-a18f-ac401bfbd9bb-istio-envoy\") pod \"9a1cb0d5-2afd-42a0-a18f-ac401bfbd9bb\" (UID: \"9a1cb0d5-2afd-42a0-a18f-ac401bfbd9bb\") " Jul 16 00:15:11.652471 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:11.652225 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a1cb0d5-2afd-42a0-a18f-ac401bfbd9bb-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "9a1cb0d5-2afd-42a0-a18f-ac401bfbd9bb" (UID: "9a1cb0d5-2afd-42a0-a18f-ac401bfbd9bb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 00:15:11.652471 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:11.652321 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a1cb0d5-2afd-42a0-a18f-ac401bfbd9bb-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "9a1cb0d5-2afd-42a0-a18f-ac401bfbd9bb" (UID: "9a1cb0d5-2afd-42a0-a18f-ac401bfbd9bb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 00:15:11.652471 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:11.652443 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a1cb0d5-2afd-42a0-a18f-ac401bfbd9bb-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "9a1cb0d5-2afd-42a0-a18f-ac401bfbd9bb" (UID: "9a1cb0d5-2afd-42a0-a18f-ac401bfbd9bb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 00:15:11.652647 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:11.652591 2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a1cb0d5-2afd-42a0-a18f-ac401bfbd9bb-workload-certs\") on node \"ip-10-0-128-197.ec2.internal\" DevicePath \"\"" Jul 16 00:15:11.652647 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:11.652610 2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a1cb0d5-2afd-42a0-a18f-ac401bfbd9bb-istiod-ca-cert\") on node \"ip-10-0-128-197.ec2.internal\" DevicePath \"\"" Jul 16 00:15:11.652647 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:11.652624 2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a1cb0d5-2afd-42a0-a18f-ac401bfbd9bb-workload-socket\") on node \"ip-10-0-128-197.ec2.internal\" DevicePath \"\"" Jul 16 00:15:11.652647 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:11.652637 2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a1cb0d5-2afd-42a0-a18f-ac401bfbd9bb-credential-socket\") on node \"ip-10-0-128-197.ec2.internal\" DevicePath \"\"" Jul 16 00:15:11.652991 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:11.652951 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a1cb0d5-2afd-42a0-a18f-ac401bfbd9bb-istio-data" (OuterVolumeSpecName: "istio-data") pod "9a1cb0d5-2afd-42a0-a18f-ac401bfbd9bb" (UID: "9a1cb0d5-2afd-42a0-a18f-ac401bfbd9bb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 00:15:11.654128 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:11.654105 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9a1cb0d5-2afd-42a0-a18f-ac401bfbd9bb-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "9a1cb0d5-2afd-42a0-a18f-ac401bfbd9bb" (UID: "9a1cb0d5-2afd-42a0-a18f-ac401bfbd9bb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 16 00:15:11.654251 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:11.654237 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a1cb0d5-2afd-42a0-a18f-ac401bfbd9bb-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "9a1cb0d5-2afd-42a0-a18f-ac401bfbd9bb" (UID: "9a1cb0d5-2afd-42a0-a18f-ac401bfbd9bb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 00:15:11.654499 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:11.654474 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a1cb0d5-2afd-42a0-a18f-ac401bfbd9bb-istio-token" (OuterVolumeSpecName: "istio-token") pod "9a1cb0d5-2afd-42a0-a18f-ac401bfbd9bb" (UID: "9a1cb0d5-2afd-42a0-a18f-ac401bfbd9bb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 00:15:11.654499 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:11.654483 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a1cb0d5-2afd-42a0-a18f-ac401bfbd9bb-kube-api-access-vs42f" (OuterVolumeSpecName: "kube-api-access-vs42f") pod "9a1cb0d5-2afd-42a0-a18f-ac401bfbd9bb" (UID: "9a1cb0d5-2afd-42a0-a18f-ac401bfbd9bb"). InnerVolumeSpecName "kube-api-access-vs42f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 00:15:11.753848 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:11.753823 2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a1cb0d5-2afd-42a0-a18f-ac401bfbd9bb-istio-data\") on node \"ip-10-0-128-197.ec2.internal\" DevicePath \"\"" Jul 16 00:15:11.753848 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:11.753846 2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a1cb0d5-2afd-42a0-a18f-ac401bfbd9bb-istio-podinfo\") on node \"ip-10-0-128-197.ec2.internal\" DevicePath \"\"" Jul 16 00:15:11.754034 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:11.753856 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vs42f\" (UniqueName: \"kubernetes.io/projected/9a1cb0d5-2afd-42a0-a18f-ac401bfbd9bb-kube-api-access-vs42f\") on node \"ip-10-0-128-197.ec2.internal\" DevicePath \"\"" Jul 16 00:15:11.754034 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:11.753865 2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a1cb0d5-2afd-42a0-a18f-ac401bfbd9bb-istio-token\") on node \"ip-10-0-128-197.ec2.internal\" DevicePath \"\"" Jul 16 00:15:11.754034 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:11.753874 2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a1cb0d5-2afd-42a0-a18f-ac401bfbd9bb-istio-envoy\") on node \"ip-10-0-128-197.ec2.internal\" DevicePath \"\"" Jul 16 00:15:11.849192 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:11.849101 2563 generic.go:356] "Generic (PLEG): container finished" podID="9a1cb0d5-2afd-42a0-a18f-ac401bfbd9bb" containerID="98c8b91e3f8e33851fb32d62b991af5f0327782a78eae2c64d08798d1486cf7a" exitCode=0 Jul 16 00:15:11.849192 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:11.849170 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-e94b3e-openshift-default-598bcf5c59-nnggv" Jul 16 00:15:11.849359 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:11.849188 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-e94b3e-openshift-default-598bcf5c59-nnggv" event={"ID":"9a1cb0d5-2afd-42a0-a18f-ac401bfbd9bb","Type":"ContainerDied","Data":"98c8b91e3f8e33851fb32d62b991af5f0327782a78eae2c64d08798d1486cf7a"} Jul 16 00:15:11.849359 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:11.849264 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-e94b3e-openshift-default-598bcf5c59-nnggv" event={"ID":"9a1cb0d5-2afd-42a0-a18f-ac401bfbd9bb","Type":"ContainerDied","Data":"fa0ffb13dd604629926600e652dc98c39242fc82cbe42d44807df5d71848848e"} Jul 16 00:15:11.849359 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:11.849288 2563 scope.go:117] "RemoveContainer" containerID="98c8b91e3f8e33851fb32d62b991af5f0327782a78eae2c64d08798d1486cf7a" Jul 16 00:15:11.859290 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:11.859274 2563 scope.go:117] "RemoveContainer" containerID="598f000f65f19d07eb07a936c9912532d2fe0f17712f61ba65535520c9da6b4a" Jul 16 00:15:11.867707 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:11.867686 2563 scope.go:117] "RemoveContainer" containerID="98c8b91e3f8e33851fb32d62b991af5f0327782a78eae2c64d08798d1486cf7a" Jul 16 00:15:11.868189 ip-10-0-128-197 kubenswrapper[2563]: E0716 00:15:11.868025 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"98c8b91e3f8e33851fb32d62b991af5f0327782a78eae2c64d08798d1486cf7a\": container with ID starting with 98c8b91e3f8e33851fb32d62b991af5f0327782a78eae2c64d08798d1486cf7a not found: ID does not exist" containerID="98c8b91e3f8e33851fb32d62b991af5f0327782a78eae2c64d08798d1486cf7a" Jul 16 00:15:11.868189 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:11.868073 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98c8b91e3f8e33851fb32d62b991af5f0327782a78eae2c64d08798d1486cf7a"} err="failed to get container status \"98c8b91e3f8e33851fb32d62b991af5f0327782a78eae2c64d08798d1486cf7a\": rpc error: code = NotFound desc = could not find container \"98c8b91e3f8e33851fb32d62b991af5f0327782a78eae2c64d08798d1486cf7a\": container with ID starting with 98c8b91e3f8e33851fb32d62b991af5f0327782a78eae2c64d08798d1486cf7a not found: ID does not exist" Jul 16 00:15:11.868189 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:11.868099 2563 scope.go:117] "RemoveContainer" containerID="598f000f65f19d07eb07a936c9912532d2fe0f17712f61ba65535520c9da6b4a" Jul 16 00:15:11.868532 ip-10-0-128-197 kubenswrapper[2563]: E0716 00:15:11.868490 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"598f000f65f19d07eb07a936c9912532d2fe0f17712f61ba65535520c9da6b4a\": container with ID starting with 598f000f65f19d07eb07a936c9912532d2fe0f17712f61ba65535520c9da6b4a not found: ID does not exist" containerID="598f000f65f19d07eb07a936c9912532d2fe0f17712f61ba65535520c9da6b4a" Jul 16 00:15:11.868532 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:11.868521 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"598f000f65f19d07eb07a936c9912532d2fe0f17712f61ba65535520c9da6b4a"} err="failed to get container status \"598f000f65f19d07eb07a936c9912532d2fe0f17712f61ba65535520c9da6b4a\": rpc error: code = NotFound desc = could not find container \"598f000f65f19d07eb07a936c9912532d2fe0f17712f61ba65535520c9da6b4a\": container with ID starting with 598f000f65f19d07eb07a936c9912532d2fe0f17712f61ba65535520c9da6b4a not found: ID does not exist" Jul 16 00:15:11.870810 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:11.870789 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-e94b3e-openshift-default-598bcf5c59-nnggv"] Jul 16 00:15:11.878332 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:11.878311 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-e94b3e-openshift-default-598bcf5c59-nnggv"] Jul 16 00:15:13.169790 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:13.169759 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9a1cb0d5-2afd-42a0-a18f-ac401bfbd9bb" path="/var/lib/kubelet/pods/9a1cb0d5-2afd-42a0-a18f-ac401bfbd9bb/volumes" Jul 16 00:15:13.536186 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:13.536160 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5dcb68dc66-xjnhk_fb5125e5-eb7a-40ea-9a0a-27c4997beef5/copy-manifests/0.log" Jul 16 00:15:13.544404 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:13.544382 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5dcb68dc66-xjnhk_fb5125e5-eb7a-40ea-9a0a-27c4997beef5/manager/0.log" Jul 16 00:15:15.346155 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:15.346124 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29735985-4ns9m_48afe7ab-7a09-4d75-816f-b46baf75b974/cleanup/0.log" Jul 16 00:15:15.584827 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:15.584802 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29736015-97t7h_f1e446bd-bc6f-435e-a57b-fb37ad6b2786/cleanup/0.log" Jul 16 00:15:15.710331 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:15.710261 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-e2e-ait-65d40585-29736015-mkfkx_890529de-9adc-4f10-9611-43ef685116c0/cleanup/0.log" Jul 16 00:15:15.827841 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:15.827812 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-e2e-api-a-a22338-29736015-k7f4f_5ce3ce3b-842f-4b98-8263-7cece84002c6/cleanup/0.log" Jul 16 00:15:15.939918 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:15.939897 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-e2e-api-b-eae7a4-29736015-zcf7j_75696f17-5b1f-4f49-bb12-fd2a041c45e2/cleanup/0.log" Jul 16 00:15:16.052993 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:16.052953 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-e2e-derive-5a03fa09-29736015-wc8vn_afd0e0fa-91e1-4191-8265-c6c319035dba/cleanup/0.log" Jul 16 00:15:16.167038 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:16.167014 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-e2e-inf-a-d3fd63-29736015-7ftm8_ac7f3cfe-2835-4253-b11b-2971857d8b1c/cleanup/0.log" Jul 16 00:15:16.284781 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:16.284759 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-e2e-inf-b-dd8441-29736015-ktwck_c18b0d5b-2e6d-4e4c-9330-f77d2eebe189/cleanup/0.log" Jul 16 00:15:16.402997 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:16.402900 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-e2e-migrate-55cbdb16-29736015-btm49_56b285e9-f9f2-4cfe-af3d-29390185eb3d/cleanup/0.log" Jul 16 00:15:16.531016 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:16.530994 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-e2e-mt-df4149da-29736015-ljkbl_c5cf1c1b-6e76-4ead-b8db-2f6a300e6bc4/cleanup/0.log" Jul 16 00:15:16.647862 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:16.647841 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-e2e-shared-a-ef3a1a-29736015-zjjfm_db00bc8b-cc11-4d4d-81c7-98f9295e4f20/cleanup/0.log" Jul 16 00:15:16.778610 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:16.778587 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-e2e-shared-b-e94b3e-29736015-265rr_ef5f616c-898d-4d47-b853-336fdcff4d60/cleanup/0.log" Jul 16 00:15:18.391232 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:18.391203 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-584bd4579f-v6wzp_2038c4d3-a937-4f11-99ee-a43b6705e985/authorino/0.log" Jul 16 00:15:19.731949 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:19.731917 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4ldjtc_a2e4216a-a127-4cfd-8510-c1352e97aa97/istio-proxy/0.log" Jul 16 00:15:20.021532 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:20.021459 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7d6684b565-gd7qb_73313ae6-33c4-4994-b37e-31324804d8f2/kube-auth-proxy/0.log" Jul 16 00:15:20.287347 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:20.287258 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-sdscx_5c655a6b-6d89-459f-b331-3f8b08d0eae0/istio-proxy/1.log" Jul 16 00:15:21.071571 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:21.071542 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-7cx2p_f5105a71-3f0e-4641-ad18-0b9c6c552cdc/main/0.log" Jul 16 00:15:21.465122 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:21.465040 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbws9xh_ce6485a1-9190-4c1a-953b-0076bced45d4/main/0.log" Jul 16 00:15:28.502936 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:28.502904 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-px62w/must-gather-m5h5q"] Jul 16 00:15:28.503365 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:28.503333 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="74652b67-44c7-4d41-b07b-c9b83a3952d8" containerName="istio-proxy" Jul 16 00:15:28.503365 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:28.503348 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="74652b67-44c7-4d41-b07b-c9b83a3952d8" containerName="istio-proxy" Jul 16 00:15:28.503365 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:28.503356 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9a1cb0d5-2afd-42a0-a18f-ac401bfbd9bb" containerName="istio-proxy" Jul 16 00:15:28.503365 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:28.503362 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="9a1cb0d5-2afd-42a0-a18f-ac401bfbd9bb" containerName="istio-proxy" Jul 16 00:15:28.503513 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:28.503370 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ac7f3cfe-2835-4253-b11b-2971857d8b1c" containerName="cleanup" Jul 16 00:15:28.503513 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:28.503376 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="ac7f3cfe-2835-4253-b11b-2971857d8b1c" containerName="cleanup" Jul 16 00:15:28.503513 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:28.503382 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef5f616c-898d-4d47-b853-336fdcff4d60" containerName="cleanup" Jul 16 00:15:28.503513 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:28.503388 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef5f616c-898d-4d47-b853-336fdcff4d60" containerName="cleanup" Jul 16 00:15:28.503513 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:28.503396 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9a1cb0d5-2afd-42a0-a18f-ac401bfbd9bb" containerName="istio-proxy" Jul 16 00:15:28.503513 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:28.503401 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="9a1cb0d5-2afd-42a0-a18f-ac401bfbd9bb" containerName="istio-proxy" Jul 16 00:15:28.503513 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:28.503411 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="75696f17-5b1f-4f49-bb12-fd2a041c45e2" containerName="cleanup" Jul 16 00:15:28.503513 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:28.503416 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="75696f17-5b1f-4f49-bb12-fd2a041c45e2" containerName="cleanup" Jul 16 00:15:28.503513 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:28.503422 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="db00bc8b-cc11-4d4d-81c7-98f9295e4f20" containerName="cleanup" Jul 16 00:15:28.503513 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:28.503427 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="db00bc8b-cc11-4d4d-81c7-98f9295e4f20" containerName="cleanup" Jul 16 00:15:28.503513 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:28.503444 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c18b0d5b-2e6d-4e4c-9330-f77d2eebe189" containerName="cleanup" Jul 16 00:15:28.503513 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:28.503449 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="c18b0d5b-2e6d-4e4c-9330-f77d2eebe189" containerName="cleanup" Jul 16 00:15:28.503513 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:28.503456 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c5cf1c1b-6e76-4ead-b8db-2f6a300e6bc4" containerName="cleanup" Jul 16 00:15:28.503513 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:28.503461 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="c5cf1c1b-6e76-4ead-b8db-2f6a300e6bc4" containerName="cleanup" Jul 16 00:15:28.503513 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:28.503471 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="56b285e9-f9f2-4cfe-af3d-29390185eb3d" containerName="cleanup" Jul 16 00:15:28.503513 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:28.503475 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="56b285e9-f9f2-4cfe-af3d-29390185eb3d" containerName="cleanup" Jul 16 00:15:28.503513 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:28.503481 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5ce3ce3b-842f-4b98-8263-7cece84002c6" containerName="cleanup" Jul 16 00:15:28.503513 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:28.503486 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="5ce3ce3b-842f-4b98-8263-7cece84002c6" containerName="cleanup" Jul 16 00:15:28.503513 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:28.503493 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="890529de-9adc-4f10-9611-43ef685116c0" containerName="cleanup" Jul 16 00:15:28.503513 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:28.503498 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="890529de-9adc-4f10-9611-43ef685116c0" containerName="cleanup" Jul 16 00:15:28.503513 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:28.503506 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="74652b67-44c7-4d41-b07b-c9b83a3952d8" containerName="istio-proxy" Jul 16 00:15:28.503513 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:28.503511 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="74652b67-44c7-4d41-b07b-c9b83a3952d8" containerName="istio-proxy" Jul 16 00:15:28.503513 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:28.503519 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="afd0e0fa-91e1-4191-8265-c6c319035dba" containerName="cleanup" Jul 16 00:15:28.503513 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:28.503524 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="afd0e0fa-91e1-4191-8265-c6c319035dba" containerName="cleanup" Jul 16 00:15:28.504285 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:28.503531 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f1e446bd-bc6f-435e-a57b-fb37ad6b2786" containerName="cleanup" Jul 16 00:15:28.504285 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:28.503536 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="f1e446bd-bc6f-435e-a57b-fb37ad6b2786" containerName="cleanup" Jul 16 00:15:28.504285 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:28.503598 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="74652b67-44c7-4d41-b07b-c9b83a3952d8" containerName="istio-proxy" Jul 16 00:15:28.504285 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:28.503606 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="afd0e0fa-91e1-4191-8265-c6c319035dba" containerName="cleanup" Jul 16 00:15:28.504285 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:28.503613 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="ac7f3cfe-2835-4253-b11b-2971857d8b1c" containerName="cleanup" Jul 16 00:15:28.504285 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:28.503618 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="ef5f616c-898d-4d47-b853-336fdcff4d60" containerName="cleanup" Jul 16 00:15:28.504285 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:28.503625 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="75696f17-5b1f-4f49-bb12-fd2a041c45e2" containerName="cleanup" Jul 16 00:15:28.504285 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:28.503629 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="db00bc8b-cc11-4d4d-81c7-98f9295e4f20" containerName="cleanup" Jul 16 00:15:28.504285 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:28.503635 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="5ce3ce3b-842f-4b98-8263-7cece84002c6" containerName="cleanup" Jul 16 00:15:28.504285 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:28.503640 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="890529de-9adc-4f10-9611-43ef685116c0" containerName="cleanup" Jul 16 00:15:28.504285 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:28.503646 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="c5cf1c1b-6e76-4ead-b8db-2f6a300e6bc4" containerName="cleanup" Jul 16 00:15:28.504285 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:28.503651 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="c18b0d5b-2e6d-4e4c-9330-f77d2eebe189" containerName="cleanup" Jul 16 00:15:28.504285 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:28.503657 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="56b285e9-f9f2-4cfe-af3d-29390185eb3d" containerName="cleanup" Jul 16 00:15:28.504285 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:28.503662 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="9a1cb0d5-2afd-42a0-a18f-ac401bfbd9bb" containerName="istio-proxy" Jul 16 00:15:28.504285 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:28.503669 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="9a1cb0d5-2afd-42a0-a18f-ac401bfbd9bb" containerName="istio-proxy" Jul 16 00:15:28.504285 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:28.503673 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="f1e446bd-bc6f-435e-a57b-fb37ad6b2786" containerName="cleanup" Jul 16 00:15:28.504285 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:28.503796 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="74652b67-44c7-4d41-b07b-c9b83a3952d8" containerName="istio-proxy" Jul 16 00:15:28.506477 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:28.506451 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:15:28.508760 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:28.508746 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-px62w/must-gather-m5h5q" Jul 16 00:15:28.510921 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:28.510898 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-px62w\"/\"kube-root-ca.crt\"" Jul 16 00:15:28.511065 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:28.511002 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-px62w\"/\"default-dockercfg-88t76\"" Jul 16 00:15:28.511796 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:28.511768 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-px62w\"/\"openshift-service-ca.crt\"" Jul 16 00:15:28.515063 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:28.515040 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-px62w/must-gather-m5h5q"] Jul 16 00:15:28.607057 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:28.607023 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:15:28.611821 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:28.611799 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/8606b257-da19-4869-89cc-6f6625527a2c-must-gather-output\") pod \"must-gather-m5h5q\" (UID: \"8606b257-da19-4869-89cc-6f6625527a2c\") " pod="openshift-must-gather-px62w/must-gather-m5h5q" Jul 16 00:15:28.611942 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:28.611870 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-krls9\" (UniqueName: \"kubernetes.io/projected/8606b257-da19-4869-89cc-6f6625527a2c-kube-api-access-krls9\") pod \"must-gather-m5h5q\" (UID: \"8606b257-da19-4869-89cc-6f6625527a2c\") " pod="openshift-must-gather-px62w/must-gather-m5h5q" Jul 16 00:15:28.713105 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:28.713067 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/8606b257-da19-4869-89cc-6f6625527a2c-must-gather-output\") pod \"must-gather-m5h5q\" (UID: \"8606b257-da19-4869-89cc-6f6625527a2c\") " pod="openshift-must-gather-px62w/must-gather-m5h5q" Jul 16 00:15:28.713232 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:28.713157 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krls9\" (UniqueName: \"kubernetes.io/projected/8606b257-da19-4869-89cc-6f6625527a2c-kube-api-access-krls9\") pod \"must-gather-m5h5q\" (UID: \"8606b257-da19-4869-89cc-6f6625527a2c\") " pod="openshift-must-gather-px62w/must-gather-m5h5q" Jul 16 00:15:28.713385 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:28.713367 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/8606b257-da19-4869-89cc-6f6625527a2c-must-gather-output\") pod \"must-gather-m5h5q\" (UID: \"8606b257-da19-4869-89cc-6f6625527a2c\") " pod="openshift-must-gather-px62w/must-gather-m5h5q" Jul 16 00:15:28.725043 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:28.725019 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-krls9\" (UniqueName: \"kubernetes.io/projected/8606b257-da19-4869-89cc-6f6625527a2c-kube-api-access-krls9\") pod \"must-gather-m5h5q\" (UID: \"8606b257-da19-4869-89cc-6f6625527a2c\") " pod="openshift-must-gather-px62w/must-gather-m5h5q" Jul 16 00:15:28.819995 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:28.819796 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-px62w/must-gather-m5h5q" Jul 16 00:15:28.950641 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:28.950611 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-px62w/must-gather-m5h5q"] Jul 16 00:15:28.952324 ip-10-0-128-197 kubenswrapper[2563]: W0716 00:15:28.952298 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8606b257_da19_4869_89cc_6f6625527a2c.slice/crio-817ddd3f90aea0046f7d72bddd3fb6c20ba6342bdb20aa1bf0a2dcca605aca9f WatchSource:0}: Error finding container 817ddd3f90aea0046f7d72bddd3fb6c20ba6342bdb20aa1bf0a2dcca605aca9f: Status 404 returned error can't find the container with id 817ddd3f90aea0046f7d72bddd3fb6c20ba6342bdb20aa1bf0a2dcca605aca9f Jul 16 00:15:29.953229 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:29.953194 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-px62w/must-gather-m5h5q" event={"ID":"8606b257-da19-4869-89cc-6f6625527a2c","Type":"ContainerStarted","Data":"817ddd3f90aea0046f7d72bddd3fb6c20ba6342bdb20aa1bf0a2dcca605aca9f"} Jul 16 00:15:30.964286 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:30.964249 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-px62w/must-gather-m5h5q" event={"ID":"8606b257-da19-4869-89cc-6f6625527a2c","Type":"ContainerStarted","Data":"d69b89ccf86b365ab298e574d594ef415f3c56126d81460d5ce90716ffa22396"} Jul 16 00:15:30.964847 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:30.964810 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-px62w/must-gather-m5h5q" event={"ID":"8606b257-da19-4869-89cc-6f6625527a2c","Type":"ContainerStarted","Data":"2de2ad35fe33f55ebdc142f7fe3b4261b46f47f24694cf92d2a7f5e094a81aba"} Jul 16 00:15:30.981115 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:30.981058 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-px62w/must-gather-m5h5q" podStartSLOduration=1.8413569 podStartE2EDuration="2.981039189s" totalImagesPullingTime="1.139682289s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 00:15:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 00:15:30.979870036 +0000 UTC m=+2346.353328003" watchObservedRunningTime="2026-07-16 00:15:30.981039189 +0000 UTC m=+2346.354497154" Jul 16 00:15:31.678208 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:31.678178 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-4zw2v_10a402bd-5fc8-41c2-852a-5df876f74bdb/global-pull-secret-syncer/0.log" Jul 16 00:15:31.804920 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:31.804887 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" Jul 16 00:15:31.895258 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:31.895234 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-128-197.ec2.internal_1c95fb996cc0f9a497fcf20794fa4db8/haproxy/0.log" Jul 16 00:15:36.999381 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:36.999297 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-584bd4579f-v6wzp_2038c4d3-a937-4f11-99ee-a43b6705e985/authorino/0.log" Jul 16 00:15:38.808755 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:38.808702 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ff59_2deddafa-cbdf-407b-98c2-7c73bd107f1b/konnectivity-agent/0.log" limit=1024 Jul 16 00:15:38.930988 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:38.930943 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_76c2c664-fc68-45f0-90d7-839b197c0fac/alertmanager/0.log" Jul 16 00:15:38.965987 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:38.965474 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_76c2c664-fc68-45f0-90d7-839b197c0fac/config-reloader/0.log" Jul 16 00:15:38.988987 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:38.988747 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_76c2c664-fc68-45f0-90d7-839b197c0fac/kube-rbac-proxy-web/0.log" Jul 16 00:15:39.020169 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:39.020141 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_76c2c664-fc68-45f0-90d7-839b197c0fac/kube-rbac-proxy/0.log" Jul 16 00:15:39.047417 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:39.047386 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_76c2c664-fc68-45f0-90d7-839b197c0fac/kube-rbac-proxy-metric/0.log" Jul 16 00:15:39.070364 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:39.070281 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_76c2c664-fc68-45f0-90d7-839b197c0fac/prom-label-proxy/0.log" Jul 16 00:15:39.096503 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:39.096467 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_76c2c664-fc68-45f0-90d7-839b197c0fac/init-config-reloader/0.log" Jul 16 00:15:39.191169 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:39.191131 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-785778ff74-2dsvn_f6b39980-f5f7-4709-b768-6cc2b2016cd1/kube-state-metrics/0.log" Jul 16 00:15:39.218857 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:39.218823 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-785778ff74-2dsvn_f6b39980-f5f7-4709-b768-6cc2b2016cd1/kube-rbac-proxy-main/0.log" Jul 16 00:15:39.239501 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:39.239467 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-785778ff74-2dsvn_f6b39980-f5f7-4709-b768-6cc2b2016cd1/kube-rbac-proxy-self/0.log" Jul 16 00:15:39.267255 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:39.267227 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-7cb86fd7bc-4bphf_692b9b9d-1237-4e9c-bf86-0255ab7732b7/metrics-server/0.log" Jul 16 00:15:39.292823 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:39.292794 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-5f68df7f58-gkz55_e0f740d6-d99f-48e0-8da0-66e359d9fb48/monitoring-plugin/0.log" Jul 16 00:15:39.490173 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:39.490145 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-w5dxc_96ec5aef-3317-46d0-be5b-952af4818e44/node-exporter/0.log" Jul 16 00:15:39.511179 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:39.511154 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-w5dxc_96ec5aef-3317-46d0-be5b-952af4818e44/kube-rbac-proxy/0.log" Jul 16 00:15:39.532846 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:39.532816 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-w5dxc_96ec5aef-3317-46d0-be5b-952af4818e44/init-textfile/0.log" Jul 16 00:15:39.561822 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:39.561780 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-84bd68b896-8sn4g_dccc9670-e502-402d-b0cd-63d1c54f136f/kube-rbac-proxy-main/0.log" Jul 16 00:15:39.588335 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:39.588308 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-84bd68b896-8sn4g_dccc9670-e502-402d-b0cd-63d1c54f136f/kube-rbac-proxy-self/0.log" Jul 16 00:15:39.613584 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:39.613465 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-84bd68b896-8sn4g_dccc9670-e502-402d-b0cd-63d1c54f136f/openshift-state-metrics/0.log" Jul 16 00:15:39.649593 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:39.649561 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_69f21f45-2600-4cef-9230-5c5d37e80c88/prometheus/0.log" Jul 16 00:15:39.670215 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:39.670185 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_69f21f45-2600-4cef-9230-5c5d37e80c88/config-reloader/0.log" Jul 16 00:15:39.695356 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:39.695331 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_69f21f45-2600-4cef-9230-5c5d37e80c88/thanos-sidecar/0.log" Jul 16 00:15:39.716403 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:39.716377 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_69f21f45-2600-4cef-9230-5c5d37e80c88/kube-rbac-proxy-web/0.log" Jul 16 00:15:39.751986 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:39.748125 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_69f21f45-2600-4cef-9230-5c5d37e80c88/kube-rbac-proxy/0.log" Jul 16 00:15:39.779404 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:39.779376 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_69f21f45-2600-4cef-9230-5c5d37e80c88/kube-rbac-proxy-thanos/0.log" Jul 16 00:15:39.803443 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:39.803348 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_69f21f45-2600-4cef-9230-5c5d37e80c88/init-config-reloader/0.log" Jul 16 00:15:39.873518 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:39.873382 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-64785b6675-gr6rk_1a03a952-c7c3-4029-b3dc-f5c233499798/prometheus-operator/0.log" Jul 16 00:15:39.894339 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:39.894310 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-64785b6675-gr6rk_1a03a952-c7c3-4029-b3dc-f5c233499798/kube-rbac-proxy/0.log" Jul 16 00:15:39.952028 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:39.951999 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-786d565755-fqg7t_7f775c78-eab6-437d-a2e9-291049e51b94/telemeter-client/0.log" Jul 16 00:15:39.977273 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:39.977238 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-786d565755-fqg7t_7f775c78-eab6-437d-a2e9-291049e51b94/reload/0.log" Jul 16 00:15:40.003849 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:40.003819 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-786d565755-fqg7t_7f775c78-eab6-437d-a2e9-291049e51b94/kube-rbac-proxy/0.log" Jul 16 00:15:40.044595 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:40.044571 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-b7fc86bbb-vrm2p_cc051ada-7c71-4701-94cf-914ad0e6c65c/thanos-query/0.log" Jul 16 00:15:40.066175 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:40.066129 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-b7fc86bbb-vrm2p_cc051ada-7c71-4701-94cf-914ad0e6c65c/kube-rbac-proxy-web/0.log" Jul 16 00:15:40.087942 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:40.087907 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-b7fc86bbb-vrm2p_cc051ada-7c71-4701-94cf-914ad0e6c65c/kube-rbac-proxy/0.log" Jul 16 00:15:40.112023 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:40.111992 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-b7fc86bbb-vrm2p_cc051ada-7c71-4701-94cf-914ad0e6c65c/prom-label-proxy/0.log" Jul 16 00:15:40.138357 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:40.138227 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-b7fc86bbb-vrm2p_cc051ada-7c71-4701-94cf-914ad0e6c65c/kube-rbac-proxy-rules/0.log" Jul 16 00:15:40.168125 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:40.168090 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-b7fc86bbb-vrm2p_cc051ada-7c71-4701-94cf-914ad0e6c65c/kube-rbac-proxy-metrics/0.log" Jul 16 00:15:41.966581 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:41.966553 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-px62w/perf-node-gather-daemonset-q9nhf"] Jul 16 00:15:41.972312 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:41.972290 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-px62w/perf-node-gather-daemonset-q9nhf" Jul 16 00:15:41.979787 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:41.979755 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-px62w/perf-node-gather-daemonset-q9nhf"] Jul 16 00:15:42.073990 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:42.073608 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d67d383a-e79e-48c7-8679-c51a3d3bbd53-lib-modules\") pod \"perf-node-gather-daemonset-q9nhf\" (UID: \"d67d383a-e79e-48c7-8679-c51a3d3bbd53\") " pod="openshift-must-gather-px62w/perf-node-gather-daemonset-q9nhf" Jul 16 00:15:42.073990 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:42.073662 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d67d383a-e79e-48c7-8679-c51a3d3bbd53-sys\") pod \"perf-node-gather-daemonset-q9nhf\" (UID: \"d67d383a-e79e-48c7-8679-c51a3d3bbd53\") " pod="openshift-must-gather-px62w/perf-node-gather-daemonset-q9nhf" Jul 16 00:15:42.073990 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:42.073715 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d67d383a-e79e-48c7-8679-c51a3d3bbd53-proc\") pod \"perf-node-gather-daemonset-q9nhf\" (UID: \"d67d383a-e79e-48c7-8679-c51a3d3bbd53\") " pod="openshift-must-gather-px62w/perf-node-gather-daemonset-q9nhf" Jul 16 00:15:42.073990 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:42.073746 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d67d383a-e79e-48c7-8679-c51a3d3bbd53-podres\") pod \"perf-node-gather-daemonset-q9nhf\" (UID: \"d67d383a-e79e-48c7-8679-c51a3d3bbd53\") " pod="openshift-must-gather-px62w/perf-node-gather-daemonset-q9nhf" Jul 16 00:15:42.073990 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:42.073770 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67dl4\" (UniqueName: \"kubernetes.io/projected/d67d383a-e79e-48c7-8679-c51a3d3bbd53-kube-api-access-67dl4\") pod \"perf-node-gather-daemonset-q9nhf\" (UID: \"d67d383a-e79e-48c7-8679-c51a3d3bbd53\") " pod="openshift-must-gather-px62w/perf-node-gather-daemonset-q9nhf" Jul 16 00:15:42.175063 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:42.175029 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d67d383a-e79e-48c7-8679-c51a3d3bbd53-lib-modules\") pod \"perf-node-gather-daemonset-q9nhf\" (UID: \"d67d383a-e79e-48c7-8679-c51a3d3bbd53\") " pod="openshift-must-gather-px62w/perf-node-gather-daemonset-q9nhf" Jul 16 00:15:42.175262 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:42.175087 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d67d383a-e79e-48c7-8679-c51a3d3bbd53-sys\") pod \"perf-node-gather-daemonset-q9nhf\" (UID: \"d67d383a-e79e-48c7-8679-c51a3d3bbd53\") " pod="openshift-must-gather-px62w/perf-node-gather-daemonset-q9nhf" Jul 16 00:15:42.175262 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:42.175131 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d67d383a-e79e-48c7-8679-c51a3d3bbd53-proc\") pod \"perf-node-gather-daemonset-q9nhf\" (UID: \"d67d383a-e79e-48c7-8679-c51a3d3bbd53\") " pod="openshift-must-gather-px62w/perf-node-gather-daemonset-q9nhf" Jul 16 00:15:42.175262 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:42.175158 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d67d383a-e79e-48c7-8679-c51a3d3bbd53-podres\") pod \"perf-node-gather-daemonset-q9nhf\" (UID: \"d67d383a-e79e-48c7-8679-c51a3d3bbd53\") " pod="openshift-must-gather-px62w/perf-node-gather-daemonset-q9nhf" Jul 16 00:15:42.175262 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:42.175194 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d67d383a-e79e-48c7-8679-c51a3d3bbd53-lib-modules\") pod \"perf-node-gather-daemonset-q9nhf\" (UID: \"d67d383a-e79e-48c7-8679-c51a3d3bbd53\") " pod="openshift-must-gather-px62w/perf-node-gather-daemonset-q9nhf" Jul 16 00:15:42.175262 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:42.175204 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d67d383a-e79e-48c7-8679-c51a3d3bbd53-sys\") pod \"perf-node-gather-daemonset-q9nhf\" (UID: \"d67d383a-e79e-48c7-8679-c51a3d3bbd53\") " pod="openshift-must-gather-px62w/perf-node-gather-daemonset-q9nhf" Jul 16 00:15:42.175262 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:42.175231 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d67d383a-e79e-48c7-8679-c51a3d3bbd53-proc\") pod \"perf-node-gather-daemonset-q9nhf\" (UID: \"d67d383a-e79e-48c7-8679-c51a3d3bbd53\") " pod="openshift-must-gather-px62w/perf-node-gather-daemonset-q9nhf" Jul 16 00:15:42.175262 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:42.175246 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d67d383a-e79e-48c7-8679-c51a3d3bbd53-podres\") pod \"perf-node-gather-daemonset-q9nhf\" (UID: \"d67d383a-e79e-48c7-8679-c51a3d3bbd53\") " pod="openshift-must-gather-px62w/perf-node-gather-daemonset-q9nhf" Jul 16 00:15:42.175567 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:42.175278 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67dl4\" (UniqueName: \"kubernetes.io/projected/d67d383a-e79e-48c7-8679-c51a3d3bbd53-kube-api-access-67dl4\") pod \"perf-node-gather-daemonset-q9nhf\" (UID: \"d67d383a-e79e-48c7-8679-c51a3d3bbd53\") " pod="openshift-must-gather-px62w/perf-node-gather-daemonset-q9nhf" Jul 16 00:15:42.184908 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:42.184883 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-67dl4\" (UniqueName: \"kubernetes.io/projected/d67d383a-e79e-48c7-8679-c51a3d3bbd53-kube-api-access-67dl4\") pod \"perf-node-gather-daemonset-q9nhf\" (UID: \"d67d383a-e79e-48c7-8679-c51a3d3bbd53\") " pod="openshift-must-gather-px62w/perf-node-gather-daemonset-q9nhf" Jul 16 00:15:42.288001 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:42.287949 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-px62w/perf-node-gather-daemonset-q9nhf" Jul 16 00:15:42.463041 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:42.463011 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-px62w/perf-node-gather-daemonset-q9nhf"] Jul 16 00:15:42.466151 ip-10-0-128-197 kubenswrapper[2563]: W0716 00:15:42.466098 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podd67d383a_e79e_48c7_8679_c51a3d3bbd53.slice/crio-626068cc1b270d2952e54299d5fd7fe201f2e46a15a1dd68c4dd39aea3870a3a WatchSource:0}: Error finding container 626068cc1b270d2952e54299d5fd7fe201f2e46a15a1dd68c4dd39aea3870a3a: Status 404 returned error can't find the container with id 626068cc1b270d2952e54299d5fd7fe201f2e46a15a1dd68c4dd39aea3870a3a Jul 16 00:15:42.531711 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:42.530977 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-7c98766766-59kcb_32a80dce-f1af-4b10-ad64-3a783d915e63/console/0.log" Jul 16 00:15:42.567185 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:42.567116 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-68c9cbb497-4xmmb_59e5d97d-b146-4ca9-96c3-a2702d1d36ae/download-server/0.log" Jul 16 00:15:43.073630 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:43.073587 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-px62w/perf-node-gather-daemonset-q9nhf" event={"ID":"d67d383a-e79e-48c7-8679-c51a3d3bbd53","Type":"ContainerStarted","Data":"8fa49b24daec6e4a4c27301db611209558b6ccaf47590049d5c4679aff9e5646"} Jul 16 00:15:43.074151 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:43.073665 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-px62w/perf-node-gather-daemonset-q9nhf" event={"ID":"d67d383a-e79e-48c7-8679-c51a3d3bbd53","Type":"ContainerStarted","Data":"626068cc1b270d2952e54299d5fd7fe201f2e46a15a1dd68c4dd39aea3870a3a"} Jul 16 00:15:43.074553 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:43.074526 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-px62w/perf-node-gather-daemonset-q9nhf" Jul 16 00:15:43.089539 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:43.089486 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-px62w/perf-node-gather-daemonset-q9nhf" podStartSLOduration=2.089465753 podStartE2EDuration="2.089465753s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 00:15:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 00:15:43.089347208 +0000 UTC m=+2358.462805189" watchObservedRunningTime="2026-07-16 00:15:43.089465753 +0000 UTC m=+2358.462923717" Jul 16 00:15:44.322118 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:44.322090 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-wkhxm_e53610cd-0777-4a19-afd0-6133052fc189/dns/0.log" Jul 16 00:15:44.347588 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:44.347553 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-wkhxm_e53610cd-0777-4a19-afd0-6133052fc189/kube-rbac-proxy/0.log" Jul 16 00:15:44.416557 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:44.416527 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-z9q4h_f9fcf8a8-82dc-4e0b-b211-4611f9436609/dns-node-resolver/0.log" Jul 16 00:15:44.976255 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:44.976221 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-khwkr_498d1e24-acb4-4d75-a5a7-410f00ecc9f9/node-ca/0.log" Jul 16 00:15:45.982168 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:45.982124 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4ldjtc_a2e4216a-a127-4cfd-8510-c1352e97aa97/istio-proxy/0.log" Jul 16 00:15:46.173686 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:46.173645 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7d6684b565-gd7qb_73313ae6-33c4-4994-b37e-31324804d8f2/kube-auth-proxy/0.log" Jul 16 00:15:46.227460 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:46.227417 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-sdscx_5c655a6b-6d89-459f-b331-3f8b08d0eae0/istio-proxy/0.log" Jul 16 00:15:46.264147 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:46.264069 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-sdscx_5c655a6b-6d89-459f-b331-3f8b08d0eae0/istio-proxy/1.log" Jul 16 00:15:47.063135 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:47.063101 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-vlt7c_def13391-3ff3-48cf-83d3-03df84d280da/serve-healthcheck-canary/0.log" Jul 16 00:15:47.570571 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:47.570542 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-5q8md_409f6854-b231-41aa-8715-a300ee9aa09e/kube-rbac-proxy/0.log" Jul 16 00:15:47.592750 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:47.592724 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-5q8md_409f6854-b231-41aa-8715-a300ee9aa09e/exporter/0.log" Jul 16 00:15:47.616756 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:47.616720 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-5q8md_409f6854-b231-41aa-8715-a300ee9aa09e/extractor/0.log" Jul 16 00:15:50.100748 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:50.100725 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-px62w/perf-node-gather-daemonset-q9nhf" Jul 16 00:15:50.367959 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:50.367880 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5dcb68dc66-xjnhk_fb5125e5-eb7a-40ea-9a0a-27c4997beef5/manager/0.log" Jul 16 00:15:50.390750 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:50.390720 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5dcb68dc66-xjnhk_fb5125e5-eb7a-40ea-9a0a-27c4997beef5/copy-manifests/0.log" Jul 16 00:15:51.596532 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:51.596505 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-787bd49fc4-pmzdt_2fc4c9e2-ffb6-4dae-9041-e2a5a5ac820b/manager/0.log" Jul 16 00:15:51.628257 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:51.628228 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-vf6zb_8e445fcf-dd96-4267-a442-8d52334e53ec/openshift-lws-operator/0.log" Jul 16 00:15:52.045856 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:52.045818 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4ldjtc_a2e4216a-a127-4cfd-8510-c1352e97aa97/istio-proxy/0.log" Jul 16 00:15:52.226889 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:52.226842 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7d6684b565-gd7qb_73313ae6-33c4-4994-b37e-31324804d8f2/kube-auth-proxy/0.log" Jul 16 00:15:52.263133 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:52.263059 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-sdscx_5c655a6b-6d89-459f-b331-3f8b08d0eae0/istio-proxy/0.log" Jul 16 00:15:52.308456 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:52.308362 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-sdscx_5c655a6b-6d89-459f-b331-3f8b08d0eae0/istio-proxy/1.log" Jul 16 00:15:53.683062 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:53.683030 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4ldjtc_a2e4216a-a127-4cfd-8510-c1352e97aa97/istio-proxy/0.log" Jul 16 00:15:53.861171 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:53.861141 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7d6684b565-gd7qb_73313ae6-33c4-4994-b37e-31324804d8f2/kube-auth-proxy/0.log" Jul 16 00:15:53.905853 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:53.905819 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-sdscx_5c655a6b-6d89-459f-b331-3f8b08d0eae0/istio-proxy/0.log" Jul 16 00:15:53.949308 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:53.949230 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-sdscx_5c655a6b-6d89-459f-b331-3f8b08d0eae0/istio-proxy/1.log" Jul 16 00:15:58.174746 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:58.174712 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-tg24d_a87574b4-736b-4e7d-9af6-3b6f89ac1c31/kube-multus-additional-cni-plugins/0.log" Jul 16 00:15:58.197378 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:58.197350 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-tg24d_a87574b4-736b-4e7d-9af6-3b6f89ac1c31/egress-router-binary-copy/0.log" Jul 16 00:15:58.217560 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:58.217536 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-tg24d_a87574b4-736b-4e7d-9af6-3b6f89ac1c31/cni-plugins/0.log" Jul 16 00:15:58.240022 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:58.239990 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-tg24d_a87574b4-736b-4e7d-9af6-3b6f89ac1c31/bond-cni-plugin/0.log" Jul 16 00:15:58.259053 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:58.259036 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-tg24d_a87574b4-736b-4e7d-9af6-3b6f89ac1c31/routeoverride-cni/0.log" Jul 16 00:15:58.278002 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:58.277984 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-tg24d_a87574b4-736b-4e7d-9af6-3b6f89ac1c31/whereabouts-cni-bincopy/0.log" Jul 16 00:15:58.299134 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:58.299114 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-tg24d_a87574b4-736b-4e7d-9af6-3b6f89ac1c31/whereabouts-cni/0.log" Jul 16 00:15:58.507575 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:58.507548 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-v62rk_12202d45-85c8-41de-b95e-f5cf403572ae/kube-multus/0.log" Jul 16 00:15:58.529307 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:58.529260 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-7n4t9_d1c182db-bcdc-40a1-bb82-4f8b64173e43/network-metrics-daemon/0.log" Jul 16 00:15:58.547623 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:58.547600 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-7n4t9_d1c182db-bcdc-40a1-bb82-4f8b64173e43/kube-rbac-proxy/0.log" Jul 16 00:15:59.659425 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:59.659380 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d8f6k_2f64cab3-104a-44f6-b125-ba458f54f058/ovn-controller/0.log" Jul 16 00:15:59.677528 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:59.677503 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d8f6k_2f64cab3-104a-44f6-b125-ba458f54f058/ovn-acl-logging/0.log" Jul 16 00:15:59.677669 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:59.677580 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d8f6k_2f64cab3-104a-44f6-b125-ba458f54f058/ovn-acl-logging/1.log" Jul 16 00:15:59.699723 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:59.699704 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d8f6k_2f64cab3-104a-44f6-b125-ba458f54f058/kube-rbac-proxy-node/0.log" Jul 16 00:15:59.722447 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:59.722423 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d8f6k_2f64cab3-104a-44f6-b125-ba458f54f058/kube-rbac-proxy-ovn-metrics/0.log" Jul 16 00:15:59.742182 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:59.742162 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d8f6k_2f64cab3-104a-44f6-b125-ba458f54f058/northd/0.log" Jul 16 00:15:59.761842 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:59.761829 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d8f6k_2f64cab3-104a-44f6-b125-ba458f54f058/nbdb/0.log" Jul 16 00:15:59.782473 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:59.782453 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d8f6k_2f64cab3-104a-44f6-b125-ba458f54f058/sbdb/0.log" Jul 16 00:15:59.841331 ip-10-0-128-197 kubenswrapper[2563]: I0716 00:15:59.841306 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d8f6k_2f64cab3-104a-44f6-b125-ba458f54f058/ovnkube-controller/0.log"