Jul 29 18:34:23.236960 ip-10-0-140-205 systemd[1]: Starting Kubernetes Kubelet... Jul 29 18:34:23.641575 ip-10-0-140-205 kubenswrapper[2560]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 29 18:34:23.641575 ip-10-0-140-205 kubenswrapper[2560]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 29 18:34:23.641575 ip-10-0-140-205 kubenswrapper[2560]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 29 18:34:23.641575 ip-10-0-140-205 kubenswrapper[2560]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 29 18:34:23.641575 ip-10-0-140-205 kubenswrapper[2560]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 29 18:34:23.644253 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.644159 2560 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 29 18:34:23.651315 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.651291 2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 29 18:34:23.651315 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.651309 2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 29 18:34:23.651315 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.651313 2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 29 18:34:23.651315 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.651316 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 29 18:34:23.651315 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.651319 2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 29 18:34:23.651315 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.651321 2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 29 18:34:23.651315 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.651324 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 29 18:34:23.651315 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.651327 2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 29 18:34:23.651605 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.651330 2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 29 18:34:23.651605 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.651333 2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 29 18:34:23.651605 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.651336 2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 29 18:34:23.651605 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.651338 2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 29 18:34:23.651605 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.651341 2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 29 18:34:23.651605 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.651343 2560 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 29 18:34:23.651605 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.651348 2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 29 18:34:23.651605 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.651351 2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 29 18:34:23.651605 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.651355 2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 29 18:34:23.651605 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.651360 2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 29 18:34:23.651605 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.651363 2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 29 18:34:23.651605 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.651366 2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 29 18:34:23.651605 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.651369 2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 29 18:34:23.651605 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.651373 2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 29 18:34:23.651605 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.651375 2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 29 18:34:23.651605 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.651378 2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 29 18:34:23.651605 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.651381 2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 29 18:34:23.651605 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.651385 2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 29 18:34:23.652079 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.651388 2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 29 18:34:23.652079 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.651391 2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 29 18:34:23.652079 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.651394 2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 29 18:34:23.652079 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.651396 2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 29 18:34:23.652079 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.651399 2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 29 18:34:23.652079 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.651402 2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 29 18:34:23.652079 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.651404 2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 29 18:34:23.652079 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.651407 2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 29 18:34:23.652079 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.651409 2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 29 18:34:23.652079 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.651412 2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 29 18:34:23.652079 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.651414 2560 feature_gate.go:328] unrecognized feature gate: Example2 Jul 29 18:34:23.652079 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.651419 2560 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 29 18:34:23.652079 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.651421 2560 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 29 18:34:23.652079 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.651424 2560 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 29 18:34:23.652079 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.651427 2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 29 18:34:23.652079 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.651430 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 29 18:34:23.652079 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.651433 2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 29 18:34:23.652079 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.651436 2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 29 18:34:23.652079 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.651438 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 29 18:34:23.652079 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.651441 2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 29 18:34:23.652621 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.651443 2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 29 18:34:23.652621 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.651446 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 29 18:34:23.652621 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.651448 2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 29 18:34:23.652621 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.651451 2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 29 18:34:23.652621 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.651454 2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 29 18:34:23.652621 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.651457 2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 29 18:34:23.652621 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.651459 2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 29 18:34:23.652621 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.651462 2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 29 18:34:23.652621 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.651465 2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 29 18:34:23.652621 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.651468 2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 29 18:34:23.652621 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.651471 2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 29 18:34:23.652621 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.651475 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 29 18:34:23.652621 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.651478 2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 29 18:34:23.652621 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.651481 2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 29 18:34:23.652621 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.651483 2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 29 18:34:23.652621 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.651486 2560 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 29 18:34:23.652621 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.651488 2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 29 18:34:23.652621 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.651491 2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 29 18:34:23.652621 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.651493 2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 29 18:34:23.653101 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.651496 2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 29 18:34:23.653101 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.651499 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 29 18:34:23.653101 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.651501 2560 feature_gate.go:328] unrecognized feature gate: Example Jul 29 18:34:23.653101 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.651504 2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 29 18:34:23.653101 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.651506 2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 29 18:34:23.653101 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.651509 2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 29 18:34:23.653101 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.651512 2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 29 18:34:23.653101 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.651514 2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 29 18:34:23.653101 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.651519 2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 29 18:34:23.653101 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.651522 2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 29 18:34:23.653101 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.651524 2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 29 18:34:23.653101 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.651527 2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 29 18:34:23.653101 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.651529 2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 29 18:34:23.653101 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.651534 2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 29 18:34:23.653101 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.651536 2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 29 18:34:23.653101 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.651539 2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 29 18:34:23.653101 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.651542 2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 29 18:34:23.653101 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.651544 2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 29 18:34:23.653101 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.651547 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 29 18:34:23.653101 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.651550 2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 29 18:34:23.653608 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.651552 2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 29 18:34:23.653608 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.651555 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 29 18:34:23.653608 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.651557 2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 29 18:34:23.653608 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.651560 2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 29 18:34:23.653608 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.651562 2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 29 18:34:23.653608 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.651565 2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 29 18:34:23.653608 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.651567 2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 29 18:34:23.653608 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.651570 2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 29 18:34:23.653608 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.651572 2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 29 18:34:23.653608 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.651575 2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 29 18:34:23.653608 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.651578 2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 29 18:34:23.653608 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.651580 2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 29 18:34:23.653608 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.651583 2560 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 29 18:34:23.653608 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.651585 2560 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 29 18:34:23.653608 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.652320 2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 29 18:34:23.653608 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.652327 2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 29 18:34:23.653608 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.652330 2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 29 18:34:23.653608 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.652333 2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 29 18:34:23.653608 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.652337 2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 29 18:34:23.653608 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.652339 2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 29 18:34:23.654127 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.652342 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 29 18:34:23.654127 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.652345 2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 29 18:34:23.654127 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.652348 2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 29 18:34:23.654127 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.652351 2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 29 18:34:23.654127 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.652353 2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 29 18:34:23.654127 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.652356 2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 29 18:34:23.654127 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.652358 2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 29 18:34:23.654127 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.652361 2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 29 18:34:23.654127 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.652364 2560 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 29 18:34:23.654127 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.652366 2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 29 18:34:23.654127 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.652369 2560 feature_gate.go:328] unrecognized feature gate: Example2 Jul 29 18:34:23.654127 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.652372 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 29 18:34:23.654127 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.652375 2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 29 18:34:23.654127 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.652377 2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 29 18:34:23.654127 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.652379 2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 29 18:34:23.654127 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.652382 2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 29 18:34:23.654127 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.652384 2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 29 18:34:23.654127 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.652386 2560 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 29 18:34:23.654127 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.652389 2560 feature_gate.go:328] unrecognized feature gate: Example Jul 29 18:34:23.654127 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.652392 2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 29 18:34:23.654652 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.652394 2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 29 18:34:23.654652 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.652396 2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 29 18:34:23.654652 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.652399 2560 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 29 18:34:23.654652 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.652401 2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 29 18:34:23.654652 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.652404 2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 29 18:34:23.654652 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.652406 2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 29 18:34:23.654652 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.652409 2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 29 18:34:23.654652 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.652411 2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 29 18:34:23.654652 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.652414 2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 29 18:34:23.654652 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.652416 2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 29 18:34:23.654652 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.652420 2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 29 18:34:23.654652 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.652422 2560 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 29 18:34:23.654652 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.652425 2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 29 18:34:23.654652 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.652428 2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 29 18:34:23.654652 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.652430 2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 29 18:34:23.654652 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.652433 2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 29 18:34:23.654652 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.652435 2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 29 18:34:23.654652 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.652439 2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 29 18:34:23.654652 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.652442 2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 29 18:34:23.655130 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.652444 2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 29 18:34:23.655130 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.652447 2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 29 18:34:23.655130 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.652449 2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 29 18:34:23.655130 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.652452 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 29 18:34:23.655130 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.652454 2560 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 29 18:34:23.655130 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.652459 2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 29 18:34:23.655130 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.652462 2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 29 18:34:23.655130 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.652465 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 29 18:34:23.655130 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.652468 2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 29 18:34:23.655130 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.652470 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 29 18:34:23.655130 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.652472 2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 29 18:34:23.655130 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.652475 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 29 18:34:23.655130 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.652477 2560 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 29 18:34:23.655130 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.652479 2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 29 18:34:23.655130 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.652482 2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 29 18:34:23.655130 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.652484 2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 29 18:34:23.655130 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.652486 2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 29 18:34:23.655130 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.652489 2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 29 18:34:23.655130 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.652491 2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 29 18:34:23.655130 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.652494 2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 29 18:34:23.655611 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.652496 2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 29 18:34:23.655611 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.652498 2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 29 18:34:23.655611 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.652501 2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 29 18:34:23.655611 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.652503 2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 29 18:34:23.655611 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.652506 2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 29 18:34:23.655611 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.652509 2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 29 18:34:23.655611 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.652512 2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 29 18:34:23.655611 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.652514 2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 29 18:34:23.655611 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.652517 2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 29 18:34:23.655611 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.652519 2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 29 18:34:23.655611 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.652522 2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 29 18:34:23.655611 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.652526 2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 29 18:34:23.655611 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.652529 2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 29 18:34:23.655611 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.652532 2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 29 18:34:23.655611 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.652535 2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 29 18:34:23.655611 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.652537 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 29 18:34:23.655611 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.652540 2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 29 18:34:23.655611 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.652542 2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 29 18:34:23.655611 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.652545 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 29 18:34:23.656112 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.652547 2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 29 18:34:23.656112 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.652550 2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 29 18:34:23.656112 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.652552 2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 29 18:34:23.656112 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.652554 2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 29 18:34:23.656112 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.652557 2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 29 18:34:23.656112 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.652559 2560 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 29 18:34:23.656112 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.652561 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 29 18:34:23.656112 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.652564 2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 29 18:34:23.656112 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.652566 2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 29 18:34:23.656112 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.652571 2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 29 18:34:23.656112 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.652574 2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 29 18:34:23.656112 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.652577 2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 29 18:34:23.656112 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.652580 2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 29 18:34:23.656112 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.652583 2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 29 18:34:23.656112 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.652585 2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 29 18:34:23.656112 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.652971 2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 29 18:34:23.656112 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.652977 2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 29 18:34:23.656112 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.652979 2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 29 18:34:23.656112 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.652983 2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 29 18:34:23.656579 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.652986 2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 29 18:34:23.656579 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.652988 2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 29 18:34:23.656579 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.652991 2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 29 18:34:23.656579 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.652993 2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 29 18:34:23.656579 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.652996 2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 29 18:34:23.656579 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.652999 2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 29 18:34:23.656579 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.653001 2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 29 18:34:23.656579 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.653004 2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 29 18:34:23.656579 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.653006 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 29 18:34:23.656579 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.653009 2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 29 18:34:23.656579 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.653011 2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 29 18:34:23.656579 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.653014 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 29 18:34:23.656579 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.653017 2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 29 18:34:23.656579 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.653019 2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 29 18:34:23.656579 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.653022 2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 29 18:34:23.656579 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.653024 2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 29 18:34:23.656579 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.653027 2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 29 18:34:23.656579 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.653029 2560 feature_gate.go:328] unrecognized feature gate: Example Jul 29 18:34:23.656579 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.653032 2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 29 18:34:23.656579 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.653034 2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 29 18:34:23.657069 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.653038 2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 29 18:34:23.657069 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.653040 2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 29 18:34:23.657069 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.653042 2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 29 18:34:23.657069 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.653045 2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 29 18:34:23.657069 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.653047 2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 29 18:34:23.657069 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.653050 2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 29 18:34:23.657069 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.653052 2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 29 18:34:23.657069 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.653054 2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 29 18:34:23.657069 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.653057 2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 29 18:34:23.657069 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.653060 2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 29 18:34:23.657069 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.653063 2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 29 18:34:23.657069 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.653066 2560 feature_gate.go:328] unrecognized feature gate: Example2 Jul 29 18:34:23.657069 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.653068 2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 29 18:34:23.657069 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.653070 2560 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 29 18:34:23.657069 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.653073 2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 29 18:34:23.657069 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.653075 2560 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 29 18:34:23.657069 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.653077 2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 29 18:34:23.657069 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.653080 2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 29 18:34:23.657069 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.653083 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 29 18:34:23.657069 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.653085 2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 29 18:34:23.657549 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.653088 2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 29 18:34:23.657549 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.653090 2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 29 18:34:23.657549 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.653092 2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 29 18:34:23.657549 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.653095 2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 29 18:34:23.657549 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.653097 2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 29 18:34:23.657549 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.653100 2560 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 29 18:34:23.657549 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.653103 2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 29 18:34:23.657549 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.653105 2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 29 18:34:23.657549 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.653108 2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 29 18:34:23.657549 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.653110 2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 29 18:34:23.657549 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.653113 2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 29 18:34:23.657549 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.653116 2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 29 18:34:23.657549 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.653119 2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 29 18:34:23.657549 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.653122 2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 29 18:34:23.657549 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.653124 2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 29 18:34:23.657549 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.653126 2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 29 18:34:23.657549 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.653130 2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 29 18:34:23.657549 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.653134 2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 29 18:34:23.657549 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.653137 2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 29 18:34:23.658020 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.653140 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 29 18:34:23.658020 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.653143 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 29 18:34:23.658020 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.653146 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 29 18:34:23.658020 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.653148 2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 29 18:34:23.658020 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.653151 2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 29 18:34:23.658020 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.653153 2560 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 29 18:34:23.658020 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.653156 2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 29 18:34:23.658020 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.653158 2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 29 18:34:23.658020 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.653161 2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 29 18:34:23.658020 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.653165 2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 29 18:34:23.658020 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.653168 2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 29 18:34:23.658020 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.653171 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 29 18:34:23.658020 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.653174 2560 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 29 18:34:23.658020 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.653177 2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 29 18:34:23.658020 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.653179 2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 29 18:34:23.658020 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.653182 2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 29 18:34:23.658020 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.653184 2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 29 18:34:23.658020 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.653187 2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 29 18:34:23.658020 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.653191 2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 29 18:34:23.658020 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.653194 2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 29 18:34:23.658524 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.653197 2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 29 18:34:23.658524 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.653201 2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 29 18:34:23.658524 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.653203 2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 29 18:34:23.658524 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.653207 2560 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 29 18:34:23.658524 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.653210 2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 29 18:34:23.658524 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.653213 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 29 18:34:23.658524 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.653215 2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 29 18:34:23.658524 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.653218 2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 29 18:34:23.658524 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.653221 2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 29 18:34:23.658524 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.653223 2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 29 18:34:23.658524 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.653226 2560 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 29 18:34:23.658524 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.653228 2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 29 18:34:23.658524 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.653230 2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 29 18:34:23.658524 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.653233 2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 29 18:34:23.658524 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.653235 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 29 18:34:23.658524 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.653238 2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 29 18:34:23.658524 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.653310 2560 flags.go:64] FLAG: --address="0.0.0.0" Jul 29 18:34:23.658524 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.653318 2560 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 29 18:34:23.658524 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.653324 2560 flags.go:64] FLAG: --anonymous-auth="true" Jul 29 18:34:23.658524 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.653329 2560 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 29 18:34:23.658524 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.653333 2560 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 29 18:34:23.659053 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.653336 2560 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 29 18:34:23.659053 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.653340 2560 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 29 18:34:23.659053 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.653345 2560 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 29 18:34:23.659053 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.653348 2560 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 29 18:34:23.659053 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.653351 2560 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 29 18:34:23.659053 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.653355 2560 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 29 18:34:23.659053 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.653358 2560 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 29 18:34:23.659053 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.653361 2560 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 29 18:34:23.659053 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.653364 2560 flags.go:64] FLAG: --cgroup-root="" Jul 29 18:34:23.659053 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.653366 2560 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 29 18:34:23.659053 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.653369 2560 flags.go:64] FLAG: --client-ca-file="" Jul 29 18:34:23.659053 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.653373 2560 flags.go:64] FLAG: --cloud-provider="external" Jul 29 18:34:23.659053 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.653375 2560 flags.go:64] FLAG: --cluster-dns="[]" Jul 29 18:34:23.659053 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.653379 2560 flags.go:64] FLAG: --cluster-domain="" Jul 29 18:34:23.659053 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.653386 2560 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 29 18:34:23.659053 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.653389 2560 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 29 18:34:23.659053 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.653393 2560 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 29 18:34:23.659053 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.653396 2560 flags.go:64] FLAG: --container-log-max-files="5" Jul 29 18:34:23.659053 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.653399 2560 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 29 18:34:23.659053 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.653402 2560 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 29 18:34:23.659053 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.653405 2560 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 29 18:34:23.659053 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.653408 2560 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 29 18:34:23.659053 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.653411 2560 flags.go:64] FLAG: --contention-profiling="false" Jul 29 18:34:23.659053 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.653414 2560 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 29 18:34:23.659615 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.653416 2560 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 29 18:34:23.659615 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.653421 2560 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 29 18:34:23.659615 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.653424 2560 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 29 18:34:23.659615 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.653428 2560 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 29 18:34:23.659615 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.653431 2560 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 29 18:34:23.659615 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.653433 2560 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 29 18:34:23.659615 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.653436 2560 flags.go:64] FLAG: --enable-load-reader="false" Jul 29 18:34:23.659615 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.653439 2560 flags.go:64] FLAG: --enable-server="true" Jul 29 18:34:23.659615 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.653442 2560 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 29 18:34:23.659615 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.653445 2560 flags.go:64] FLAG: --event-burst="100" Jul 29 18:34:23.659615 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.653448 2560 flags.go:64] FLAG: --event-qps="50" Jul 29 18:34:23.659615 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.653451 2560 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 29 18:34:23.659615 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.653454 2560 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 29 18:34:23.659615 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.653457 2560 flags.go:64] FLAG: --eviction-hard="" Jul 29 18:34:23.659615 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.653460 2560 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 29 18:34:23.659615 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.653463 2560 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 29 18:34:23.659615 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.653466 2560 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 29 18:34:23.659615 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.653468 2560 flags.go:64] FLAG: --eviction-soft="" Jul 29 18:34:23.659615 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.653471 2560 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 29 18:34:23.659615 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.653474 2560 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 29 18:34:23.659615 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.653477 2560 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 29 18:34:23.659615 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.653480 2560 flags.go:64] FLAG: --experimental-mounter-path="" Jul 29 18:34:23.659615 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.653484 2560 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 29 18:34:23.659615 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.653487 2560 flags.go:64] FLAG: --fail-swap-on="true" Jul 29 18:34:23.659615 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.653490 2560 flags.go:64] FLAG: --feature-gates="" Jul 29 18:34:23.660317 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.653493 2560 flags.go:64] FLAG: --file-check-frequency="20s" Jul 29 18:34:23.660317 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.653496 2560 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 29 18:34:23.660317 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.653499 2560 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 29 18:34:23.660317 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.653502 2560 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 29 18:34:23.660317 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.653505 2560 flags.go:64] FLAG: --healthz-port="10248" Jul 29 18:34:23.660317 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.653507 2560 flags.go:64] FLAG: --help="false" Jul 29 18:34:23.660317 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.653510 2560 flags.go:64] FLAG: --hostname-override="ip-10-0-140-205.ec2.internal" Jul 29 18:34:23.660317 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.653513 2560 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 29 18:34:23.660317 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.653517 2560 flags.go:64] FLAG: --http-check-frequency="20s" Jul 29 18:34:23.660317 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.653520 2560 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 29 18:34:23.660317 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.653523 2560 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 29 18:34:23.660317 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.653526 2560 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 29 18:34:23.660317 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.653529 2560 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 29 18:34:23.660317 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.653532 2560 flags.go:64] FLAG: --image-service-endpoint="" Jul 29 18:34:23.660317 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.653534 2560 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 29 18:34:23.660317 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.653537 2560 flags.go:64] FLAG: --kube-api-burst="100" Jul 29 18:34:23.660317 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.653540 2560 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 29 18:34:23.660317 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.653543 2560 flags.go:64] FLAG: --kube-api-qps="50" Jul 29 18:34:23.660317 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.653545 2560 flags.go:64] FLAG: --kube-reserved="" Jul 29 18:34:23.660317 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.653548 2560 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 29 18:34:23.660317 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.653551 2560 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 29 18:34:23.660317 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.653554 2560 flags.go:64] FLAG: --kubelet-cgroups="" Jul 29 18:34:23.660317 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.653556 2560 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 29 18:34:23.660317 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.653559 2560 flags.go:64] FLAG: --lock-file="" Jul 29 18:34:23.660990 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.653562 2560 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 29 18:34:23.660990 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.653564 2560 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 29 18:34:23.660990 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.653567 2560 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 29 18:34:23.660990 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.653572 2560 flags.go:64] FLAG: --log-json-split-stream="false" Jul 29 18:34:23.660990 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.653575 2560 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 29 18:34:23.660990 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.653579 2560 flags.go:64] FLAG: --log-text-split-stream="false" Jul 29 18:34:23.660990 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.653582 2560 flags.go:64] FLAG: --logging-format="text" Jul 29 18:34:23.660990 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.653584 2560 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 29 18:34:23.660990 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.653588 2560 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 29 18:34:23.660990 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.653591 2560 flags.go:64] FLAG: --manifest-url="" Jul 29 18:34:23.660990 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.653593 2560 flags.go:64] FLAG: --manifest-url-header="" Jul 29 18:34:23.660990 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.653597 2560 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 29 18:34:23.660990 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.653600 2560 flags.go:64] FLAG: --max-open-files="1000000" Jul 29 18:34:23.660990 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.653604 2560 flags.go:64] FLAG: --max-pods="110" Jul 29 18:34:23.660990 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.653607 2560 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 29 18:34:23.660990 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.653610 2560 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 29 18:34:23.660990 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.653614 2560 flags.go:64] FLAG: --memory-manager-policy="None" Jul 29 18:34:23.660990 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.653617 2560 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 29 18:34:23.660990 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.653620 2560 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 29 18:34:23.660990 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.653623 2560 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 29 18:34:23.660990 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.653626 2560 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 29 18:34:23.660990 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.653632 2560 flags.go:64] FLAG: --node-status-max-images="50" Jul 29 18:34:23.660990 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.653635 2560 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 29 18:34:23.660990 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.653638 2560 flags.go:64] FLAG: --oom-score-adj="-999" Jul 29 18:34:23.660990 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.653641 2560 flags.go:64] FLAG: --pod-cidr="" Jul 29 18:34:23.661577 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.653644 2560 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb" Jul 29 18:34:23.661577 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.653649 2560 flags.go:64] FLAG: --pod-manifest-path="" Jul 29 18:34:23.661577 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.653652 2560 flags.go:64] FLAG: --pod-max-pids="-1" Jul 29 18:34:23.661577 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.653655 2560 flags.go:64] FLAG: --pods-per-core="0" Jul 29 18:34:23.661577 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.653658 2560 flags.go:64] FLAG: --port="10250" Jul 29 18:34:23.661577 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.653661 2560 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 29 18:34:23.661577 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.653677 2560 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-02dc90cd4625b8c67" Jul 29 18:34:23.661577 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.653681 2560 flags.go:64] FLAG: --qos-reserved="" Jul 29 18:34:23.661577 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.653683 2560 flags.go:64] FLAG: --read-only-port="10255" Jul 29 18:34:23.661577 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.653687 2560 flags.go:64] FLAG: --register-node="true" Jul 29 18:34:23.661577 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.653689 2560 flags.go:64] FLAG: --register-with-taints="" Jul 29 18:34:23.661577 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.653693 2560 flags.go:64] FLAG: --registry-burst="10" Jul 29 18:34:23.661577 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.653696 2560 flags.go:64] FLAG: --registry-qps="5" Jul 29 18:34:23.661577 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.653700 2560 flags.go:64] FLAG: --reserved-cpus="" Jul 29 18:34:23.661577 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.653703 2560 flags.go:64] FLAG: --reserved-memory="" Jul 29 18:34:23.661577 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.653706 2560 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 29 18:34:23.661577 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.653709 2560 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 29 18:34:23.661577 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.653712 2560 flags.go:64] FLAG: --rotate-certificates="false" Jul 29 18:34:23.661577 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.653715 2560 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 29 18:34:23.661577 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.653718 2560 flags.go:64] FLAG: --runonce="false" Jul 29 18:34:23.661577 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.653720 2560 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 29 18:34:23.661577 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.653723 2560 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 29 18:34:23.661577 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.653726 2560 flags.go:64] FLAG: --seccomp-default="false" Jul 29 18:34:23.661577 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.653729 2560 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 29 18:34:23.661577 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.653732 2560 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 29 18:34:23.662201 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.653735 2560 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 29 18:34:23.662201 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.653738 2560 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 29 18:34:23.662201 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.653741 2560 flags.go:64] FLAG: --storage-driver-password="root" Jul 29 18:34:23.662201 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.653744 2560 flags.go:64] FLAG: --storage-driver-secure="false" Jul 29 18:34:23.662201 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.653746 2560 flags.go:64] FLAG: --storage-driver-table="stats" Jul 29 18:34:23.662201 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.653749 2560 flags.go:64] FLAG: --storage-driver-user="root" Jul 29 18:34:23.662201 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.653752 2560 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 29 18:34:23.662201 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.653755 2560 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 29 18:34:23.662201 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.653758 2560 flags.go:64] FLAG: --system-cgroups="" Jul 29 18:34:23.662201 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.653760 2560 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 29 18:34:23.662201 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.653766 2560 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 29 18:34:23.662201 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.653768 2560 flags.go:64] FLAG: --tls-cert-file="" Jul 29 18:34:23.662201 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.653771 2560 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 29 18:34:23.662201 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.653774 2560 flags.go:64] FLAG: --tls-min-version="" Jul 29 18:34:23.662201 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.653777 2560 flags.go:64] FLAG: --tls-private-key-file="" Jul 29 18:34:23.662201 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.653780 2560 flags.go:64] FLAG: --topology-manager-policy="none" Jul 29 18:34:23.662201 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.653782 2560 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 29 18:34:23.662201 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.653785 2560 flags.go:64] FLAG: --topology-manager-scope="container" Jul 29 18:34:23.662201 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.653788 2560 flags.go:64] FLAG: --v="2" Jul 29 18:34:23.662201 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.653792 2560 flags.go:64] FLAG: --version="false" Jul 29 18:34:23.662201 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.653797 2560 flags.go:64] FLAG: --vmodule="" Jul 29 18:34:23.662201 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.653800 2560 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 29 18:34:23.662201 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.653803 2560 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 29 18:34:23.662201 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.653933 2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 29 18:34:23.662201 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.653936 2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 29 18:34:23.662808 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.653939 2560 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 29 18:34:23.662808 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.653942 2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 29 18:34:23.662808 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.653945 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 29 18:34:23.662808 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.653948 2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 29 18:34:23.662808 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.653951 2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 29 18:34:23.662808 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.653954 2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 29 18:34:23.662808 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.653957 2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 29 18:34:23.662808 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.653959 2560 feature_gate.go:328] unrecognized feature gate: Example2 Jul 29 18:34:23.662808 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.653962 2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 29 18:34:23.662808 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.653965 2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 29 18:34:23.662808 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.653968 2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 29 18:34:23.662808 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.653971 2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 29 18:34:23.662808 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.653973 2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 29 18:34:23.662808 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.653976 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 29 18:34:23.662808 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.653978 2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 29 18:34:23.662808 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.653981 2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 29 18:34:23.662808 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.653984 2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 29 18:34:23.662808 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.653986 2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 29 18:34:23.662808 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.653989 2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 29 18:34:23.662808 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.653991 2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 29 18:34:23.663279 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.653994 2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 29 18:34:23.663279 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.653996 2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 29 18:34:23.663279 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.653999 2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 29 18:34:23.663279 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.654001 2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 29 18:34:23.663279 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.654004 2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 29 18:34:23.663279 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.654006 2560 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 29 18:34:23.663279 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.654009 2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 29 18:34:23.663279 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.654013 2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 29 18:34:23.663279 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.654016 2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 29 18:34:23.663279 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.654018 2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 29 18:34:23.663279 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.654021 2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 29 18:34:23.663279 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.654023 2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 29 18:34:23.663279 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.654025 2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 29 18:34:23.663279 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.654028 2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 29 18:34:23.663279 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.654030 2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 29 18:34:23.663279 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.654034 2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 29 18:34:23.663279 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.654037 2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 29 18:34:23.663279 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.654040 2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 29 18:34:23.663279 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.654043 2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 29 18:34:23.663279 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.654045 2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 29 18:34:23.663797 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.654048 2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 29 18:34:23.663797 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.654050 2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 29 18:34:23.663797 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.654053 2560 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 29 18:34:23.663797 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.654055 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 29 18:34:23.663797 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.654058 2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 29 18:34:23.663797 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.654060 2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 29 18:34:23.663797 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.654062 2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 29 18:34:23.663797 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.654065 2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 29 18:34:23.663797 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.654067 2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 29 18:34:23.663797 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.654070 2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 29 18:34:23.663797 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.654073 2560 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 29 18:34:23.663797 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.654075 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 29 18:34:23.663797 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.654078 2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 29 18:34:23.663797 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.654080 2560 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 29 18:34:23.663797 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.654083 2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 29 18:34:23.663797 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.654085 2560 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 29 18:34:23.663797 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.654088 2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 29 18:34:23.663797 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.654090 2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 29 18:34:23.663797 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.654093 2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 29 18:34:23.663797 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.654096 2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 29 18:34:23.664286 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.654099 2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 29 18:34:23.664286 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.654101 2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 29 18:34:23.664286 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.654104 2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 29 18:34:23.664286 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.654106 2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 29 18:34:23.664286 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.654109 2560 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 29 18:34:23.664286 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.654111 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 29 18:34:23.664286 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.654114 2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 29 18:34:23.664286 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.654116 2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 29 18:34:23.664286 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.654119 2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 29 18:34:23.664286 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.654121 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 29 18:34:23.664286 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.654124 2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 29 18:34:23.664286 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.654126 2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 29 18:34:23.664286 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.654129 2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 29 18:34:23.664286 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.654131 2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 29 18:34:23.664286 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.654134 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 29 18:34:23.664286 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.654137 2560 feature_gate.go:328] unrecognized feature gate: Example Jul 29 18:34:23.664286 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.654140 2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 29 18:34:23.664286 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.654142 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 29 18:34:23.664286 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.654145 2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 29 18:34:23.664286 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.654147 2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 29 18:34:23.664816 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.654150 2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 29 18:34:23.664816 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.654152 2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 29 18:34:23.664816 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.654155 2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 29 18:34:23.664816 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.654157 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 29 18:34:23.664816 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.654160 2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 29 18:34:23.664816 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.654162 2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 29 18:34:23.664816 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.654166 2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 29 18:34:23.664816 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.654169 2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 29 18:34:23.664816 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.654171 2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 29 18:34:23.664816 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.654174 2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 29 18:34:23.664816 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.654176 2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 29 18:34:23.664816 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.654181 2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 29 18:34:23.664816 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.654185 2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 29 18:34:23.664816 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.654187 2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 29 18:34:23.664816 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.654191 2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 29 18:34:23.664816 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.654194 2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 29 18:34:23.664816 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.654197 2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 29 18:34:23.665229 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.654844 2560 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 29 18:34:23.665229 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.661703 2560 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 29 18:34:23.665229 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.661718 2560 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 29 18:34:23.665229 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.661740 2560 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 29 18:34:23.665229 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.661746 2560 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 29 18:34:23.665229 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.661804 2560 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 29 18:34:23.665229 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.661808 2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 29 18:34:23.665229 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.661811 2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 29 18:34:23.665229 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.661814 2560 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 29 18:34:23.665229 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.661817 2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 29 18:34:23.665229 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.661820 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 29 18:34:23.665229 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.661823 2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 29 18:34:23.665229 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.661826 2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 29 18:34:23.665229 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.661829 2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 29 18:34:23.665229 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.661832 2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 29 18:34:23.665595 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.661834 2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 29 18:34:23.665595 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.661837 2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 29 18:34:23.665595 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.661839 2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 29 18:34:23.665595 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.661842 2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 29 18:34:23.665595 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.661844 2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 29 18:34:23.665595 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.661847 2560 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 29 18:34:23.665595 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.661849 2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 29 18:34:23.665595 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.661853 2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 29 18:34:23.665595 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.661855 2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 29 18:34:23.665595 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.661858 2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 29 18:34:23.665595 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.661861 2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 29 18:34:23.665595 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.661863 2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 29 18:34:23.665595 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.661866 2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 29 18:34:23.665595 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.661868 2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 29 18:34:23.665595 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.661871 2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 29 18:34:23.665595 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.661873 2560 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 29 18:34:23.665595 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.661876 2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 29 18:34:23.665595 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.661878 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 29 18:34:23.665595 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.661881 2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 29 18:34:23.665595 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.661893 2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 29 18:34:23.666089 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.661896 2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 29 18:34:23.666089 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.661898 2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 29 18:34:23.666089 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.661902 2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 29 18:34:23.666089 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.661904 2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 29 18:34:23.666089 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.661908 2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 29 18:34:23.666089 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.661911 2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 29 18:34:23.666089 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.661914 2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 29 18:34:23.666089 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.661918 2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 29 18:34:23.666089 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.661920 2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 29 18:34:23.666089 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.661923 2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 29 18:34:23.666089 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.661925 2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 29 18:34:23.666089 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.661927 2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 29 18:34:23.666089 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.661930 2560 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 29 18:34:23.666089 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.661933 2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 29 18:34:23.666089 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.661935 2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 29 18:34:23.666089 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.661938 2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 29 18:34:23.666089 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.661940 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 29 18:34:23.666089 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.661943 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 29 18:34:23.666089 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.661945 2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 29 18:34:23.666546 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.661948 2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 29 18:34:23.666546 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.661950 2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 29 18:34:23.666546 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.661952 2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 29 18:34:23.666546 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.661955 2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 29 18:34:23.666546 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.661958 2560 feature_gate.go:328] unrecognized feature gate: Example2 Jul 29 18:34:23.666546 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.661960 2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 29 18:34:23.666546 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.661963 2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 29 18:34:23.666546 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.661965 2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 29 18:34:23.666546 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.661967 2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 29 18:34:23.666546 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.661970 2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 29 18:34:23.666546 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.661972 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 29 18:34:23.666546 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.661975 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 29 18:34:23.666546 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.661979 2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 29 18:34:23.666546 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.661982 2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 29 18:34:23.666546 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.661985 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 29 18:34:23.666546 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.661987 2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 29 18:34:23.666546 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.661990 2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 29 18:34:23.666546 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.661992 2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 29 18:34:23.666546 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.661995 2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 29 18:34:23.666546 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.661997 2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 29 18:34:23.667071 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.662000 2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 29 18:34:23.667071 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.662002 2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 29 18:34:23.667071 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.662006 2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 29 18:34:23.667071 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.662010 2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 29 18:34:23.667071 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.662012 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 29 18:34:23.667071 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.662015 2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 29 18:34:23.667071 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.662018 2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 29 18:34:23.667071 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.662020 2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 29 18:34:23.667071 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.662022 2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 29 18:34:23.667071 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.662025 2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 29 18:34:23.667071 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.662027 2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 29 18:34:23.667071 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.662030 2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 29 18:34:23.667071 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.662032 2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 29 18:34:23.667071 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.662035 2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 29 18:34:23.667071 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.662038 2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 29 18:34:23.667071 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.662041 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 29 18:34:23.667071 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.662044 2560 feature_gate.go:328] unrecognized feature gate: Example Jul 29 18:34:23.667071 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.662047 2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 29 18:34:23.667071 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.662049 2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 29 18:34:23.667519 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.662052 2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 29 18:34:23.667519 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.662054 2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 29 18:34:23.667519 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.662057 2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 29 18:34:23.667519 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.662059 2560 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 29 18:34:23.667519 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.662062 2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 29 18:34:23.667519 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.662065 2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 29 18:34:23.667519 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.662068 2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 29 18:34:23.667519 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.662070 2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 29 18:34:23.667519 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.662073 2560 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 29 18:34:23.667519 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.662075 2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 29 18:34:23.667519 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.662078 2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 29 18:34:23.667519 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.662083 2560 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 29 18:34:23.667519 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.662191 2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 29 18:34:23.667519 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.662197 2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 29 18:34:23.667519 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.662200 2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 29 18:34:23.667903 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.662204 2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 29 18:34:23.667903 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.662208 2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 29 18:34:23.667903 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.662210 2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 29 18:34:23.667903 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.662213 2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 29 18:34:23.667903 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.662216 2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 29 18:34:23.667903 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.662219 2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 29 18:34:23.667903 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.662222 2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 29 18:34:23.667903 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.662224 2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 29 18:34:23.667903 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.662226 2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 29 18:34:23.667903 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.662229 2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 29 18:34:23.667903 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.662231 2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 29 18:34:23.667903 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.662234 2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 29 18:34:23.667903 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.662236 2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 29 18:34:23.667903 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.662239 2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 29 18:34:23.667903 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.662242 2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 29 18:34:23.667903 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.662245 2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 29 18:34:23.667903 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.662248 2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 29 18:34:23.667903 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.662251 2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 29 18:34:23.668348 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.662254 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 29 18:34:23.668348 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.662256 2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 29 18:34:23.668348 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.662259 2560 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 29 18:34:23.668348 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.662262 2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 29 18:34:23.668348 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.662265 2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 29 18:34:23.668348 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.662267 2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 29 18:34:23.668348 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.662270 2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 29 18:34:23.668348 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.662276 2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 29 18:34:23.668348 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.662279 2560 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 29 18:34:23.668348 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.662281 2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 29 18:34:23.668348 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.662284 2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 29 18:34:23.668348 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.662286 2560 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 29 18:34:23.668348 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.662288 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 29 18:34:23.668348 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.662291 2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 29 18:34:23.668348 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.662293 2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 29 18:34:23.668348 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.662295 2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 29 18:34:23.668348 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.662298 2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 29 18:34:23.668348 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.662300 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 29 18:34:23.668348 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.662303 2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 29 18:34:23.668348 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.662305 2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 29 18:34:23.668883 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.662307 2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 29 18:34:23.668883 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.662310 2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 29 18:34:23.668883 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.662312 2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 29 18:34:23.668883 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.662314 2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 29 18:34:23.668883 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.662317 2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 29 18:34:23.668883 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.662319 2560 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 29 18:34:23.668883 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.662322 2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 29 18:34:23.668883 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.662324 2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 29 18:34:23.668883 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.662327 2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 29 18:34:23.668883 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.662329 2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 29 18:34:23.668883 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.662332 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 29 18:34:23.668883 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.662334 2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 29 18:34:23.668883 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.662337 2560 feature_gate.go:328] unrecognized feature gate: Example2 Jul 29 18:34:23.668883 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.662339 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 29 18:34:23.668883 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.662341 2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 29 18:34:23.668883 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.662344 2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 29 18:34:23.668883 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.662346 2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 29 18:34:23.668883 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.662349 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 29 18:34:23.668883 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.662351 2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 29 18:34:23.668883 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.662354 2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 29 18:34:23.669366 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.662357 2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 29 18:34:23.669366 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.662359 2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 29 18:34:23.669366 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.662362 2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 29 18:34:23.669366 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.662364 2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 29 18:34:23.669366 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.662366 2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 29 18:34:23.669366 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.662369 2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 29 18:34:23.669366 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.662371 2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 29 18:34:23.669366 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.662374 2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 29 18:34:23.669366 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.662376 2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 29 18:34:23.669366 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.662378 2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 29 18:34:23.669366 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.662381 2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 29 18:34:23.669366 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.662383 2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 29 18:34:23.669366 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.662385 2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 29 18:34:23.669366 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.662388 2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 29 18:34:23.669366 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.662390 2560 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 29 18:34:23.669366 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.662392 2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 29 18:34:23.669366 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.662395 2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 29 18:34:23.669366 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.662397 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 29 18:34:23.669366 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.662400 2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 29 18:34:23.669366 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.662402 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 29 18:34:23.669987 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.662404 2560 feature_gate.go:328] unrecognized feature gate: Example Jul 29 18:34:23.669987 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.662407 2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 29 18:34:23.669987 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.662409 2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 29 18:34:23.669987 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.662412 2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 29 18:34:23.669987 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.662414 2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 29 18:34:23.669987 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.662417 2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 29 18:34:23.669987 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.662419 2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 29 18:34:23.669987 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.662421 2560 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 29 18:34:23.669987 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.662424 2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 29 18:34:23.669987 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.662426 2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 29 18:34:23.669987 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.662429 2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 29 18:34:23.669987 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.662431 2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 29 18:34:23.669987 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.662434 2560 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 29 18:34:23.669987 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.662437 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 29 18:34:23.669987 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.662439 2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 29 18:34:23.669987 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.662442 2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 29 18:34:23.669987 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.662444 2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 29 18:34:23.669987 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:23.662447 2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 29 18:34:23.670417 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.662451 2560 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 29 18:34:23.670417 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.662557 2560 server.go:962] "Client rotation is on, will bootstrap in background" Jul 29 18:34:23.670417 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.664594 2560 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 29 18:34:23.670417 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.666095 2560 server.go:1019] "Starting client certificate rotation" Jul 29 18:34:23.670417 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.666187 2560 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 29 18:34:23.670417 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.666220 2560 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 29 18:34:23.670417 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.668702 2560 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 29 18:34:23.671585 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.669831 2560 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 29 18:34:23.681076 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.681061 2560 log.go:25] "Validated CRI v1 runtime API" Jul 29 18:34:23.684170 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.684152 2560 log.go:25] "Validated CRI v1 image API" Jul 29 18:34:23.687256 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.687240 2560 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 29 18:34:23.690794 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.690772 2560 fs.go:135] Filesystem UUIDs: map[585cd725-a907-4a86-8709-35af8e11a2df:/dev/nvme0n1p4 6958d17b-7974-4e01-b210-cd635d5917a0:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2] Jul 29 18:34:23.690868 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.690792 2560 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 29 18:34:23.696000 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.695835 2560 manager.go:216] Machine: {Timestamp:2026-07-29 18:34:23.694474788 +0000 UTC m=+0.337823301 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3065055 MemoryCapacity:32812171264 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec205c6f62df26a5e4403ff42c050f85 SystemUUID:ec205c6f-62df-26a5-e440-3ff42c050f85 BootID:1c09f175-995e-4829-b2b4-2f2144c2c7ce Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406085632 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406085632 Type:vfs Inodes:4005392 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:1b:99:dc:36:c3 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:1b:99:dc:36:c3 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:b6:aa:ee:76:40:0a Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812171264 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 29 18:34:23.696000 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.695991 2560 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 29 18:34:23.696138 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.696090 2560 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 29 18:34:23.697864 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.697845 2560 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 29 18:34:23.699629 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.699602 2560 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 29 18:34:23.699772 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.699631 2560 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-140-205.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 29 18:34:23.700379 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.700369 2560 topology_manager.go:138] "Creating topology manager with none policy" Jul 29 18:34:23.700412 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.700382 2560 container_manager_linux.go:309] "Creating device plugin manager" Jul 29 18:34:23.700412 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.700389 2560 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 29 18:34:23.701182 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.701172 2560 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 29 18:34:23.702161 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.702151 2560 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 29 18:34:23.704123 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.704111 2560 state_mem.go:36] "Initialized new in-memory state store" Jul 29 18:34:23.704247 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.704237 2560 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 29 18:34:23.704280 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.704267 2560 kubelet.go:486] "Attempting to sync node with API server" Jul 29 18:34:23.704309 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.704279 2560 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 29 18:34:23.704309 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.704302 2560 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 29 18:34:23.704360 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.704313 2560 kubelet.go:398] "Adding apiserver pod source" Jul 29 18:34:23.704360 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.704322 2560 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 29 18:34:23.704704 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.704663 2560 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 29 18:34:23.704754 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.704721 2560 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 29 18:34:23.707857 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.707835 2560 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 29 18:34:23.709895 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.709882 2560 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 29 18:34:23.709946 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.709914 2560 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 29 18:34:23.712631 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.712613 2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 29 18:34:23.712729 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.712641 2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 29 18:34:23.712729 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.712653 2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 29 18:34:23.712729 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.712680 2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 29 18:34:23.712729 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.712713 2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 29 18:34:23.712729 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.712723 2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 29 18:34:23.712930 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.712734 2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 29 18:34:23.712930 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.712743 2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 29 18:34:23.712930 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.712753 2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 29 18:34:23.712930 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.712760 2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 29 18:34:23.712930 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.712776 2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 29 18:34:23.712930 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.712782 2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 29 18:34:23.713493 ip-10-0-140-205 kubenswrapper[2560]: E0729 18:34:23.713475 2560 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 29 18:34:23.713493 ip-10-0-140-205 kubenswrapper[2560]: E0729 18:34:23.713483 2560 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-140-205.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 29 18:34:23.713629 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.713619 2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 29 18:34:23.713663 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.713631 2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 29 18:34:23.716767 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.716754 2560 server.go:1268] "Started kubelet" Jul 29 18:34:23.716888 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.716865 2560 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 29 18:34:23.720796 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.720773 2560 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 29 18:34:23.721109 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.721087 2560 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-7wzxk" Jul 29 18:34:23.721471 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.720698 2560 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 29 18:34:23.722029 ip-10-0-140-205 systemd[1]: Started Kubernetes Kubelet. Jul 29 18:34:23.723923 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.723337 2560 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 29 18:34:23.723923 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.723472 2560 server.go:310] "Adding debug handlers to kubelet server" Jul 29 18:34:23.725003 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.724982 2560 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-7wzxk" Jul 29 18:34:23.726428 ip-10-0-140-205 kubenswrapper[2560]: E0729 18:34:23.726410 2560 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 29 18:34:23.727228 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.727211 2560 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-140-205.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 29 18:34:23.728169 ip-10-0-140-205 kubenswrapper[2560]: E0729 18:34:23.727236 2560 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-205.ec2.internal.18c6d66c7b8e6900 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-205.ec2.internal,UID:ip-10-0-140-205.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-140-205.ec2.internal,},FirstTimestamp:2026-07-29 18:34:23.716722944 +0000 UTC m=+0.360071456,LastTimestamp:2026-07-29 18:34:23.716722944 +0000 UTC m=+0.360071456,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-205.ec2.internal,}" Jul 29 18:34:23.732632 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.732616 2560 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 29 18:34:23.732632 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.732627 2560 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 29 18:34:23.732760 ip-10-0-140-205 kubenswrapper[2560]: E0729 18:34:23.732709 2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-205.ec2.internal\" not found" Jul 29 18:34:23.733253 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.733236 2560 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 29 18:34:23.733253 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.733255 2560 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 29 18:34:23.733452 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.733233 2560 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 29 18:34:23.733520 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.733503 2560 factory.go:55] Registering systemd factory Jul 29 18:34:23.733568 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.733524 2560 factory.go:223] Registration of the systemd container factory successfully Jul 29 18:34:23.733568 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.733526 2560 reconstruct.go:97] "Volume reconstruction finished" Jul 29 18:34:23.733653 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.733568 2560 reconciler.go:29] "Reconciler: start to sync state" Jul 29 18:34:23.733735 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.733720 2560 factory.go:153] Registering CRI-O factory Jul 29 18:34:23.733735 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.733734 2560 factory.go:223] Registration of the crio container factory successfully Jul 29 18:34:23.733856 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.733773 2560 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 29 18:34:23.733856 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.733791 2560 factory.go:103] Registering Raw factory Jul 29 18:34:23.733856 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.733800 2560 manager.go:1196] Started watching for new ooms in manager Jul 29 18:34:23.734457 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.734445 2560 manager.go:349] Starting recovery of all containers Jul 29 18:34:23.734927 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.734909 2560 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 29 18:34:23.738047 ip-10-0-140-205 kubenswrapper[2560]: E0729 18:34:23.738023 2560 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-140-205.ec2.internal\" not found" node="ip-10-0-140-205.ec2.internal" Jul 29 18:34:23.744601 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.744583 2560 manager.go:354] Recovery completed Jul 29 18:34:23.749299 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.749194 2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 29 18:34:23.750980 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.750965 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-205.ec2.internal" event="NodeHasSufficientMemory" Jul 29 18:34:23.751106 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.750997 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-205.ec2.internal" event="NodeHasNoDiskPressure" Jul 29 18:34:23.751106 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.751015 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-205.ec2.internal" event="NodeHasSufficientPID" Jul 29 18:34:23.751533 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.751518 2560 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 29 18:34:23.751533 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.751530 2560 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 29 18:34:23.751690 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.751545 2560 state_mem.go:36] "Initialized new in-memory state store" Jul 29 18:34:23.751690 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.751532 2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 29 18:34:23.752418 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.752403 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-205.ec2.internal" event="NodeHasSufficientMemory" Jul 29 18:34:23.752496 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.752430 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-205.ec2.internal" event="NodeHasNoDiskPressure" Jul 29 18:34:23.752496 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.752443 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-205.ec2.internal" event="NodeHasSufficientPID" Jul 29 18:34:23.753233 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.753221 2560 policy_none.go:49] "None policy: Start" Jul 29 18:34:23.753269 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.753238 2560 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 29 18:34:23.753269 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.753248 2560 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 29 18:34:23.754238 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.754228 2560 policy_none.go:47] "Start" Jul 29 18:34:23.789239 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.789223 2560 manager.go:337] "Starting Device Plugin manager" Jul 29 18:34:23.789331 ip-10-0-140-205 kubenswrapper[2560]: E0729 18:34:23.789259 2560 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 29 18:34:23.789331 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.789273 2560 server.go:87] "Starting device plugin registration server" Jul 29 18:34:23.789543 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.789529 2560 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 29 18:34:23.789599 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.789547 2560 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 29 18:34:23.789658 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.789640 2560 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 29 18:34:23.789750 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.789738 2560 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 29 18:34:23.789750 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.789750 2560 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 29 18:34:23.790280 ip-10-0-140-205 kubenswrapper[2560]: E0729 18:34:23.790258 2560 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 29 18:34:23.790377 ip-10-0-140-205 kubenswrapper[2560]: E0729 18:34:23.790308 2560 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-140-205.ec2.internal\" not found" Jul 29 18:34:23.889877 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.889858 2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 29 18:34:23.891173 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.890812 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-205.ec2.internal" event="NodeHasSufficientMemory" Jul 29 18:34:23.891173 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.890837 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-205.ec2.internal" event="NodeHasNoDiskPressure" Jul 29 18:34:23.891173 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.890847 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-205.ec2.internal" event="NodeHasSufficientPID" Jul 29 18:34:23.891173 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.890877 2560 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-140-205.ec2.internal" Jul 29 18:34:23.898891 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.898859 2560 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-140-205.ec2.internal" Jul 29 18:34:23.898891 ip-10-0-140-205 kubenswrapper[2560]: E0729 18:34:23.898879 2560 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-140-205.ec2.internal\": node \"ip-10-0-140-205.ec2.internal\" not found" Jul 29 18:34:23.912645 ip-10-0-140-205 kubenswrapper[2560]: E0729 18:34:23.912626 2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-205.ec2.internal\" not found" Jul 29 18:34:23.928277 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.928256 2560 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 29 18:34:23.929705 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.929686 2560 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 29 18:34:23.929781 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.929720 2560 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 29 18:34:23.929781 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.929738 2560 kubelet.go:2448] "Starting kubelet main sync loop" Jul 29 18:34:23.929781 ip-10-0-140-205 kubenswrapper[2560]: E0729 18:34:23.929766 2560 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 29 18:34:23.932090 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:23.932074 2560 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 29 18:34:24.013587 ip-10-0-140-205 kubenswrapper[2560]: E0729 18:34:24.013563 2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-205.ec2.internal\" not found" Jul 29 18:34:24.030728 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:24.030700 2560 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-205.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-140-205.ec2.internal"] Jul 29 18:34:24.030785 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:24.030766 2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 29 18:34:24.031630 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:24.031607 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-205.ec2.internal" event="NodeHasSufficientMemory" Jul 29 18:34:24.031732 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:24.031636 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-205.ec2.internal" event="NodeHasNoDiskPressure" Jul 29 18:34:24.031732 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:24.031650 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-205.ec2.internal" event="NodeHasSufficientPID" Jul 29 18:34:24.033065 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:24.033054 2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 29 18:34:24.033195 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:24.033180 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-205.ec2.internal" Jul 29 18:34:24.033231 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:24.033211 2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 29 18:34:24.033759 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:24.033743 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-205.ec2.internal" event="NodeHasSufficientMemory" Jul 29 18:34:24.033818 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:24.033771 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-205.ec2.internal" event="NodeHasNoDiskPressure" Jul 29 18:34:24.033818 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:24.033780 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-205.ec2.internal" event="NodeHasSufficientPID" Jul 29 18:34:24.033879 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:24.033743 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-205.ec2.internal" event="NodeHasSufficientMemory" Jul 29 18:34:24.033879 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:24.033849 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-205.ec2.internal" event="NodeHasNoDiskPressure" Jul 29 18:34:24.033879 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:24.033863 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-205.ec2.internal" event="NodeHasSufficientPID" Jul 29 18:34:24.035037 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:24.035020 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-205.ec2.internal" Jul 29 18:34:24.035037 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:24.035030 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0d56f34abf69a77b3a700a4e3019bb67-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-205.ec2.internal\" (UID: \"0d56f34abf69a77b3a700a4e3019bb67\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-205.ec2.internal" Jul 29 18:34:24.035164 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:24.035046 2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 29 18:34:24.035164 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:24.035053 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0d56f34abf69a77b3a700a4e3019bb67-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-205.ec2.internal\" (UID: \"0d56f34abf69a77b3a700a4e3019bb67\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-205.ec2.internal" Jul 29 18:34:24.035164 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:24.035069 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/89287a5239330a896427ad6104c0beff-config\") pod \"kube-apiserver-proxy-ip-10-0-140-205.ec2.internal\" (UID: \"89287a5239330a896427ad6104c0beff\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-205.ec2.internal" Jul 29 18:34:24.035790 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:24.035773 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-205.ec2.internal" event="NodeHasSufficientMemory" Jul 29 18:34:24.035867 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:24.035803 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-205.ec2.internal" event="NodeHasNoDiskPressure" Jul 29 18:34:24.035867 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:24.035816 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-205.ec2.internal" event="NodeHasSufficientPID" Jul 29 18:34:24.060227 ip-10-0-140-205 kubenswrapper[2560]: E0729 18:34:24.060203 2560 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-205.ec2.internal\" not found" node="ip-10-0-140-205.ec2.internal" Jul 29 18:34:24.064405 ip-10-0-140-205 kubenswrapper[2560]: E0729 18:34:24.064387 2560 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-205.ec2.internal\" not found" node="ip-10-0-140-205.ec2.internal" Jul 29 18:34:24.114525 ip-10-0-140-205 kubenswrapper[2560]: E0729 18:34:24.114505 2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-205.ec2.internal\" not found" Jul 29 18:34:24.135158 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:24.135139 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/89287a5239330a896427ad6104c0beff-config\") pod \"kube-apiserver-proxy-ip-10-0-140-205.ec2.internal\" (UID: \"89287a5239330a896427ad6104c0beff\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-205.ec2.internal" Jul 29 18:34:24.135223 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:24.135170 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0d56f34abf69a77b3a700a4e3019bb67-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-205.ec2.internal\" (UID: \"0d56f34abf69a77b3a700a4e3019bb67\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-205.ec2.internal" Jul 29 18:34:24.135223 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:24.135192 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0d56f34abf69a77b3a700a4e3019bb67-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-205.ec2.internal\" (UID: \"0d56f34abf69a77b3a700a4e3019bb67\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-205.ec2.internal" Jul 29 18:34:24.135285 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:24.135236 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/89287a5239330a896427ad6104c0beff-config\") pod \"kube-apiserver-proxy-ip-10-0-140-205.ec2.internal\" (UID: \"89287a5239330a896427ad6104c0beff\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-205.ec2.internal" Jul 29 18:34:24.135285 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:24.135238 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0d56f34abf69a77b3a700a4e3019bb67-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-205.ec2.internal\" (UID: \"0d56f34abf69a77b3a700a4e3019bb67\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-205.ec2.internal" Jul 29 18:34:24.135285 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:24.135268 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0d56f34abf69a77b3a700a4e3019bb67-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-205.ec2.internal\" (UID: \"0d56f34abf69a77b3a700a4e3019bb67\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-205.ec2.internal" Jul 29 18:34:24.214845 ip-10-0-140-205 kubenswrapper[2560]: E0729 18:34:24.214780 2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-205.ec2.internal\" not found" Jul 29 18:34:24.315330 ip-10-0-140-205 kubenswrapper[2560]: E0729 18:34:24.315307 2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-205.ec2.internal\" not found" Jul 29 18:34:24.363779 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:24.363755 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-205.ec2.internal" Jul 29 18:34:24.367972 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:24.367652 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-205.ec2.internal" Jul 29 18:34:24.416282 ip-10-0-140-205 kubenswrapper[2560]: E0729 18:34:24.416260 2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-205.ec2.internal\" not found" Jul 29 18:34:25.934506 ip-10-0-140-205 kubenswrapper[2560]: E0729 18:34:25.934453 2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-205.ec2.internal\" not found" Jul 29 18:34:26.035024 ip-10-0-140-205 kubenswrapper[2560]: E0729 18:34:26.034992 2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-205.ec2.internal\" not found" Jul 29 18:34:26.083473 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:26.083442 2560 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 29 18:34:26.083949 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:26.083631 2560 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 29 18:34:26.083949 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:26.083631 2560 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 29 18:34:26.135925 ip-10-0-140-205 kubenswrapper[2560]: E0729 18:34:26.135897 2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-205.ec2.internal\" not found" Jul 29 18:34:26.147668 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:26.147639 2560 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-28 18:29:25 +0000 UTC" deadline="2028-01-13 17:44:08.756481174 +0000 UTC" Jul 29 18:34:26.147668 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:26.147662 2560 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12791h9m42.608821146s" Jul 29 18:34:26.236644 ip-10-0-140-205 kubenswrapper[2560]: E0729 18:34:26.236563 2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-205.ec2.internal\" not found" Jul 29 18:34:26.253474 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:26.253442 2560 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 29 18:34:26.332388 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:26.332335 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d56f34abf69a77b3a700a4e3019bb67.slice/crio-fa3624b1da4d9c718b08385a1c5e8a0d39366a3ec0489475a4570460fa79a76e WatchSource:0}: Error finding container fa3624b1da4d9c718b08385a1c5e8a0d39366a3ec0489475a4570460fa79a76e: Status 404 returned error can't find the container with id fa3624b1da4d9c718b08385a1c5e8a0d39366a3ec0489475a4570460fa79a76e Jul 29 18:34:26.336486 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:26.336471 2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 29 18:34:26.350495 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:26.350467 2560 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-205.ec2.internal" Jul 29 18:34:26.350732 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:26.350698 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-205.ec2.internal" event={"ID":"0d56f34abf69a77b3a700a4e3019bb67","Type":"ContainerStarted","Data":"fa3624b1da4d9c718b08385a1c5e8a0d39366a3ec0489475a4570460fa79a76e"} Jul 29 18:34:26.362999 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:26.362978 2560 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 29 18:34:26.364855 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:26.364840 2560 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-205.ec2.internal" Jul 29 18:34:26.376782 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:26.376764 2560 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 29 18:34:26.662839 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:26.662810 2560 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 29 18:34:27.110212 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.110186 2560 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 29 18:34:27.123355 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.123329 2560 apiserver.go:52] "Watching apiserver" Jul 29 18:34:27.132755 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.132733 2560 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 29 18:34:27.133853 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.133830 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-69g8w","kube-system/kube-apiserver-proxy-ip-10-0-140-205.ec2.internal","openshift-cluster-node-tuning-operator/tuned-84dhd","openshift-multus/multus-additional-cni-plugins-xvbfs","openshift-multus/multus-z7pg7","openshift-network-diagnostics/network-check-target-8qdhk","kube-system/global-pull-secret-syncer-zbwwp","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-249zt","openshift-dns/node-resolver-nnlkl","openshift-image-registry/node-ca-nvp4r","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-205.ec2.internal","openshift-multus/network-metrics-daemon-mrshg","openshift-network-operator/iptables-alerter-nxn8k","openshift-ovn-kubernetes/ovnkube-node-v4n5x"] Jul 29 18:34:27.135454 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.135435 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zbwwp" Jul 29 18:34:27.135555 ip-10-0-140-205 kubenswrapper[2560]: E0729 18:34:27.135531 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zbwwp" podUID="13fd7656-9a1f-4221-a5ed-6ebecf3664c9" Jul 29 18:34:27.136885 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.136865 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-nxn8k" Jul 29 18:34:27.136987 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:27.136969 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89287a5239330a896427ad6104c0beff.slice/crio-8562741694e2748a6039f09b2536fb6d416c7f5a2412ed3b65281cc0a77c6235 WatchSource:0}: Error finding container 8562741694e2748a6039f09b2536fb6d416c7f5a2412ed3b65281cc0a77c6235: Status 404 returned error can't find the container with id 8562741694e2748a6039f09b2536fb6d416c7f5a2412ed3b65281cc0a77c6235 Jul 29 18:34:27.137993 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.137974 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-84dhd" Jul 29 18:34:27.139053 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.139032 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 29 18:34:27.139161 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.139078 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 29 18:34:27.139161 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.139135 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 29 18:34:27.139161 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.139143 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-ckvns\"" Jul 29 18:34:27.139576 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.139556 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-xvbfs" Jul 29 18:34:27.140086 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.140071 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-wgzfr\"" Jul 29 18:34:27.140462 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.140183 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 29 18:34:27.140462 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.140183 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 29 18:34:27.140836 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.140815 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-z7pg7" Jul 29 18:34:27.141524 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.141499 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 29 18:34:27.141611 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.141538 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 29 18:34:27.141611 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.141600 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 29 18:34:27.141728 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.141676 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 29 18:34:27.141728 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.141695 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-tvhvm\"" Jul 29 18:34:27.142008 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.141990 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 29 18:34:27.142291 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.142274 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8qdhk" Jul 29 18:34:27.142378 ip-10-0-140-205 kubenswrapper[2560]: E0729 18:34:27.142336 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8qdhk" podUID="88336825-e84e-4595-9f68-f4d6b4e6edda" Jul 29 18:34:27.142635 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.142620 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 29 18:34:27.142908 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.142882 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-cvhs5\"" Jul 29 18:34:27.143296 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.143275 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-nnlkl" Jul 29 18:34:27.144874 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.144855 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-69g8w" Jul 29 18:34:27.145212 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.145196 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-qbddp\"" Jul 29 18:34:27.145295 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.145220 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 29 18:34:27.145463 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.145449 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 29 18:34:27.146607 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.146592 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 29 18:34:27.146733 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.146718 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-v4n5x" Jul 29 18:34:27.146825 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.146811 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-8blk5\"" Jul 29 18:34:27.147057 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.147043 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 29 18:34:27.148057 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.148039 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-nvp4r" Jul 29 18:34:27.148669 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.148650 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 29 18:34:27.148758 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.148698 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 29 18:34:27.148758 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.148708 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-zhk5w\"" Jul 29 18:34:27.149682 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.149663 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 29 18:34:27.149778 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.149665 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 29 18:34:27.149778 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.149743 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 29 18:34:27.150064 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.150047 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 29 18:34:27.150140 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.150092 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-l4fsz\"" Jul 29 18:34:27.150140 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.150104 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 29 18:34:27.150232 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.150155 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 29 18:34:27.150232 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.150205 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-249zt" Jul 29 18:34:27.150302 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.150236 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 29 18:34:27.151097 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.151076 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mrshg" Jul 29 18:34:27.152813 ip-10-0-140-205 kubenswrapper[2560]: E0729 18:34:27.151293 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mrshg" podUID="70307ff7-a0ca-4a2b-95be-94a5dde5d96e" Jul 29 18:34:27.152813 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.151377 2560 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 29 18:34:27.153066 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.153000 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 29 18:34:27.153149 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.153129 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 29 18:34:27.153277 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.153260 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-czv7p\"" Jul 29 18:34:27.153612 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.153593 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 29 18:34:27.154255 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.154237 2560 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 29 18:34:27.160636 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.160617 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/96c83b74-1875-4a28-8e0e-1470f61a31c9-host\") pod \"node-ca-nvp4r\" (UID: \"96c83b74-1875-4a28-8e0e-1470f61a31c9\") " pod="openshift-image-registry/node-ca-nvp4r" Jul 29 18:34:27.160723 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.160645 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d5b867d8-3ad7-4aef-a0d8-6e8938ae7c68-host-var-lib-cni-bin\") pod \"multus-z7pg7\" (UID: \"d5b867d8-3ad7-4aef-a0d8-6e8938ae7c68\") " pod="openshift-multus/multus-z7pg7" Jul 29 18:34:27.160723 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.160671 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tsrsw\" (UniqueName: \"kubernetes.io/projected/2fbfc68e-382a-4371-8291-ce71b8eec8ce-kube-api-access-tsrsw\") pod \"node-resolver-nnlkl\" (UID: \"2fbfc68e-382a-4371-8291-ce71b8eec8ce\") " pod="openshift-dns/node-resolver-nnlkl" Jul 29 18:34:27.160723 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.160693 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1727fb40-fbed-4984-91d4-a5890d0f4103-var-lib-openvswitch\") pod \"ovnkube-node-v4n5x\" (UID: \"1727fb40-fbed-4984-91d4-a5890d0f4103\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4n5x" Jul 29 18:34:27.160723 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.160718 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c46b0e07-5bf9-469e-bde5-bc4708ce64f2-etc-systemd\") pod \"tuned-84dhd\" (UID: \"c46b0e07-5bf9-469e-bde5-bc4708ce64f2\") " pod="openshift-cluster-node-tuning-operator/tuned-84dhd" Jul 29 18:34:27.160898 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.160733 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fb2816f1-d921-4e68-908b-ecc463ffb899-system-cni-dir\") pod \"multus-additional-cni-plugins-xvbfs\" (UID: \"fb2816f1-d921-4e68-908b-ecc463ffb899\") " pod="openshift-multus/multus-additional-cni-plugins-xvbfs" Jul 29 18:34:27.160898 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.160770 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fb2816f1-d921-4e68-908b-ecc463ffb899-tuning-conf-dir\") pod \"multus-additional-cni-plugins-xvbfs\" (UID: \"fb2816f1-d921-4e68-908b-ecc463ffb899\") " pod="openshift-multus/multus-additional-cni-plugins-xvbfs" Jul 29 18:34:27.160898 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.160803 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4hp6\" (UniqueName: \"kubernetes.io/projected/fb2816f1-d921-4e68-908b-ecc463ffb899-kube-api-access-l4hp6\") pod \"multus-additional-cni-plugins-xvbfs\" (UID: \"fb2816f1-d921-4e68-908b-ecc463ffb899\") " pod="openshift-multus/multus-additional-cni-plugins-xvbfs" Jul 29 18:34:27.160898 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.160829 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1727fb40-fbed-4984-91d4-a5890d0f4103-host-kubelet\") pod \"ovnkube-node-v4n5x\" (UID: \"1727fb40-fbed-4984-91d4-a5890d0f4103\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4n5x" Jul 29 18:34:27.160898 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.160855 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1727fb40-fbed-4984-91d4-a5890d0f4103-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-v4n5x\" (UID: \"1727fb40-fbed-4984-91d4-a5890d0f4103\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4n5x" Jul 29 18:34:27.160898 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.160887 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/aeadd3c9-a1ee-45a2-a683-3cfb51804716-etc-selinux\") pod \"aws-ebs-csi-driver-node-249zt\" (UID: \"aeadd3c9-a1ee-45a2-a683-3cfb51804716\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-249zt" Jul 29 18:34:27.161177 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.160937 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c46b0e07-5bf9-469e-bde5-bc4708ce64f2-etc-sysctl-conf\") pod \"tuned-84dhd\" (UID: \"c46b0e07-5bf9-469e-bde5-bc4708ce64f2\") " pod="openshift-cluster-node-tuning-operator/tuned-84dhd" Jul 29 18:34:27.161177 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.160965 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c46b0e07-5bf9-469e-bde5-bc4708ce64f2-host\") pod \"tuned-84dhd\" (UID: \"c46b0e07-5bf9-469e-bde5-bc4708ce64f2\") " pod="openshift-cluster-node-tuning-operator/tuned-84dhd" Jul 29 18:34:27.161177 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.160990 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/fb2816f1-d921-4e68-908b-ecc463ffb899-os-release\") pod \"multus-additional-cni-plugins-xvbfs\" (UID: \"fb2816f1-d921-4e68-908b-ecc463ffb899\") " pod="openshift-multus/multus-additional-cni-plugins-xvbfs" Jul 29 18:34:27.161177 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.161014 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d5b867d8-3ad7-4aef-a0d8-6e8938ae7c68-multus-daemon-config\") pod \"multus-z7pg7\" (UID: \"d5b867d8-3ad7-4aef-a0d8-6e8938ae7c68\") " pod="openshift-multus/multus-z7pg7" Jul 29 18:34:27.161177 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.161038 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1727fb40-fbed-4984-91d4-a5890d0f4103-host-run-ovn-kubernetes\") pod \"ovnkube-node-v4n5x\" (UID: \"1727fb40-fbed-4984-91d4-a5890d0f4103\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4n5x" Jul 29 18:34:27.161177 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.161058 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1727fb40-fbed-4984-91d4-a5890d0f4103-ovnkube-config\") pod \"ovnkube-node-v4n5x\" (UID: \"1727fb40-fbed-4984-91d4-a5890d0f4103\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4n5x" Jul 29 18:34:27.161177 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.161084 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/aeadd3c9-a1ee-45a2-a683-3cfb51804716-socket-dir\") pod \"aws-ebs-csi-driver-node-249zt\" (UID: \"aeadd3c9-a1ee-45a2-a683-3cfb51804716\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-249zt" Jul 29 18:34:27.161177 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.161107 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c46b0e07-5bf9-469e-bde5-bc4708ce64f2-etc-kubernetes\") pod \"tuned-84dhd\" (UID: \"c46b0e07-5bf9-469e-bde5-bc4708ce64f2\") " pod="openshift-cluster-node-tuning-operator/tuned-84dhd" Jul 29 18:34:27.161177 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.161144 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d5b867d8-3ad7-4aef-a0d8-6e8938ae7c68-cnibin\") pod \"multus-z7pg7\" (UID: \"d5b867d8-3ad7-4aef-a0d8-6e8938ae7c68\") " pod="openshift-multus/multus-z7pg7" Jul 29 18:34:27.161542 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.161180 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/13fd7656-9a1f-4221-a5ed-6ebecf3664c9-original-pull-secret\") pod \"global-pull-secret-syncer-zbwwp\" (UID: \"13fd7656-9a1f-4221-a5ed-6ebecf3664c9\") " pod="kube-system/global-pull-secret-syncer-zbwwp" Jul 29 18:34:27.161542 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.161218 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c46b0e07-5bf9-469e-bde5-bc4708ce64f2-run\") pod \"tuned-84dhd\" (UID: \"c46b0e07-5bf9-469e-bde5-bc4708ce64f2\") " pod="openshift-cluster-node-tuning-operator/tuned-84dhd" Jul 29 18:34:27.161542 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.161260 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/fb2816f1-d921-4e68-908b-ecc463ffb899-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-xvbfs\" (UID: \"fb2816f1-d921-4e68-908b-ecc463ffb899\") " pod="openshift-multus/multus-additional-cni-plugins-xvbfs" Jul 29 18:34:27.161542 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.161291 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/07b2a3b0-456d-4696-92a3-968f3cf7d956-konnectivity-ca\") pod \"konnectivity-agent-69g8w\" (UID: \"07b2a3b0-456d-4696-92a3-968f3cf7d956\") " pod="kube-system/konnectivity-agent-69g8w" Jul 29 18:34:27.161542 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.161315 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1727fb40-fbed-4984-91d4-a5890d0f4103-host-cni-bin\") pod \"ovnkube-node-v4n5x\" (UID: \"1727fb40-fbed-4984-91d4-a5890d0f4103\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4n5x" Jul 29 18:34:27.161542 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.161372 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lmh28\" (UniqueName: \"kubernetes.io/projected/1727fb40-fbed-4984-91d4-a5890d0f4103-kube-api-access-lmh28\") pod \"ovnkube-node-v4n5x\" (UID: \"1727fb40-fbed-4984-91d4-a5890d0f4103\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4n5x" Jul 29 18:34:27.161542 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.161403 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/13fd7656-9a1f-4221-a5ed-6ebecf3664c9-dbus\") pod \"global-pull-secret-syncer-zbwwp\" (UID: \"13fd7656-9a1f-4221-a5ed-6ebecf3664c9\") " pod="kube-system/global-pull-secret-syncer-zbwwp" Jul 29 18:34:27.161542 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.161436 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c46b0e07-5bf9-469e-bde5-bc4708ce64f2-etc-modprobe-d\") pod \"tuned-84dhd\" (UID: \"c46b0e07-5bf9-469e-bde5-bc4708ce64f2\") " pod="openshift-cluster-node-tuning-operator/tuned-84dhd" Jul 29 18:34:27.161542 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.161462 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c46b0e07-5bf9-469e-bde5-bc4708ce64f2-tmp\") pod \"tuned-84dhd\" (UID: \"c46b0e07-5bf9-469e-bde5-bc4708ce64f2\") " pod="openshift-cluster-node-tuning-operator/tuned-84dhd" Jul 29 18:34:27.161542 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.161506 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1727fb40-fbed-4984-91d4-a5890d0f4103-run-ovn\") pod \"ovnkube-node-v4n5x\" (UID: \"1727fb40-fbed-4984-91d4-a5890d0f4103\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4n5x" Jul 29 18:34:27.161542 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.161527 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1727fb40-fbed-4984-91d4-a5890d0f4103-host-cni-netd\") pod \"ovnkube-node-v4n5x\" (UID: \"1727fb40-fbed-4984-91d4-a5890d0f4103\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4n5x" Jul 29 18:34:27.161542 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.161542 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1727fb40-fbed-4984-91d4-a5890d0f4103-env-overrides\") pod \"ovnkube-node-v4n5x\" (UID: \"1727fb40-fbed-4984-91d4-a5890d0f4103\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4n5x" Jul 29 18:34:27.161935 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.161560 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/aeadd3c9-a1ee-45a2-a683-3cfb51804716-sys-fs\") pod \"aws-ebs-csi-driver-node-249zt\" (UID: \"aeadd3c9-a1ee-45a2-a683-3cfb51804716\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-249zt" Jul 29 18:34:27.161935 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.161574 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c46b0e07-5bf9-469e-bde5-bc4708ce64f2-var-lib-kubelet\") pod \"tuned-84dhd\" (UID: \"c46b0e07-5bf9-469e-bde5-bc4708ce64f2\") " pod="openshift-cluster-node-tuning-operator/tuned-84dhd" Jul 29 18:34:27.161935 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.161586 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/96c83b74-1875-4a28-8e0e-1470f61a31c9-serviceca\") pod \"node-ca-nvp4r\" (UID: \"96c83b74-1875-4a28-8e0e-1470f61a31c9\") " pod="openshift-image-registry/node-ca-nvp4r" Jul 29 18:34:27.161935 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.161599 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d5b867d8-3ad7-4aef-a0d8-6e8938ae7c68-host-run-netns\") pod \"multus-z7pg7\" (UID: \"d5b867d8-3ad7-4aef-a0d8-6e8938ae7c68\") " pod="openshift-multus/multus-z7pg7" Jul 29 18:34:27.161935 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.161618 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d5b867d8-3ad7-4aef-a0d8-6e8938ae7c68-host-var-lib-kubelet\") pod \"multus-z7pg7\" (UID: \"d5b867d8-3ad7-4aef-a0d8-6e8938ae7c68\") " pod="openshift-multus/multus-z7pg7" Jul 29 18:34:27.161935 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.161631 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1727fb40-fbed-4984-91d4-a5890d0f4103-host-slash\") pod \"ovnkube-node-v4n5x\" (UID: \"1727fb40-fbed-4984-91d4-a5890d0f4103\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4n5x" Jul 29 18:34:27.161935 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.161646 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1727fb40-fbed-4984-91d4-a5890d0f4103-ovnkube-script-lib\") pod \"ovnkube-node-v4n5x\" (UID: \"1727fb40-fbed-4984-91d4-a5890d0f4103\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4n5x" Jul 29 18:34:27.161935 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.161663 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-blmpz\" (UniqueName: \"kubernetes.io/projected/aeadd3c9-a1ee-45a2-a683-3cfb51804716-kube-api-access-blmpz\") pod \"aws-ebs-csi-driver-node-249zt\" (UID: \"aeadd3c9-a1ee-45a2-a683-3cfb51804716\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-249zt" Jul 29 18:34:27.161935 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.161680 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pnkgr\" (UniqueName: \"kubernetes.io/projected/90d823ce-32c1-4cf9-8165-161155b5c5d1-kube-api-access-pnkgr\") pod \"iptables-alerter-nxn8k\" (UID: \"90d823ce-32c1-4cf9-8165-161155b5c5d1\") " pod="openshift-network-operator/iptables-alerter-nxn8k" Jul 29 18:34:27.161935 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.161708 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d5b867d8-3ad7-4aef-a0d8-6e8938ae7c68-cni-binary-copy\") pod \"multus-z7pg7\" (UID: \"d5b867d8-3ad7-4aef-a0d8-6e8938ae7c68\") " pod="openshift-multus/multus-z7pg7" Jul 29 18:34:27.161935 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.161728 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d5b867d8-3ad7-4aef-a0d8-6e8938ae7c68-etc-kubernetes\") pod \"multus-z7pg7\" (UID: \"d5b867d8-3ad7-4aef-a0d8-6e8938ae7c68\") " pod="openshift-multus/multus-z7pg7" Jul 29 18:34:27.161935 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.161744 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phzf8\" (UniqueName: \"kubernetes.io/projected/88336825-e84e-4595-9f68-f4d6b4e6edda-kube-api-access-phzf8\") pod \"network-check-target-8qdhk\" (UID: \"88336825-e84e-4595-9f68-f4d6b4e6edda\") " pod="openshift-network-diagnostics/network-check-target-8qdhk" Jul 29 18:34:27.161935 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.161761 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1727fb40-fbed-4984-91d4-a5890d0f4103-systemd-units\") pod \"ovnkube-node-v4n5x\" (UID: \"1727fb40-fbed-4984-91d4-a5890d0f4103\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4n5x" Jul 29 18:34:27.161935 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.161775 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/aeadd3c9-a1ee-45a2-a683-3cfb51804716-registration-dir\") pod \"aws-ebs-csi-driver-node-249zt\" (UID: \"aeadd3c9-a1ee-45a2-a683-3cfb51804716\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-249zt" Jul 29 18:34:27.161935 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.161805 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d5b867d8-3ad7-4aef-a0d8-6e8938ae7c68-multus-cni-dir\") pod \"multus-z7pg7\" (UID: \"d5b867d8-3ad7-4aef-a0d8-6e8938ae7c68\") " pod="openshift-multus/multus-z7pg7" Jul 29 18:34:27.161935 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.161842 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khhzl\" (UniqueName: \"kubernetes.io/projected/d5b867d8-3ad7-4aef-a0d8-6e8938ae7c68-kube-api-access-khhzl\") pod \"multus-z7pg7\" (UID: \"d5b867d8-3ad7-4aef-a0d8-6e8938ae7c68\") " pod="openshift-multus/multus-z7pg7" Jul 29 18:34:27.162439 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.161876 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1727fb40-fbed-4984-91d4-a5890d0f4103-log-socket\") pod \"ovnkube-node-v4n5x\" (UID: \"1727fb40-fbed-4984-91d4-a5890d0f4103\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4n5x" Jul 29 18:34:27.162439 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.161907 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1727fb40-fbed-4984-91d4-a5890d0f4103-ovn-node-metrics-cert\") pod \"ovnkube-node-v4n5x\" (UID: \"1727fb40-fbed-4984-91d4-a5890d0f4103\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4n5x" Jul 29 18:34:27.162439 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.161952 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c46b0e07-5bf9-469e-bde5-bc4708ce64f2-etc-sysconfig\") pod \"tuned-84dhd\" (UID: \"c46b0e07-5bf9-469e-bde5-bc4708ce64f2\") " pod="openshift-cluster-node-tuning-operator/tuned-84dhd" Jul 29 18:34:27.162439 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.161982 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c46b0e07-5bf9-469e-bde5-bc4708ce64f2-etc-sysctl-d\") pod \"tuned-84dhd\" (UID: \"c46b0e07-5bf9-469e-bde5-bc4708ce64f2\") " pod="openshift-cluster-node-tuning-operator/tuned-84dhd" Jul 29 18:34:27.162439 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.162011 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/fb2816f1-d921-4e68-908b-ecc463ffb899-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-xvbfs\" (UID: \"fb2816f1-d921-4e68-908b-ecc463ffb899\") " pod="openshift-multus/multus-additional-cni-plugins-xvbfs" Jul 29 18:34:27.162439 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.162037 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvv7r\" (UniqueName: \"kubernetes.io/projected/70307ff7-a0ca-4a2b-95be-94a5dde5d96e-kube-api-access-zvv7r\") pod \"network-metrics-daemon-mrshg\" (UID: \"70307ff7-a0ca-4a2b-95be-94a5dde5d96e\") " pod="openshift-multus/network-metrics-daemon-mrshg" Jul 29 18:34:27.162439 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.162060 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/90d823ce-32c1-4cf9-8165-161155b5c5d1-host-slash\") pod \"iptables-alerter-nxn8k\" (UID: \"90d823ce-32c1-4cf9-8165-161155b5c5d1\") " pod="openshift-network-operator/iptables-alerter-nxn8k" Jul 29 18:34:27.162439 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.162085 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/2fbfc68e-382a-4371-8291-ce71b8eec8ce-hosts-file\") pod \"node-resolver-nnlkl\" (UID: \"2fbfc68e-382a-4371-8291-ce71b8eec8ce\") " pod="openshift-dns/node-resolver-nnlkl" Jul 29 18:34:27.162439 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.162108 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1727fb40-fbed-4984-91d4-a5890d0f4103-etc-openvswitch\") pod \"ovnkube-node-v4n5x\" (UID: \"1727fb40-fbed-4984-91d4-a5890d0f4103\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4n5x" Jul 29 18:34:27.162439 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.162133 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5lxr\" (UniqueName: \"kubernetes.io/projected/96c83b74-1875-4a28-8e0e-1470f61a31c9-kube-api-access-q5lxr\") pod \"node-ca-nvp4r\" (UID: \"96c83b74-1875-4a28-8e0e-1470f61a31c9\") " pod="openshift-image-registry/node-ca-nvp4r" Jul 29 18:34:27.162439 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.162152 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70307ff7-a0ca-4a2b-95be-94a5dde5d96e-metrics-certs\") pod \"network-metrics-daemon-mrshg\" (UID: \"70307ff7-a0ca-4a2b-95be-94a5dde5d96e\") " pod="openshift-multus/network-metrics-daemon-mrshg" Jul 29 18:34:27.162439 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.162171 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1727fb40-fbed-4984-91d4-a5890d0f4103-run-systemd\") pod \"ovnkube-node-v4n5x\" (UID: \"1727fb40-fbed-4984-91d4-a5890d0f4103\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4n5x" Jul 29 18:34:27.162439 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.162197 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1727fb40-fbed-4984-91d4-a5890d0f4103-node-log\") pod \"ovnkube-node-v4n5x\" (UID: \"1727fb40-fbed-4984-91d4-a5890d0f4103\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4n5x" Jul 29 18:34:27.162439 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.162226 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d5b867d8-3ad7-4aef-a0d8-6e8938ae7c68-system-cni-dir\") pod \"multus-z7pg7\" (UID: \"d5b867d8-3ad7-4aef-a0d8-6e8938ae7c68\") " pod="openshift-multus/multus-z7pg7" Jul 29 18:34:27.162439 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.162244 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d5b867d8-3ad7-4aef-a0d8-6e8938ae7c68-host-var-lib-cni-multus\") pod \"multus-z7pg7\" (UID: \"d5b867d8-3ad7-4aef-a0d8-6e8938ae7c68\") " pod="openshift-multus/multus-z7pg7" Jul 29 18:34:27.162439 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.162267 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/90d823ce-32c1-4cf9-8165-161155b5c5d1-iptables-alerter-script\") pod \"iptables-alerter-nxn8k\" (UID: \"90d823ce-32c1-4cf9-8165-161155b5c5d1\") " pod="openshift-network-operator/iptables-alerter-nxn8k" Jul 29 18:34:27.162962 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.162292 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d5b867d8-3ad7-4aef-a0d8-6e8938ae7c68-host-run-k8s-cni-cncf-io\") pod \"multus-z7pg7\" (UID: \"d5b867d8-3ad7-4aef-a0d8-6e8938ae7c68\") " pod="openshift-multus/multus-z7pg7" Jul 29 18:34:27.162962 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.162318 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/13fd7656-9a1f-4221-a5ed-6ebecf3664c9-kubelet-config\") pod \"global-pull-secret-syncer-zbwwp\" (UID: \"13fd7656-9a1f-4221-a5ed-6ebecf3664c9\") " pod="kube-system/global-pull-secret-syncer-zbwwp" Jul 29 18:34:27.162962 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.162339 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c46b0e07-5bf9-469e-bde5-bc4708ce64f2-sys\") pod \"tuned-84dhd\" (UID: \"c46b0e07-5bf9-469e-bde5-bc4708ce64f2\") " pod="openshift-cluster-node-tuning-operator/tuned-84dhd" Jul 29 18:34:27.162962 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.162384 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fb2816f1-d921-4e68-908b-ecc463ffb899-cni-binary-copy\") pod \"multus-additional-cni-plugins-xvbfs\" (UID: \"fb2816f1-d921-4e68-908b-ecc463ffb899\") " pod="openshift-multus/multus-additional-cni-plugins-xvbfs" Jul 29 18:34:27.162962 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.162399 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/d5b867d8-3ad7-4aef-a0d8-6e8938ae7c68-os-release\") pod \"multus-z7pg7\" (UID: \"d5b867d8-3ad7-4aef-a0d8-6e8938ae7c68\") " pod="openshift-multus/multus-z7pg7" Jul 29 18:34:27.162962 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.162411 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d5b867d8-3ad7-4aef-a0d8-6e8938ae7c68-hostroot\") pod \"multus-z7pg7\" (UID: \"d5b867d8-3ad7-4aef-a0d8-6e8938ae7c68\") " pod="openshift-multus/multus-z7pg7" Jul 29 18:34:27.162962 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.162424 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/07b2a3b0-456d-4696-92a3-968f3cf7d956-agent-certs\") pod \"konnectivity-agent-69g8w\" (UID: \"07b2a3b0-456d-4696-92a3-968f3cf7d956\") " pod="kube-system/konnectivity-agent-69g8w" Jul 29 18:34:27.162962 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.162442 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c46b0e07-5bf9-469e-bde5-bc4708ce64f2-etc-tuned\") pod \"tuned-84dhd\" (UID: \"c46b0e07-5bf9-469e-bde5-bc4708ce64f2\") " pod="openshift-cluster-node-tuning-operator/tuned-84dhd" Jul 29 18:34:27.162962 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.162459 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fb2816f1-d921-4e68-908b-ecc463ffb899-cnibin\") pod \"multus-additional-cni-plugins-xvbfs\" (UID: \"fb2816f1-d921-4e68-908b-ecc463ffb899\") " pod="openshift-multus/multus-additional-cni-plugins-xvbfs" Jul 29 18:34:27.162962 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.162477 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d5b867d8-3ad7-4aef-a0d8-6e8938ae7c68-multus-conf-dir\") pod \"multus-z7pg7\" (UID: \"d5b867d8-3ad7-4aef-a0d8-6e8938ae7c68\") " pod="openshift-multus/multus-z7pg7" Jul 29 18:34:27.162962 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.162487 2560 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 29 18:34:27.162962 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.162505 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2fbfc68e-382a-4371-8291-ce71b8eec8ce-tmp-dir\") pod \"node-resolver-nnlkl\" (UID: \"2fbfc68e-382a-4371-8291-ce71b8eec8ce\") " pod="openshift-dns/node-resolver-nnlkl" Jul 29 18:34:27.162962 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.162540 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1727fb40-fbed-4984-91d4-a5890d0f4103-host-run-netns\") pod \"ovnkube-node-v4n5x\" (UID: \"1727fb40-fbed-4984-91d4-a5890d0f4103\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4n5x" Jul 29 18:34:27.162962 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.162560 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/aeadd3c9-a1ee-45a2-a683-3cfb51804716-device-dir\") pod \"aws-ebs-csi-driver-node-249zt\" (UID: \"aeadd3c9-a1ee-45a2-a683-3cfb51804716\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-249zt" Jul 29 18:34:27.162962 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.162575 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d5b867d8-3ad7-4aef-a0d8-6e8938ae7c68-multus-socket-dir-parent\") pod \"multus-z7pg7\" (UID: \"d5b867d8-3ad7-4aef-a0d8-6e8938ae7c68\") " pod="openshift-multus/multus-z7pg7" Jul 29 18:34:27.162962 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.162590 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d5b867d8-3ad7-4aef-a0d8-6e8938ae7c68-host-run-multus-certs\") pod \"multus-z7pg7\" (UID: \"d5b867d8-3ad7-4aef-a0d8-6e8938ae7c68\") " pod="openshift-multus/multus-z7pg7" Jul 29 18:34:27.162962 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.162609 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1727fb40-fbed-4984-91d4-a5890d0f4103-run-openvswitch\") pod \"ovnkube-node-v4n5x\" (UID: \"1727fb40-fbed-4984-91d4-a5890d0f4103\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4n5x" Jul 29 18:34:27.163454 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.162635 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/aeadd3c9-a1ee-45a2-a683-3cfb51804716-kubelet-dir\") pod \"aws-ebs-csi-driver-node-249zt\" (UID: \"aeadd3c9-a1ee-45a2-a683-3cfb51804716\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-249zt" Jul 29 18:34:27.163454 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.162660 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c46b0e07-5bf9-469e-bde5-bc4708ce64f2-lib-modules\") pod \"tuned-84dhd\" (UID: \"c46b0e07-5bf9-469e-bde5-bc4708ce64f2\") " pod="openshift-cluster-node-tuning-operator/tuned-84dhd" Jul 29 18:34:27.163454 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.162677 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vsk9g\" (UniqueName: \"kubernetes.io/projected/c46b0e07-5bf9-469e-bde5-bc4708ce64f2-kube-api-access-vsk9g\") pod \"tuned-84dhd\" (UID: \"c46b0e07-5bf9-469e-bde5-bc4708ce64f2\") " pod="openshift-cluster-node-tuning-operator/tuned-84dhd" Jul 29 18:34:27.171432 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.171418 2560 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 29 18:34:27.190137 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.190120 2560 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-kmhvs" Jul 29 18:34:27.201043 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.201027 2560 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-kmhvs" Jul 29 18:34:27.263908 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.263873 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/90d823ce-32c1-4cf9-8165-161155b5c5d1-host-slash\") pod \"iptables-alerter-nxn8k\" (UID: \"90d823ce-32c1-4cf9-8165-161155b5c5d1\") " pod="openshift-network-operator/iptables-alerter-nxn8k" Jul 29 18:34:27.263908 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.263907 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/2fbfc68e-382a-4371-8291-ce71b8eec8ce-hosts-file\") pod \"node-resolver-nnlkl\" (UID: \"2fbfc68e-382a-4371-8291-ce71b8eec8ce\") " pod="openshift-dns/node-resolver-nnlkl" Jul 29 18:34:27.264136 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.263928 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1727fb40-fbed-4984-91d4-a5890d0f4103-etc-openvswitch\") pod \"ovnkube-node-v4n5x\" (UID: \"1727fb40-fbed-4984-91d4-a5890d0f4103\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4n5x" Jul 29 18:34:27.264136 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.263949 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5lxr\" (UniqueName: \"kubernetes.io/projected/96c83b74-1875-4a28-8e0e-1470f61a31c9-kube-api-access-q5lxr\") pod \"node-ca-nvp4r\" (UID: \"96c83b74-1875-4a28-8e0e-1470f61a31c9\") " pod="openshift-image-registry/node-ca-nvp4r" Jul 29 18:34:27.264136 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.263983 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/90d823ce-32c1-4cf9-8165-161155b5c5d1-host-slash\") pod \"iptables-alerter-nxn8k\" (UID: \"90d823ce-32c1-4cf9-8165-161155b5c5d1\") " pod="openshift-network-operator/iptables-alerter-nxn8k" Jul 29 18:34:27.264136 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.263986 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70307ff7-a0ca-4a2b-95be-94a5dde5d96e-metrics-certs\") pod \"network-metrics-daemon-mrshg\" (UID: \"70307ff7-a0ca-4a2b-95be-94a5dde5d96e\") " pod="openshift-multus/network-metrics-daemon-mrshg" Jul 29 18:34:27.264136 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.264014 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1727fb40-fbed-4984-91d4-a5890d0f4103-etc-openvswitch\") pod \"ovnkube-node-v4n5x\" (UID: \"1727fb40-fbed-4984-91d4-a5890d0f4103\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4n5x" Jul 29 18:34:27.264136 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.264019 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1727fb40-fbed-4984-91d4-a5890d0f4103-run-systemd\") pod \"ovnkube-node-v4n5x\" (UID: \"1727fb40-fbed-4984-91d4-a5890d0f4103\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4n5x" Jul 29 18:34:27.264136 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.264026 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/2fbfc68e-382a-4371-8291-ce71b8eec8ce-hosts-file\") pod \"node-resolver-nnlkl\" (UID: \"2fbfc68e-382a-4371-8291-ce71b8eec8ce\") " pod="openshift-dns/node-resolver-nnlkl" Jul 29 18:34:27.264136 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.264040 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1727fb40-fbed-4984-91d4-a5890d0f4103-node-log\") pod \"ovnkube-node-v4n5x\" (UID: \"1727fb40-fbed-4984-91d4-a5890d0f4103\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4n5x" Jul 29 18:34:27.264136 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.264099 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1727fb40-fbed-4984-91d4-a5890d0f4103-run-systemd\") pod \"ovnkube-node-v4n5x\" (UID: \"1727fb40-fbed-4984-91d4-a5890d0f4103\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4n5x" Jul 29 18:34:27.264649 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.264113 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d5b867d8-3ad7-4aef-a0d8-6e8938ae7c68-system-cni-dir\") pod \"multus-z7pg7\" (UID: \"d5b867d8-3ad7-4aef-a0d8-6e8938ae7c68\") " pod="openshift-multus/multus-z7pg7" Jul 29 18:34:27.264649 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.264169 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1727fb40-fbed-4984-91d4-a5890d0f4103-node-log\") pod \"ovnkube-node-v4n5x\" (UID: \"1727fb40-fbed-4984-91d4-a5890d0f4103\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4n5x" Jul 29 18:34:27.264649 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.264183 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d5b867d8-3ad7-4aef-a0d8-6e8938ae7c68-host-var-lib-cni-multus\") pod \"multus-z7pg7\" (UID: \"d5b867d8-3ad7-4aef-a0d8-6e8938ae7c68\") " pod="openshift-multus/multus-z7pg7" Jul 29 18:34:27.264649 ip-10-0-140-205 kubenswrapper[2560]: E0729 18:34:27.264220 2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 18:34:27.264649 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.264247 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d5b867d8-3ad7-4aef-a0d8-6e8938ae7c68-system-cni-dir\") pod \"multus-z7pg7\" (UID: \"d5b867d8-3ad7-4aef-a0d8-6e8938ae7c68\") " pod="openshift-multus/multus-z7pg7" Jul 29 18:34:27.264649 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.264269 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/90d823ce-32c1-4cf9-8165-161155b5c5d1-iptables-alerter-script\") pod \"iptables-alerter-nxn8k\" (UID: \"90d823ce-32c1-4cf9-8165-161155b5c5d1\") " pod="openshift-network-operator/iptables-alerter-nxn8k" Jul 29 18:34:27.264649 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.264265 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d5b867d8-3ad7-4aef-a0d8-6e8938ae7c68-host-var-lib-cni-multus\") pod \"multus-z7pg7\" (UID: \"d5b867d8-3ad7-4aef-a0d8-6e8938ae7c68\") " pod="openshift-multus/multus-z7pg7" Jul 29 18:34:27.264649 ip-10-0-140-205 kubenswrapper[2560]: E0729 18:34:27.264300 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/70307ff7-a0ca-4a2b-95be-94a5dde5d96e-metrics-certs podName:70307ff7-a0ca-4a2b-95be-94a5dde5d96e nodeName:}" failed. No retries permitted until 2026-07-29 18:34:27.764263708 +0000 UTC m=+2.989932558 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/70307ff7-a0ca-4a2b-95be-94a5dde5d96e-metrics-certs") pod "network-metrics-daemon-mrshg" (UID: "70307ff7-a0ca-4a2b-95be-94a5dde5d96e") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 18:34:27.264649 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.264370 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d5b867d8-3ad7-4aef-a0d8-6e8938ae7c68-host-run-k8s-cni-cncf-io\") pod \"multus-z7pg7\" (UID: \"d5b867d8-3ad7-4aef-a0d8-6e8938ae7c68\") " pod="openshift-multus/multus-z7pg7" Jul 29 18:34:27.264649 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.264404 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/13fd7656-9a1f-4221-a5ed-6ebecf3664c9-kubelet-config\") pod \"global-pull-secret-syncer-zbwwp\" (UID: \"13fd7656-9a1f-4221-a5ed-6ebecf3664c9\") " pod="kube-system/global-pull-secret-syncer-zbwwp" Jul 29 18:34:27.264649 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.264427 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c46b0e07-5bf9-469e-bde5-bc4708ce64f2-sys\") pod \"tuned-84dhd\" (UID: \"c46b0e07-5bf9-469e-bde5-bc4708ce64f2\") " pod="openshift-cluster-node-tuning-operator/tuned-84dhd" Jul 29 18:34:27.264649 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.264436 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d5b867d8-3ad7-4aef-a0d8-6e8938ae7c68-host-run-k8s-cni-cncf-io\") pod \"multus-z7pg7\" (UID: \"d5b867d8-3ad7-4aef-a0d8-6e8938ae7c68\") " pod="openshift-multus/multus-z7pg7" Jul 29 18:34:27.264649 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.264450 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fb2816f1-d921-4e68-908b-ecc463ffb899-cni-binary-copy\") pod \"multus-additional-cni-plugins-xvbfs\" (UID: \"fb2816f1-d921-4e68-908b-ecc463ffb899\") " pod="openshift-multus/multus-additional-cni-plugins-xvbfs" Jul 29 18:34:27.264649 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.264474 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/d5b867d8-3ad7-4aef-a0d8-6e8938ae7c68-os-release\") pod \"multus-z7pg7\" (UID: \"d5b867d8-3ad7-4aef-a0d8-6e8938ae7c68\") " pod="openshift-multus/multus-z7pg7" Jul 29 18:34:27.264649 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.264499 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d5b867d8-3ad7-4aef-a0d8-6e8938ae7c68-hostroot\") pod \"multus-z7pg7\" (UID: \"d5b867d8-3ad7-4aef-a0d8-6e8938ae7c68\") " pod="openshift-multus/multus-z7pg7" Jul 29 18:34:27.264649 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.264474 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/13fd7656-9a1f-4221-a5ed-6ebecf3664c9-kubelet-config\") pod \"global-pull-secret-syncer-zbwwp\" (UID: \"13fd7656-9a1f-4221-a5ed-6ebecf3664c9\") " pod="kube-system/global-pull-secret-syncer-zbwwp" Jul 29 18:34:27.264649 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.264525 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/07b2a3b0-456d-4696-92a3-968f3cf7d956-agent-certs\") pod \"konnectivity-agent-69g8w\" (UID: \"07b2a3b0-456d-4696-92a3-968f3cf7d956\") " pod="kube-system/konnectivity-agent-69g8w" Jul 29 18:34:27.265457 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.264548 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c46b0e07-5bf9-469e-bde5-bc4708ce64f2-etc-tuned\") pod \"tuned-84dhd\" (UID: \"c46b0e07-5bf9-469e-bde5-bc4708ce64f2\") " pod="openshift-cluster-node-tuning-operator/tuned-84dhd" Jul 29 18:34:27.265457 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.264557 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/d5b867d8-3ad7-4aef-a0d8-6e8938ae7c68-os-release\") pod \"multus-z7pg7\" (UID: \"d5b867d8-3ad7-4aef-a0d8-6e8938ae7c68\") " pod="openshift-multus/multus-z7pg7" Jul 29 18:34:27.265457 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.264560 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d5b867d8-3ad7-4aef-a0d8-6e8938ae7c68-hostroot\") pod \"multus-z7pg7\" (UID: \"d5b867d8-3ad7-4aef-a0d8-6e8938ae7c68\") " pod="openshift-multus/multus-z7pg7" Jul 29 18:34:27.265457 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.264573 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fb2816f1-d921-4e68-908b-ecc463ffb899-cnibin\") pod \"multus-additional-cni-plugins-xvbfs\" (UID: \"fb2816f1-d921-4e68-908b-ecc463ffb899\") " pod="openshift-multus/multus-additional-cni-plugins-xvbfs" Jul 29 18:34:27.265457 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.264486 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c46b0e07-5bf9-469e-bde5-bc4708ce64f2-sys\") pod \"tuned-84dhd\" (UID: \"c46b0e07-5bf9-469e-bde5-bc4708ce64f2\") " pod="openshift-cluster-node-tuning-operator/tuned-84dhd" Jul 29 18:34:27.265457 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.264595 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d5b867d8-3ad7-4aef-a0d8-6e8938ae7c68-multus-conf-dir\") pod \"multus-z7pg7\" (UID: \"d5b867d8-3ad7-4aef-a0d8-6e8938ae7c68\") " pod="openshift-multus/multus-z7pg7" Jul 29 18:34:27.265457 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.264636 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fb2816f1-d921-4e68-908b-ecc463ffb899-cnibin\") pod \"multus-additional-cni-plugins-xvbfs\" (UID: \"fb2816f1-d921-4e68-908b-ecc463ffb899\") " pod="openshift-multus/multus-additional-cni-plugins-xvbfs" Jul 29 18:34:27.265457 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.264657 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2fbfc68e-382a-4371-8291-ce71b8eec8ce-tmp-dir\") pod \"node-resolver-nnlkl\" (UID: \"2fbfc68e-382a-4371-8291-ce71b8eec8ce\") " pod="openshift-dns/node-resolver-nnlkl" Jul 29 18:34:27.265457 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.264684 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1727fb40-fbed-4984-91d4-a5890d0f4103-host-run-netns\") pod \"ovnkube-node-v4n5x\" (UID: \"1727fb40-fbed-4984-91d4-a5890d0f4103\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4n5x" Jul 29 18:34:27.265457 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.264702 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d5b867d8-3ad7-4aef-a0d8-6e8938ae7c68-multus-conf-dir\") pod \"multus-z7pg7\" (UID: \"d5b867d8-3ad7-4aef-a0d8-6e8938ae7c68\") " pod="openshift-multus/multus-z7pg7" Jul 29 18:34:27.265457 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.264709 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/aeadd3c9-a1ee-45a2-a683-3cfb51804716-device-dir\") pod \"aws-ebs-csi-driver-node-249zt\" (UID: \"aeadd3c9-a1ee-45a2-a683-3cfb51804716\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-249zt" Jul 29 18:34:27.265457 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.264731 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d5b867d8-3ad7-4aef-a0d8-6e8938ae7c68-multus-socket-dir-parent\") pod \"multus-z7pg7\" (UID: \"d5b867d8-3ad7-4aef-a0d8-6e8938ae7c68\") " pod="openshift-multus/multus-z7pg7" Jul 29 18:34:27.265457 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.264744 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1727fb40-fbed-4984-91d4-a5890d0f4103-host-run-netns\") pod \"ovnkube-node-v4n5x\" (UID: \"1727fb40-fbed-4984-91d4-a5890d0f4103\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4n5x" Jul 29 18:34:27.265457 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.264753 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d5b867d8-3ad7-4aef-a0d8-6e8938ae7c68-host-run-multus-certs\") pod \"multus-z7pg7\" (UID: \"d5b867d8-3ad7-4aef-a0d8-6e8938ae7c68\") " pod="openshift-multus/multus-z7pg7" Jul 29 18:34:27.265457 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.264777 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1727fb40-fbed-4984-91d4-a5890d0f4103-run-openvswitch\") pod \"ovnkube-node-v4n5x\" (UID: \"1727fb40-fbed-4984-91d4-a5890d0f4103\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4n5x" Jul 29 18:34:27.265457 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.264796 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/aeadd3c9-a1ee-45a2-a683-3cfb51804716-device-dir\") pod \"aws-ebs-csi-driver-node-249zt\" (UID: \"aeadd3c9-a1ee-45a2-a683-3cfb51804716\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-249zt" Jul 29 18:34:27.265457 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.264801 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/aeadd3c9-a1ee-45a2-a683-3cfb51804716-kubelet-dir\") pod \"aws-ebs-csi-driver-node-249zt\" (UID: \"aeadd3c9-a1ee-45a2-a683-3cfb51804716\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-249zt" Jul 29 18:34:27.265457 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.264857 2560 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 29 18:34:27.266281 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.264878 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d5b867d8-3ad7-4aef-a0d8-6e8938ae7c68-host-run-multus-certs\") pod \"multus-z7pg7\" (UID: \"d5b867d8-3ad7-4aef-a0d8-6e8938ae7c68\") " pod="openshift-multus/multus-z7pg7" Jul 29 18:34:27.266281 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.264865 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1727fb40-fbed-4984-91d4-a5890d0f4103-run-openvswitch\") pod \"ovnkube-node-v4n5x\" (UID: \"1727fb40-fbed-4984-91d4-a5890d0f4103\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4n5x" Jul 29 18:34:27.266281 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.264863 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d5b867d8-3ad7-4aef-a0d8-6e8938ae7c68-multus-socket-dir-parent\") pod \"multus-z7pg7\" (UID: \"d5b867d8-3ad7-4aef-a0d8-6e8938ae7c68\") " pod="openshift-multus/multus-z7pg7" Jul 29 18:34:27.266281 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.264951 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/aeadd3c9-a1ee-45a2-a683-3cfb51804716-kubelet-dir\") pod \"aws-ebs-csi-driver-node-249zt\" (UID: \"aeadd3c9-a1ee-45a2-a683-3cfb51804716\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-249zt" Jul 29 18:34:27.266281 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.264969 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c46b0e07-5bf9-469e-bde5-bc4708ce64f2-lib-modules\") pod \"tuned-84dhd\" (UID: \"c46b0e07-5bf9-469e-bde5-bc4708ce64f2\") " pod="openshift-cluster-node-tuning-operator/tuned-84dhd" Jul 29 18:34:27.266281 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.264994 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vsk9g\" (UniqueName: \"kubernetes.io/projected/c46b0e07-5bf9-469e-bde5-bc4708ce64f2-kube-api-access-vsk9g\") pod \"tuned-84dhd\" (UID: \"c46b0e07-5bf9-469e-bde5-bc4708ce64f2\") " pod="openshift-cluster-node-tuning-operator/tuned-84dhd" Jul 29 18:34:27.266281 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.265012 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/96c83b74-1875-4a28-8e0e-1470f61a31c9-host\") pod \"node-ca-nvp4r\" (UID: \"96c83b74-1875-4a28-8e0e-1470f61a31c9\") " pod="openshift-image-registry/node-ca-nvp4r" Jul 29 18:34:27.266281 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.265017 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2fbfc68e-382a-4371-8291-ce71b8eec8ce-tmp-dir\") pod \"node-resolver-nnlkl\" (UID: \"2fbfc68e-382a-4371-8291-ce71b8eec8ce\") " pod="openshift-dns/node-resolver-nnlkl" Jul 29 18:34:27.266281 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.265033 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d5b867d8-3ad7-4aef-a0d8-6e8938ae7c68-host-var-lib-cni-bin\") pod \"multus-z7pg7\" (UID: \"d5b867d8-3ad7-4aef-a0d8-6e8938ae7c68\") " pod="openshift-multus/multus-z7pg7" Jul 29 18:34:27.266281 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.265078 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tsrsw\" (UniqueName: \"kubernetes.io/projected/2fbfc68e-382a-4371-8291-ce71b8eec8ce-kube-api-access-tsrsw\") pod \"node-resolver-nnlkl\" (UID: \"2fbfc68e-382a-4371-8291-ce71b8eec8ce\") " pod="openshift-dns/node-resolver-nnlkl" Jul 29 18:34:27.266281 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.265116 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/96c83b74-1875-4a28-8e0e-1470f61a31c9-host\") pod \"node-ca-nvp4r\" (UID: \"96c83b74-1875-4a28-8e0e-1470f61a31c9\") " pod="openshift-image-registry/node-ca-nvp4r" Jul 29 18:34:27.266281 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.265176 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d5b867d8-3ad7-4aef-a0d8-6e8938ae7c68-host-var-lib-cni-bin\") pod \"multus-z7pg7\" (UID: \"d5b867d8-3ad7-4aef-a0d8-6e8938ae7c68\") " pod="openshift-multus/multus-z7pg7" Jul 29 18:34:27.266281 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.265209 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c46b0e07-5bf9-469e-bde5-bc4708ce64f2-lib-modules\") pod \"tuned-84dhd\" (UID: \"c46b0e07-5bf9-469e-bde5-bc4708ce64f2\") " pod="openshift-cluster-node-tuning-operator/tuned-84dhd" Jul 29 18:34:27.266281 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.265210 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1727fb40-fbed-4984-91d4-a5890d0f4103-var-lib-openvswitch\") pod \"ovnkube-node-v4n5x\" (UID: \"1727fb40-fbed-4984-91d4-a5890d0f4103\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4n5x" Jul 29 18:34:27.266281 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.265248 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1727fb40-fbed-4984-91d4-a5890d0f4103-var-lib-openvswitch\") pod \"ovnkube-node-v4n5x\" (UID: \"1727fb40-fbed-4984-91d4-a5890d0f4103\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4n5x" Jul 29 18:34:27.266281 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.265260 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c46b0e07-5bf9-469e-bde5-bc4708ce64f2-etc-systemd\") pod \"tuned-84dhd\" (UID: \"c46b0e07-5bf9-469e-bde5-bc4708ce64f2\") " pod="openshift-cluster-node-tuning-operator/tuned-84dhd" Jul 29 18:34:27.266281 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.265288 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fb2816f1-d921-4e68-908b-ecc463ffb899-system-cni-dir\") pod \"multus-additional-cni-plugins-xvbfs\" (UID: \"fb2816f1-d921-4e68-908b-ecc463ffb899\") " pod="openshift-multus/multus-additional-cni-plugins-xvbfs" Jul 29 18:34:27.266281 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.265316 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fb2816f1-d921-4e68-908b-ecc463ffb899-tuning-conf-dir\") pod \"multus-additional-cni-plugins-xvbfs\" (UID: \"fb2816f1-d921-4e68-908b-ecc463ffb899\") " pod="openshift-multus/multus-additional-cni-plugins-xvbfs" Jul 29 18:34:27.267199 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.265342 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4hp6\" (UniqueName: \"kubernetes.io/projected/fb2816f1-d921-4e68-908b-ecc463ffb899-kube-api-access-l4hp6\") pod \"multus-additional-cni-plugins-xvbfs\" (UID: \"fb2816f1-d921-4e68-908b-ecc463ffb899\") " pod="openshift-multus/multus-additional-cni-plugins-xvbfs" Jul 29 18:34:27.267199 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.265390 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1727fb40-fbed-4984-91d4-a5890d0f4103-host-kubelet\") pod \"ovnkube-node-v4n5x\" (UID: \"1727fb40-fbed-4984-91d4-a5890d0f4103\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4n5x" Jul 29 18:34:27.267199 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.265393 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fb2816f1-d921-4e68-908b-ecc463ffb899-system-cni-dir\") pod \"multus-additional-cni-plugins-xvbfs\" (UID: \"fb2816f1-d921-4e68-908b-ecc463ffb899\") " pod="openshift-multus/multus-additional-cni-plugins-xvbfs" Jul 29 18:34:27.267199 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.265416 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1727fb40-fbed-4984-91d4-a5890d0f4103-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-v4n5x\" (UID: \"1727fb40-fbed-4984-91d4-a5890d0f4103\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4n5x" Jul 29 18:34:27.267199 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.265443 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/aeadd3c9-a1ee-45a2-a683-3cfb51804716-etc-selinux\") pod \"aws-ebs-csi-driver-node-249zt\" (UID: \"aeadd3c9-a1ee-45a2-a683-3cfb51804716\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-249zt" Jul 29 18:34:27.267199 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.265464 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c46b0e07-5bf9-469e-bde5-bc4708ce64f2-etc-systemd\") pod \"tuned-84dhd\" (UID: \"c46b0e07-5bf9-469e-bde5-bc4708ce64f2\") " pod="openshift-cluster-node-tuning-operator/tuned-84dhd" Jul 29 18:34:27.267199 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.265468 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c46b0e07-5bf9-469e-bde5-bc4708ce64f2-etc-sysctl-conf\") pod \"tuned-84dhd\" (UID: \"c46b0e07-5bf9-469e-bde5-bc4708ce64f2\") " pod="openshift-cluster-node-tuning-operator/tuned-84dhd" Jul 29 18:34:27.267199 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.265493 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c46b0e07-5bf9-469e-bde5-bc4708ce64f2-host\") pod \"tuned-84dhd\" (UID: \"c46b0e07-5bf9-469e-bde5-bc4708ce64f2\") " pod="openshift-cluster-node-tuning-operator/tuned-84dhd" Jul 29 18:34:27.267199 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.265521 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/fb2816f1-d921-4e68-908b-ecc463ffb899-os-release\") pod \"multus-additional-cni-plugins-xvbfs\" (UID: \"fb2816f1-d921-4e68-908b-ecc463ffb899\") " pod="openshift-multus/multus-additional-cni-plugins-xvbfs" Jul 29 18:34:27.267199 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.265549 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d5b867d8-3ad7-4aef-a0d8-6e8938ae7c68-multus-daemon-config\") pod \"multus-z7pg7\" (UID: \"d5b867d8-3ad7-4aef-a0d8-6e8938ae7c68\") " pod="openshift-multus/multus-z7pg7" Jul 29 18:34:27.267199 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.265567 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fb2816f1-d921-4e68-908b-ecc463ffb899-cni-binary-copy\") pod \"multus-additional-cni-plugins-xvbfs\" (UID: \"fb2816f1-d921-4e68-908b-ecc463ffb899\") " pod="openshift-multus/multus-additional-cni-plugins-xvbfs" Jul 29 18:34:27.267199 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.265578 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1727fb40-fbed-4984-91d4-a5890d0f4103-host-run-ovn-kubernetes\") pod \"ovnkube-node-v4n5x\" (UID: \"1727fb40-fbed-4984-91d4-a5890d0f4103\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4n5x" Jul 29 18:34:27.267199 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.265555 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1727fb40-fbed-4984-91d4-a5890d0f4103-host-kubelet\") pod \"ovnkube-node-v4n5x\" (UID: \"1727fb40-fbed-4984-91d4-a5890d0f4103\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4n5x" Jul 29 18:34:27.267199 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.265605 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1727fb40-fbed-4984-91d4-a5890d0f4103-ovnkube-config\") pod \"ovnkube-node-v4n5x\" (UID: \"1727fb40-fbed-4984-91d4-a5890d0f4103\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4n5x" Jul 29 18:34:27.267199 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.265520 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1727fb40-fbed-4984-91d4-a5890d0f4103-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-v4n5x\" (UID: \"1727fb40-fbed-4984-91d4-a5890d0f4103\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4n5x" Jul 29 18:34:27.267199 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.265634 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/aeadd3c9-a1ee-45a2-a683-3cfb51804716-socket-dir\") pod \"aws-ebs-csi-driver-node-249zt\" (UID: \"aeadd3c9-a1ee-45a2-a683-3cfb51804716\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-249zt" Jul 29 18:34:27.267199 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.265677 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c46b0e07-5bf9-469e-bde5-bc4708ce64f2-etc-kubernetes\") pod \"tuned-84dhd\" (UID: \"c46b0e07-5bf9-469e-bde5-bc4708ce64f2\") " pod="openshift-cluster-node-tuning-operator/tuned-84dhd" Jul 29 18:34:27.268007 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.265708 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d5b867d8-3ad7-4aef-a0d8-6e8938ae7c68-cnibin\") pod \"multus-z7pg7\" (UID: \"d5b867d8-3ad7-4aef-a0d8-6e8938ae7c68\") " pod="openshift-multus/multus-z7pg7" Jul 29 18:34:27.268007 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.265718 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1727fb40-fbed-4984-91d4-a5890d0f4103-host-run-ovn-kubernetes\") pod \"ovnkube-node-v4n5x\" (UID: \"1727fb40-fbed-4984-91d4-a5890d0f4103\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4n5x" Jul 29 18:34:27.268007 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.265731 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/13fd7656-9a1f-4221-a5ed-6ebecf3664c9-original-pull-secret\") pod \"global-pull-secret-syncer-zbwwp\" (UID: \"13fd7656-9a1f-4221-a5ed-6ebecf3664c9\") " pod="kube-system/global-pull-secret-syncer-zbwwp" Jul 29 18:34:27.268007 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.265751 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c46b0e07-5bf9-469e-bde5-bc4708ce64f2-run\") pod \"tuned-84dhd\" (UID: \"c46b0e07-5bf9-469e-bde5-bc4708ce64f2\") " pod="openshift-cluster-node-tuning-operator/tuned-84dhd" Jul 29 18:34:27.268007 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.265659 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/fb2816f1-d921-4e68-908b-ecc463ffb899-os-release\") pod \"multus-additional-cni-plugins-xvbfs\" (UID: \"fb2816f1-d921-4e68-908b-ecc463ffb899\") " pod="openshift-multus/multus-additional-cni-plugins-xvbfs" Jul 29 18:34:27.268007 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.265796 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/aeadd3c9-a1ee-45a2-a683-3cfb51804716-socket-dir\") pod \"aws-ebs-csi-driver-node-249zt\" (UID: \"aeadd3c9-a1ee-45a2-a683-3cfb51804716\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-249zt" Jul 29 18:34:27.268007 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.265794 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/aeadd3c9-a1ee-45a2-a683-3cfb51804716-etc-selinux\") pod \"aws-ebs-csi-driver-node-249zt\" (UID: \"aeadd3c9-a1ee-45a2-a683-3cfb51804716\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-249zt" Jul 29 18:34:27.268007 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.265797 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/fb2816f1-d921-4e68-908b-ecc463ffb899-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-xvbfs\" (UID: \"fb2816f1-d921-4e68-908b-ecc463ffb899\") " pod="openshift-multus/multus-additional-cni-plugins-xvbfs" Jul 29 18:34:27.268007 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.265842 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/07b2a3b0-456d-4696-92a3-968f3cf7d956-konnectivity-ca\") pod \"konnectivity-agent-69g8w\" (UID: \"07b2a3b0-456d-4696-92a3-968f3cf7d956\") " pod="kube-system/konnectivity-agent-69g8w" Jul 29 18:34:27.268007 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.265882 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1727fb40-fbed-4984-91d4-a5890d0f4103-host-cni-bin\") pod \"ovnkube-node-v4n5x\" (UID: \"1727fb40-fbed-4984-91d4-a5890d0f4103\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4n5x" Jul 29 18:34:27.268007 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.265905 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lmh28\" (UniqueName: \"kubernetes.io/projected/1727fb40-fbed-4984-91d4-a5890d0f4103-kube-api-access-lmh28\") pod \"ovnkube-node-v4n5x\" (UID: \"1727fb40-fbed-4984-91d4-a5890d0f4103\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4n5x" Jul 29 18:34:27.268007 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.265913 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c46b0e07-5bf9-469e-bde5-bc4708ce64f2-etc-sysctl-conf\") pod \"tuned-84dhd\" (UID: \"c46b0e07-5bf9-469e-bde5-bc4708ce64f2\") " pod="openshift-cluster-node-tuning-operator/tuned-84dhd" Jul 29 18:34:27.268007 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.265942 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fb2816f1-d921-4e68-908b-ecc463ffb899-tuning-conf-dir\") pod \"multus-additional-cni-plugins-xvbfs\" (UID: \"fb2816f1-d921-4e68-908b-ecc463ffb899\") " pod="openshift-multus/multus-additional-cni-plugins-xvbfs" Jul 29 18:34:27.268007 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.265971 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/13fd7656-9a1f-4221-a5ed-6ebecf3664c9-dbus\") pod \"global-pull-secret-syncer-zbwwp\" (UID: \"13fd7656-9a1f-4221-a5ed-6ebecf3664c9\") " pod="kube-system/global-pull-secret-syncer-zbwwp" Jul 29 18:34:27.268007 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.265969 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1727fb40-fbed-4984-91d4-a5890d0f4103-host-cni-bin\") pod \"ovnkube-node-v4n5x\" (UID: \"1727fb40-fbed-4984-91d4-a5890d0f4103\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4n5x" Jul 29 18:34:27.268007 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.266002 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d5b867d8-3ad7-4aef-a0d8-6e8938ae7c68-cnibin\") pod \"multus-z7pg7\" (UID: \"d5b867d8-3ad7-4aef-a0d8-6e8938ae7c68\") " pod="openshift-multus/multus-z7pg7" Jul 29 18:34:27.268007 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.266045 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c46b0e07-5bf9-469e-bde5-bc4708ce64f2-etc-kubernetes\") pod \"tuned-84dhd\" (UID: \"c46b0e07-5bf9-469e-bde5-bc4708ce64f2\") " pod="openshift-cluster-node-tuning-operator/tuned-84dhd" Jul 29 18:34:27.268762 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.266036 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c46b0e07-5bf9-469e-bde5-bc4708ce64f2-run\") pod \"tuned-84dhd\" (UID: \"c46b0e07-5bf9-469e-bde5-bc4708ce64f2\") " pod="openshift-cluster-node-tuning-operator/tuned-84dhd" Jul 29 18:34:27.268762 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.266111 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c46b0e07-5bf9-469e-bde5-bc4708ce64f2-host\") pod \"tuned-84dhd\" (UID: \"c46b0e07-5bf9-469e-bde5-bc4708ce64f2\") " pod="openshift-cluster-node-tuning-operator/tuned-84dhd" Jul 29 18:34:27.268762 ip-10-0-140-205 kubenswrapper[2560]: E0729 18:34:27.266128 2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 29 18:34:27.268762 ip-10-0-140-205 kubenswrapper[2560]: E0729 18:34:27.266189 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/13fd7656-9a1f-4221-a5ed-6ebecf3664c9-original-pull-secret podName:13fd7656-9a1f-4221-a5ed-6ebecf3664c9 nodeName:}" failed. No retries permitted until 2026-07-29 18:34:27.766169487 +0000 UTC m=+2.991838351 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/13fd7656-9a1f-4221-a5ed-6ebecf3664c9-original-pull-secret") pod "global-pull-secret-syncer-zbwwp" (UID: "13fd7656-9a1f-4221-a5ed-6ebecf3664c9") : object "kube-system"/"original-pull-secret" not registered Jul 29 18:34:27.268762 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.266207 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d5b867d8-3ad7-4aef-a0d8-6e8938ae7c68-multus-daemon-config\") pod \"multus-z7pg7\" (UID: \"d5b867d8-3ad7-4aef-a0d8-6e8938ae7c68\") " pod="openshift-multus/multus-z7pg7" Jul 29 18:34:27.268762 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.266075 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c46b0e07-5bf9-469e-bde5-bc4708ce64f2-etc-modprobe-d\") pod \"tuned-84dhd\" (UID: \"c46b0e07-5bf9-469e-bde5-bc4708ce64f2\") " pod="openshift-cluster-node-tuning-operator/tuned-84dhd" Jul 29 18:34:27.268762 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.266252 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c46b0e07-5bf9-469e-bde5-bc4708ce64f2-tmp\") pod \"tuned-84dhd\" (UID: \"c46b0e07-5bf9-469e-bde5-bc4708ce64f2\") " pod="openshift-cluster-node-tuning-operator/tuned-84dhd" Jul 29 18:34:27.268762 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.266257 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c46b0e07-5bf9-469e-bde5-bc4708ce64f2-etc-modprobe-d\") pod \"tuned-84dhd\" (UID: \"c46b0e07-5bf9-469e-bde5-bc4708ce64f2\") " pod="openshift-cluster-node-tuning-operator/tuned-84dhd" Jul 29 18:34:27.268762 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.266277 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1727fb40-fbed-4984-91d4-a5890d0f4103-run-ovn\") pod \"ovnkube-node-v4n5x\" (UID: \"1727fb40-fbed-4984-91d4-a5890d0f4103\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4n5x" Jul 29 18:34:27.268762 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.266281 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/13fd7656-9a1f-4221-a5ed-6ebecf3664c9-dbus\") pod \"global-pull-secret-syncer-zbwwp\" (UID: \"13fd7656-9a1f-4221-a5ed-6ebecf3664c9\") " pod="kube-system/global-pull-secret-syncer-zbwwp" Jul 29 18:34:27.268762 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.266300 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1727fb40-fbed-4984-91d4-a5890d0f4103-host-cni-netd\") pod \"ovnkube-node-v4n5x\" (UID: \"1727fb40-fbed-4984-91d4-a5890d0f4103\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4n5x" Jul 29 18:34:27.268762 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.266331 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1727fb40-fbed-4984-91d4-a5890d0f4103-env-overrides\") pod \"ovnkube-node-v4n5x\" (UID: \"1727fb40-fbed-4984-91d4-a5890d0f4103\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4n5x" Jul 29 18:34:27.268762 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.266375 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/aeadd3c9-a1ee-45a2-a683-3cfb51804716-sys-fs\") pod \"aws-ebs-csi-driver-node-249zt\" (UID: \"aeadd3c9-a1ee-45a2-a683-3cfb51804716\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-249zt" Jul 29 18:34:27.268762 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.266395 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1727fb40-fbed-4984-91d4-a5890d0f4103-run-ovn\") pod \"ovnkube-node-v4n5x\" (UID: \"1727fb40-fbed-4984-91d4-a5890d0f4103\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4n5x" Jul 29 18:34:27.268762 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.266400 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c46b0e07-5bf9-469e-bde5-bc4708ce64f2-var-lib-kubelet\") pod \"tuned-84dhd\" (UID: \"c46b0e07-5bf9-469e-bde5-bc4708ce64f2\") " pod="openshift-cluster-node-tuning-operator/tuned-84dhd" Jul 29 18:34:27.268762 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.266423 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/96c83b74-1875-4a28-8e0e-1470f61a31c9-serviceca\") pod \"node-ca-nvp4r\" (UID: \"96c83b74-1875-4a28-8e0e-1470f61a31c9\") " pod="openshift-image-registry/node-ca-nvp4r" Jul 29 18:34:27.268762 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.266444 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d5b867d8-3ad7-4aef-a0d8-6e8938ae7c68-host-run-netns\") pod \"multus-z7pg7\" (UID: \"d5b867d8-3ad7-4aef-a0d8-6e8938ae7c68\") " pod="openshift-multus/multus-z7pg7" Jul 29 18:34:27.268762 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.266466 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d5b867d8-3ad7-4aef-a0d8-6e8938ae7c68-host-var-lib-kubelet\") pod \"multus-z7pg7\" (UID: \"d5b867d8-3ad7-4aef-a0d8-6e8938ae7c68\") " pod="openshift-multus/multus-z7pg7" Jul 29 18:34:27.269510 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.266475 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1727fb40-fbed-4984-91d4-a5890d0f4103-ovnkube-config\") pod \"ovnkube-node-v4n5x\" (UID: \"1727fb40-fbed-4984-91d4-a5890d0f4103\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4n5x" Jul 29 18:34:27.269510 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.266490 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1727fb40-fbed-4984-91d4-a5890d0f4103-host-slash\") pod \"ovnkube-node-v4n5x\" (UID: \"1727fb40-fbed-4984-91d4-a5890d0f4103\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4n5x" Jul 29 18:34:27.269510 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.266526 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1727fb40-fbed-4984-91d4-a5890d0f4103-host-slash\") pod \"ovnkube-node-v4n5x\" (UID: \"1727fb40-fbed-4984-91d4-a5890d0f4103\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4n5x" Jul 29 18:34:27.269510 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.266525 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1727fb40-fbed-4984-91d4-a5890d0f4103-ovnkube-script-lib\") pod \"ovnkube-node-v4n5x\" (UID: \"1727fb40-fbed-4984-91d4-a5890d0f4103\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4n5x" Jul 29 18:34:27.269510 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.266579 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/aeadd3c9-a1ee-45a2-a683-3cfb51804716-sys-fs\") pod \"aws-ebs-csi-driver-node-249zt\" (UID: \"aeadd3c9-a1ee-45a2-a683-3cfb51804716\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-249zt" Jul 29 18:34:27.269510 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.266588 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-blmpz\" (UniqueName: \"kubernetes.io/projected/aeadd3c9-a1ee-45a2-a683-3cfb51804716-kube-api-access-blmpz\") pod \"aws-ebs-csi-driver-node-249zt\" (UID: \"aeadd3c9-a1ee-45a2-a683-3cfb51804716\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-249zt" Jul 29 18:34:27.269510 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.266614 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pnkgr\" (UniqueName: \"kubernetes.io/projected/90d823ce-32c1-4cf9-8165-161155b5c5d1-kube-api-access-pnkgr\") pod \"iptables-alerter-nxn8k\" (UID: \"90d823ce-32c1-4cf9-8165-161155b5c5d1\") " pod="openshift-network-operator/iptables-alerter-nxn8k" Jul 29 18:34:27.269510 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.266627 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c46b0e07-5bf9-469e-bde5-bc4708ce64f2-var-lib-kubelet\") pod \"tuned-84dhd\" (UID: \"c46b0e07-5bf9-469e-bde5-bc4708ce64f2\") " pod="openshift-cluster-node-tuning-operator/tuned-84dhd" Jul 29 18:34:27.269510 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.266662 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d5b867d8-3ad7-4aef-a0d8-6e8938ae7c68-cni-binary-copy\") pod \"multus-z7pg7\" (UID: \"d5b867d8-3ad7-4aef-a0d8-6e8938ae7c68\") " pod="openshift-multus/multus-z7pg7" Jul 29 18:34:27.269510 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.266680 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/fb2816f1-d921-4e68-908b-ecc463ffb899-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-xvbfs\" (UID: \"fb2816f1-d921-4e68-908b-ecc463ffb899\") " pod="openshift-multus/multus-additional-cni-plugins-xvbfs" Jul 29 18:34:27.269510 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.266686 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d5b867d8-3ad7-4aef-a0d8-6e8938ae7c68-etc-kubernetes\") pod \"multus-z7pg7\" (UID: \"d5b867d8-3ad7-4aef-a0d8-6e8938ae7c68\") " pod="openshift-multus/multus-z7pg7" Jul 29 18:34:27.269510 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.266738 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phzf8\" (UniqueName: \"kubernetes.io/projected/88336825-e84e-4595-9f68-f4d6b4e6edda-kube-api-access-phzf8\") pod \"network-check-target-8qdhk\" (UID: \"88336825-e84e-4595-9f68-f4d6b4e6edda\") " pod="openshift-network-diagnostics/network-check-target-8qdhk" Jul 29 18:34:27.269510 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.266767 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1727fb40-fbed-4984-91d4-a5890d0f4103-systemd-units\") pod \"ovnkube-node-v4n5x\" (UID: \"1727fb40-fbed-4984-91d4-a5890d0f4103\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4n5x" Jul 29 18:34:27.269510 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.266798 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/aeadd3c9-a1ee-45a2-a683-3cfb51804716-registration-dir\") pod \"aws-ebs-csi-driver-node-249zt\" (UID: \"aeadd3c9-a1ee-45a2-a683-3cfb51804716\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-249zt" Jul 29 18:34:27.269510 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.266828 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d5b867d8-3ad7-4aef-a0d8-6e8938ae7c68-multus-cni-dir\") pod \"multus-z7pg7\" (UID: \"d5b867d8-3ad7-4aef-a0d8-6e8938ae7c68\") " pod="openshift-multus/multus-z7pg7" Jul 29 18:34:27.269510 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.266839 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1727fb40-fbed-4984-91d4-a5890d0f4103-env-overrides\") pod \"ovnkube-node-v4n5x\" (UID: \"1727fb40-fbed-4984-91d4-a5890d0f4103\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4n5x" Jul 29 18:34:27.269510 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.266858 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khhzl\" (UniqueName: \"kubernetes.io/projected/d5b867d8-3ad7-4aef-a0d8-6e8938ae7c68-kube-api-access-khhzl\") pod \"multus-z7pg7\" (UID: \"d5b867d8-3ad7-4aef-a0d8-6e8938ae7c68\") " pod="openshift-multus/multus-z7pg7" Jul 29 18:34:27.270237 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.266889 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1727fb40-fbed-4984-91d4-a5890d0f4103-log-socket\") pod \"ovnkube-node-v4n5x\" (UID: \"1727fb40-fbed-4984-91d4-a5890d0f4103\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4n5x" Jul 29 18:34:27.270237 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.266738 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d5b867d8-3ad7-4aef-a0d8-6e8938ae7c68-etc-kubernetes\") pod \"multus-z7pg7\" (UID: \"d5b867d8-3ad7-4aef-a0d8-6e8938ae7c68\") " pod="openshift-multus/multus-z7pg7" Jul 29 18:34:27.270237 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.266981 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1727fb40-fbed-4984-91d4-a5890d0f4103-host-cni-netd\") pod \"ovnkube-node-v4n5x\" (UID: \"1727fb40-fbed-4984-91d4-a5890d0f4103\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4n5x" Jul 29 18:34:27.270237 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.267012 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1727fb40-fbed-4984-91d4-a5890d0f4103-ovn-node-metrics-cert\") pod \"ovnkube-node-v4n5x\" (UID: \"1727fb40-fbed-4984-91d4-a5890d0f4103\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4n5x" Jul 29 18:34:27.270237 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.267051 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/96c83b74-1875-4a28-8e0e-1470f61a31c9-serviceca\") pod \"node-ca-nvp4r\" (UID: \"96c83b74-1875-4a28-8e0e-1470f61a31c9\") " pod="openshift-image-registry/node-ca-nvp4r" Jul 29 18:34:27.270237 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.267068 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c46b0e07-5bf9-469e-bde5-bc4708ce64f2-etc-sysconfig\") pod \"tuned-84dhd\" (UID: \"c46b0e07-5bf9-469e-bde5-bc4708ce64f2\") " pod="openshift-cluster-node-tuning-operator/tuned-84dhd" Jul 29 18:34:27.270237 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.267134 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/aeadd3c9-a1ee-45a2-a683-3cfb51804716-registration-dir\") pod \"aws-ebs-csi-driver-node-249zt\" (UID: \"aeadd3c9-a1ee-45a2-a683-3cfb51804716\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-249zt" Jul 29 18:34:27.270237 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.267143 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c46b0e07-5bf9-469e-bde5-bc4708ce64f2-etc-sysctl-d\") pod \"tuned-84dhd\" (UID: \"c46b0e07-5bf9-469e-bde5-bc4708ce64f2\") " pod="openshift-cluster-node-tuning-operator/tuned-84dhd" Jul 29 18:34:27.270237 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.267231 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c46b0e07-5bf9-469e-bde5-bc4708ce64f2-etc-sysctl-d\") pod \"tuned-84dhd\" (UID: \"c46b0e07-5bf9-469e-bde5-bc4708ce64f2\") " pod="openshift-cluster-node-tuning-operator/tuned-84dhd" Jul 29 18:34:27.270237 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.267316 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1727fb40-fbed-4984-91d4-a5890d0f4103-ovnkube-script-lib\") pod \"ovnkube-node-v4n5x\" (UID: \"1727fb40-fbed-4984-91d4-a5890d0f4103\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4n5x" Jul 29 18:34:27.270237 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.267322 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d5b867d8-3ad7-4aef-a0d8-6e8938ae7c68-multus-cni-dir\") pod \"multus-z7pg7\" (UID: \"d5b867d8-3ad7-4aef-a0d8-6e8938ae7c68\") " pod="openshift-multus/multus-z7pg7" Jul 29 18:34:27.270237 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.267491 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1727fb40-fbed-4984-91d4-a5890d0f4103-log-socket\") pod \"ovnkube-node-v4n5x\" (UID: \"1727fb40-fbed-4984-91d4-a5890d0f4103\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4n5x" Jul 29 18:34:27.270237 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.267551 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d5b867d8-3ad7-4aef-a0d8-6e8938ae7c68-host-var-lib-kubelet\") pod \"multus-z7pg7\" (UID: \"d5b867d8-3ad7-4aef-a0d8-6e8938ae7c68\") " pod="openshift-multus/multus-z7pg7" Jul 29 18:34:27.270237 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.267668 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/07b2a3b0-456d-4696-92a3-968f3cf7d956-konnectivity-ca\") pod \"konnectivity-agent-69g8w\" (UID: \"07b2a3b0-456d-4696-92a3-968f3cf7d956\") " pod="kube-system/konnectivity-agent-69g8w" Jul 29 18:34:27.270237 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.267726 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d5b867d8-3ad7-4aef-a0d8-6e8938ae7c68-host-run-netns\") pod \"multus-z7pg7\" (UID: \"d5b867d8-3ad7-4aef-a0d8-6e8938ae7c68\") " pod="openshift-multus/multus-z7pg7" Jul 29 18:34:27.270237 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.267771 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c46b0e07-5bf9-469e-bde5-bc4708ce64f2-etc-sysconfig\") pod \"tuned-84dhd\" (UID: \"c46b0e07-5bf9-469e-bde5-bc4708ce64f2\") " pod="openshift-cluster-node-tuning-operator/tuned-84dhd" Jul 29 18:34:27.270237 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.267796 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/fb2816f1-d921-4e68-908b-ecc463ffb899-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-xvbfs\" (UID: \"fb2816f1-d921-4e68-908b-ecc463ffb899\") " pod="openshift-multus/multus-additional-cni-plugins-xvbfs" Jul 29 18:34:27.270237 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.267832 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvv7r\" (UniqueName: \"kubernetes.io/projected/70307ff7-a0ca-4a2b-95be-94a5dde5d96e-kube-api-access-zvv7r\") pod \"network-metrics-daemon-mrshg\" (UID: \"70307ff7-a0ca-4a2b-95be-94a5dde5d96e\") " pod="openshift-multus/network-metrics-daemon-mrshg" Jul 29 18:34:27.271034 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.267942 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d5b867d8-3ad7-4aef-a0d8-6e8938ae7c68-cni-binary-copy\") pod \"multus-z7pg7\" (UID: \"d5b867d8-3ad7-4aef-a0d8-6e8938ae7c68\") " pod="openshift-multus/multus-z7pg7" Jul 29 18:34:27.271034 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.268103 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1727fb40-fbed-4984-91d4-a5890d0f4103-systemd-units\") pod \"ovnkube-node-v4n5x\" (UID: \"1727fb40-fbed-4984-91d4-a5890d0f4103\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4n5x" Jul 29 18:34:27.271034 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.268413 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/fb2816f1-d921-4e68-908b-ecc463ffb899-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-xvbfs\" (UID: \"fb2816f1-d921-4e68-908b-ecc463ffb899\") " pod="openshift-multus/multus-additional-cni-plugins-xvbfs" Jul 29 18:34:27.271034 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.268500 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/90d823ce-32c1-4cf9-8165-161155b5c5d1-iptables-alerter-script\") pod \"iptables-alerter-nxn8k\" (UID: \"90d823ce-32c1-4cf9-8165-161155b5c5d1\") " pod="openshift-network-operator/iptables-alerter-nxn8k" Jul 29 18:34:27.271034 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.268731 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c46b0e07-5bf9-469e-bde5-bc4708ce64f2-etc-tuned\") pod \"tuned-84dhd\" (UID: \"c46b0e07-5bf9-469e-bde5-bc4708ce64f2\") " pod="openshift-cluster-node-tuning-operator/tuned-84dhd" Jul 29 18:34:27.271034 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.268772 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c46b0e07-5bf9-469e-bde5-bc4708ce64f2-tmp\") pod \"tuned-84dhd\" (UID: \"c46b0e07-5bf9-469e-bde5-bc4708ce64f2\") " pod="openshift-cluster-node-tuning-operator/tuned-84dhd" Jul 29 18:34:27.271034 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.269369 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/07b2a3b0-456d-4696-92a3-968f3cf7d956-agent-certs\") pod \"konnectivity-agent-69g8w\" (UID: \"07b2a3b0-456d-4696-92a3-968f3cf7d956\") " pod="kube-system/konnectivity-agent-69g8w" Jul 29 18:34:27.271034 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.270339 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1727fb40-fbed-4984-91d4-a5890d0f4103-ovn-node-metrics-cert\") pod \"ovnkube-node-v4n5x\" (UID: \"1727fb40-fbed-4984-91d4-a5890d0f4103\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4n5x" Jul 29 18:34:27.274100 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.274078 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vsk9g\" (UniqueName: \"kubernetes.io/projected/c46b0e07-5bf9-469e-bde5-bc4708ce64f2-kube-api-access-vsk9g\") pod \"tuned-84dhd\" (UID: \"c46b0e07-5bf9-469e-bde5-bc4708ce64f2\") " pod="openshift-cluster-node-tuning-operator/tuned-84dhd" Jul 29 18:34:27.274533 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.274509 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tsrsw\" (UniqueName: \"kubernetes.io/projected/2fbfc68e-382a-4371-8291-ce71b8eec8ce-kube-api-access-tsrsw\") pod \"node-resolver-nnlkl\" (UID: \"2fbfc68e-382a-4371-8291-ce71b8eec8ce\") " pod="openshift-dns/node-resolver-nnlkl" Jul 29 18:34:27.274699 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.274637 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4hp6\" (UniqueName: \"kubernetes.io/projected/fb2816f1-d921-4e68-908b-ecc463ffb899-kube-api-access-l4hp6\") pod \"multus-additional-cni-plugins-xvbfs\" (UID: \"fb2816f1-d921-4e68-908b-ecc463ffb899\") " pod="openshift-multus/multus-additional-cni-plugins-xvbfs" Jul 29 18:34:27.275689 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.275665 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5lxr\" (UniqueName: \"kubernetes.io/projected/96c83b74-1875-4a28-8e0e-1470f61a31c9-kube-api-access-q5lxr\") pod \"node-ca-nvp4r\" (UID: \"96c83b74-1875-4a28-8e0e-1470f61a31c9\") " pod="openshift-image-registry/node-ca-nvp4r" Jul 29 18:34:27.276309 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.276288 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvv7r\" (UniqueName: \"kubernetes.io/projected/70307ff7-a0ca-4a2b-95be-94a5dde5d96e-kube-api-access-zvv7r\") pod \"network-metrics-daemon-mrshg\" (UID: \"70307ff7-a0ca-4a2b-95be-94a5dde5d96e\") " pod="openshift-multus/network-metrics-daemon-mrshg" Jul 29 18:34:27.276821 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.276802 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pnkgr\" (UniqueName: \"kubernetes.io/projected/90d823ce-32c1-4cf9-8165-161155b5c5d1-kube-api-access-pnkgr\") pod \"iptables-alerter-nxn8k\" (UID: \"90d823ce-32c1-4cf9-8165-161155b5c5d1\") " pod="openshift-network-operator/iptables-alerter-nxn8k" Jul 29 18:34:27.277970 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.277946 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-blmpz\" (UniqueName: \"kubernetes.io/projected/aeadd3c9-a1ee-45a2-a683-3cfb51804716-kube-api-access-blmpz\") pod \"aws-ebs-csi-driver-node-249zt\" (UID: \"aeadd3c9-a1ee-45a2-a683-3cfb51804716\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-249zt" Jul 29 18:34:27.278436 ip-10-0-140-205 kubenswrapper[2560]: E0729 18:34:27.278419 2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 29 18:34:27.278526 ip-10-0-140-205 kubenswrapper[2560]: E0729 18:34:27.278441 2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 29 18:34:27.278526 ip-10-0-140-205 kubenswrapper[2560]: E0729 18:34:27.278453 2560 projected.go:196] Error preparing data for projected volume kube-api-access-phzf8 for pod openshift-network-diagnostics/network-check-target-8qdhk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 18:34:27.278526 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.278463 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmh28\" (UniqueName: \"kubernetes.io/projected/1727fb40-fbed-4984-91d4-a5890d0f4103-kube-api-access-lmh28\") pod \"ovnkube-node-v4n5x\" (UID: \"1727fb40-fbed-4984-91d4-a5890d0f4103\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4n5x" Jul 29 18:34:27.278682 ip-10-0-140-205 kubenswrapper[2560]: E0729 18:34:27.278548 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/88336825-e84e-4595-9f68-f4d6b4e6edda-kube-api-access-phzf8 podName:88336825-e84e-4595-9f68-f4d6b4e6edda nodeName:}" failed. No retries permitted until 2026-07-29 18:34:27.778532206 +0000 UTC m=+3.004201089 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-phzf8" (UniqueName: "kubernetes.io/projected/88336825-e84e-4595-9f68-f4d6b4e6edda-kube-api-access-phzf8") pod "network-check-target-8qdhk" (UID: "88336825-e84e-4595-9f68-f4d6b4e6edda") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 18:34:27.280144 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.280125 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-khhzl\" (UniqueName: \"kubernetes.io/projected/d5b867d8-3ad7-4aef-a0d8-6e8938ae7c68-kube-api-access-khhzl\") pod \"multus-z7pg7\" (UID: \"d5b867d8-3ad7-4aef-a0d8-6e8938ae7c68\") " pod="openshift-multus/multus-z7pg7" Jul 29 18:34:27.352261 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.352230 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-205.ec2.internal" event={"ID":"89287a5239330a896427ad6104c0beff","Type":"ContainerStarted","Data":"8562741694e2748a6039f09b2536fb6d416c7f5a2412ed3b65281cc0a77c6235"} Jul 29 18:34:27.448860 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.448796 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-nxn8k" Jul 29 18:34:27.456739 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.456720 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-84dhd" Jul 29 18:34:27.457083 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:27.457057 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90d823ce_32c1_4cf9_8165_161155b5c5d1.slice/crio-3c817b7450d40c0d969a20186acb68ddd2b06392fb2743d759ae447404d68054 WatchSource:0}: Error finding container 3c817b7450d40c0d969a20186acb68ddd2b06392fb2743d759ae447404d68054: Status 404 returned error can't find the container with id 3c817b7450d40c0d969a20186acb68ddd2b06392fb2743d759ae447404d68054 Jul 29 18:34:27.464407 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.464386 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-xvbfs" Jul 29 18:34:27.464652 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:27.464634 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc46b0e07_5bf9_469e_bde5_bc4708ce64f2.slice/crio-6f1f962ee338633849225a2b4578b80d729688d8b709836baa61350411341b09 WatchSource:0}: Error finding container 6f1f962ee338633849225a2b4578b80d729688d8b709836baa61350411341b09: Status 404 returned error can't find the container with id 6f1f962ee338633849225a2b4578b80d729688d8b709836baa61350411341b09 Jul 29 18:34:27.467850 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.467834 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-z7pg7" Jul 29 18:34:27.471616 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:27.471596 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb2816f1_d921_4e68_908b_ecc463ffb899.slice/crio-a7042ce713ab55d43386365b4a68020ff5e19c52c47e8dc7c51ee671ffce0e7c WatchSource:0}: Error finding container a7042ce713ab55d43386365b4a68020ff5e19c52c47e8dc7c51ee671ffce0e7c: Status 404 returned error can't find the container with id a7042ce713ab55d43386365b4a68020ff5e19c52c47e8dc7c51ee671ffce0e7c Jul 29 18:34:27.472651 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.472637 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-nnlkl" Jul 29 18:34:27.475799 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:27.475777 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5b867d8_3ad7_4aef_a0d8_6e8938ae7c68.slice/crio-6dbb0b4a3ab1b9a8a1a17ac13dc25509a05df6c4179fb6b66d47c60bcb295a46 WatchSource:0}: Error finding container 6dbb0b4a3ab1b9a8a1a17ac13dc25509a05df6c4179fb6b66d47c60bcb295a46: Status 404 returned error can't find the container with id 6dbb0b4a3ab1b9a8a1a17ac13dc25509a05df6c4179fb6b66d47c60bcb295a46 Jul 29 18:34:27.478640 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.478549 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-69g8w" Jul 29 18:34:27.480565 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:27.480543 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fbfc68e_382a_4371_8291_ce71b8eec8ce.slice/crio-26563f9516789c1965aacc7f826ce392c91dadad5d4f96f3426149d4c17a4a7e WatchSource:0}: Error finding container 26563f9516789c1965aacc7f826ce392c91dadad5d4f96f3426149d4c17a4a7e: Status 404 returned error can't find the container with id 26563f9516789c1965aacc7f826ce392c91dadad5d4f96f3426149d4c17a4a7e Jul 29 18:34:27.483466 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.483448 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-v4n5x" Jul 29 18:34:27.486603 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:27.486582 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07b2a3b0_456d_4696_92a3_968f3cf7d956.slice/crio-307558172f558a751061e5f14c7f329c5fee2fb091989a76821235ddce067efc WatchSource:0}: Error finding container 307558172f558a751061e5f14c7f329c5fee2fb091989a76821235ddce067efc: Status 404 returned error can't find the container with id 307558172f558a751061e5f14c7f329c5fee2fb091989a76821235ddce067efc Jul 29 18:34:27.489577 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.489560 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-nvp4r" Jul 29 18:34:27.491006 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:27.490987 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1727fb40_fbed_4984_91d4_a5890d0f4103.slice/crio-cf989dc4887662923ed9cda78e26e382f1774cc0b22977dba587eef8203c665f WatchSource:0}: Error finding container cf989dc4887662923ed9cda78e26e382f1774cc0b22977dba587eef8203c665f: Status 404 returned error can't find the container with id cf989dc4887662923ed9cda78e26e382f1774cc0b22977dba587eef8203c665f Jul 29 18:34:27.492779 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.492765 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-249zt" Jul 29 18:34:27.497959 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:27.497938 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96c83b74_1875_4a28_8e0e_1470f61a31c9.slice/crio-011d0e4d663388f840f50068407a9edcea4b35b35ff36808f349297e1c9d2e06 WatchSource:0}: Error finding container 011d0e4d663388f840f50068407a9edcea4b35b35ff36808f349297e1c9d2e06: Status 404 returned error can't find the container with id 011d0e4d663388f840f50068407a9edcea4b35b35ff36808f349297e1c9d2e06 Jul 29 18:34:27.501916 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:34:27.501898 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaeadd3c9_a1ee_45a2_a683_3cfb51804716.slice/crio-7e549ad61593ec48a1190afbc73b34b7992603c9997fd590911ff9ae30c268ba WatchSource:0}: Error finding container 7e549ad61593ec48a1190afbc73b34b7992603c9997fd590911ff9ae30c268ba: Status 404 returned error can't find the container with id 7e549ad61593ec48a1190afbc73b34b7992603c9997fd590911ff9ae30c268ba Jul 29 18:34:27.771324 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.770635 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70307ff7-a0ca-4a2b-95be-94a5dde5d96e-metrics-certs\") pod \"network-metrics-daemon-mrshg\" (UID: \"70307ff7-a0ca-4a2b-95be-94a5dde5d96e\") " pod="openshift-multus/network-metrics-daemon-mrshg" Jul 29 18:34:27.771324 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.770704 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/13fd7656-9a1f-4221-a5ed-6ebecf3664c9-original-pull-secret\") pod \"global-pull-secret-syncer-zbwwp\" (UID: \"13fd7656-9a1f-4221-a5ed-6ebecf3664c9\") " pod="kube-system/global-pull-secret-syncer-zbwwp" Jul 29 18:34:27.771324 ip-10-0-140-205 kubenswrapper[2560]: E0729 18:34:27.770836 2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 29 18:34:27.771324 ip-10-0-140-205 kubenswrapper[2560]: E0729 18:34:27.770893 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/13fd7656-9a1f-4221-a5ed-6ebecf3664c9-original-pull-secret podName:13fd7656-9a1f-4221-a5ed-6ebecf3664c9 nodeName:}" failed. No retries permitted until 2026-07-29 18:34:28.770874084 +0000 UTC m=+3.996542952 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/13fd7656-9a1f-4221-a5ed-6ebecf3664c9-original-pull-secret") pod "global-pull-secret-syncer-zbwwp" (UID: "13fd7656-9a1f-4221-a5ed-6ebecf3664c9") : object "kube-system"/"original-pull-secret" not registered Jul 29 18:34:27.771324 ip-10-0-140-205 kubenswrapper[2560]: E0729 18:34:27.770958 2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 18:34:27.771324 ip-10-0-140-205 kubenswrapper[2560]: E0729 18:34:27.771008 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/70307ff7-a0ca-4a2b-95be-94a5dde5d96e-metrics-certs podName:70307ff7-a0ca-4a2b-95be-94a5dde5d96e nodeName:}" failed. No retries permitted until 2026-07-29 18:34:28.770996185 +0000 UTC m=+3.996665048 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/70307ff7-a0ca-4a2b-95be-94a5dde5d96e-metrics-certs") pod "network-metrics-daemon-mrshg" (UID: "70307ff7-a0ca-4a2b-95be-94a5dde5d96e") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 18:34:27.871239 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:27.871204 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phzf8\" (UniqueName: \"kubernetes.io/projected/88336825-e84e-4595-9f68-f4d6b4e6edda-kube-api-access-phzf8\") pod \"network-check-target-8qdhk\" (UID: \"88336825-e84e-4595-9f68-f4d6b4e6edda\") " pod="openshift-network-diagnostics/network-check-target-8qdhk" Jul 29 18:34:27.871421 ip-10-0-140-205 kubenswrapper[2560]: E0729 18:34:27.871370 2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 29 18:34:27.871421 ip-10-0-140-205 kubenswrapper[2560]: E0729 18:34:27.871392 2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 29 18:34:27.871421 ip-10-0-140-205 kubenswrapper[2560]: E0729 18:34:27.871403 2560 projected.go:196] Error preparing data for projected volume kube-api-access-phzf8 for pod openshift-network-diagnostics/network-check-target-8qdhk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 18:34:27.871589 ip-10-0-140-205 kubenswrapper[2560]: E0729 18:34:27.871469 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/88336825-e84e-4595-9f68-f4d6b4e6edda-kube-api-access-phzf8 podName:88336825-e84e-4595-9f68-f4d6b4e6edda nodeName:}" failed. No retries permitted until 2026-07-29 18:34:28.871440509 +0000 UTC m=+4.097109362 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-phzf8" (UniqueName: "kubernetes.io/projected/88336825-e84e-4595-9f68-f4d6b4e6edda-kube-api-access-phzf8") pod "network-check-target-8qdhk" (UID: "88336825-e84e-4595-9f68-f4d6b4e6edda") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 18:34:28.203846 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:28.203803 2560 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-28 18:29:27 +0000 UTC" deadline="2028-01-10 00:38:21.912528338 +0000 UTC" Jul 29 18:34:28.203846 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:28.203845 2560 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12702h3m53.708688388s" Jul 29 18:34:28.347913 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:28.347841 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mrshg" Jul 29 18:34:28.348060 ip-10-0-140-205 kubenswrapper[2560]: E0729 18:34:28.347990 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mrshg" podUID="70307ff7-a0ca-4a2b-95be-94a5dde5d96e" Jul 29 18:34:28.348118 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:28.348079 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zbwwp" Jul 29 18:34:28.348184 ip-10-0-140-205 kubenswrapper[2560]: E0729 18:34:28.348161 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zbwwp" podUID="13fd7656-9a1f-4221-a5ed-6ebecf3664c9" Jul 29 18:34:28.378249 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:28.378212 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-z7pg7" event={"ID":"d5b867d8-3ad7-4aef-a0d8-6e8938ae7c68","Type":"ContainerStarted","Data":"6dbb0b4a3ab1b9a8a1a17ac13dc25509a05df6c4179fb6b66d47c60bcb295a46"} Jul 29 18:34:28.385416 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:28.385385 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xvbfs" event={"ID":"fb2816f1-d921-4e68-908b-ecc463ffb899","Type":"ContainerStarted","Data":"a7042ce713ab55d43386365b4a68020ff5e19c52c47e8dc7c51ee671ffce0e7c"} Jul 29 18:34:28.407362 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:28.407262 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-84dhd" event={"ID":"c46b0e07-5bf9-469e-bde5-bc4708ce64f2","Type":"ContainerStarted","Data":"6f1f962ee338633849225a2b4578b80d729688d8b709836baa61350411341b09"} Jul 29 18:34:28.410582 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:28.410547 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-nxn8k" event={"ID":"90d823ce-32c1-4cf9-8165-161155b5c5d1","Type":"ContainerStarted","Data":"3c817b7450d40c0d969a20186acb68ddd2b06392fb2743d759ae447404d68054"} Jul 29 18:34:28.420636 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:28.420597 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-249zt" event={"ID":"aeadd3c9-a1ee-45a2-a683-3cfb51804716","Type":"ContainerStarted","Data":"7e549ad61593ec48a1190afbc73b34b7992603c9997fd590911ff9ae30c268ba"} Jul 29 18:34:28.430577 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:28.430548 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-nvp4r" event={"ID":"96c83b74-1875-4a28-8e0e-1470f61a31c9","Type":"ContainerStarted","Data":"011d0e4d663388f840f50068407a9edcea4b35b35ff36808f349297e1c9d2e06"} Jul 29 18:34:28.443965 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:28.443933 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-v4n5x" event={"ID":"1727fb40-fbed-4984-91d4-a5890d0f4103","Type":"ContainerStarted","Data":"cf989dc4887662923ed9cda78e26e382f1774cc0b22977dba587eef8203c665f"} Jul 29 18:34:28.460000 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:28.459924 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-69g8w" event={"ID":"07b2a3b0-456d-4696-92a3-968f3cf7d956","Type":"ContainerStarted","Data":"307558172f558a751061e5f14c7f329c5fee2fb091989a76821235ddce067efc"} Jul 29 18:34:28.466715 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:28.466680 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-nnlkl" event={"ID":"2fbfc68e-382a-4371-8291-ce71b8eec8ce","Type":"ContainerStarted","Data":"26563f9516789c1965aacc7f826ce392c91dadad5d4f96f3426149d4c17a4a7e"} Jul 29 18:34:28.779679 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:28.779639 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/13fd7656-9a1f-4221-a5ed-6ebecf3664c9-original-pull-secret\") pod \"global-pull-secret-syncer-zbwwp\" (UID: \"13fd7656-9a1f-4221-a5ed-6ebecf3664c9\") " pod="kube-system/global-pull-secret-syncer-zbwwp" Jul 29 18:34:28.779862 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:28.779740 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70307ff7-a0ca-4a2b-95be-94a5dde5d96e-metrics-certs\") pod \"network-metrics-daemon-mrshg\" (UID: \"70307ff7-a0ca-4a2b-95be-94a5dde5d96e\") " pod="openshift-multus/network-metrics-daemon-mrshg" Jul 29 18:34:28.783768 ip-10-0-140-205 kubenswrapper[2560]: E0729 18:34:28.783739 2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 29 18:34:28.783901 ip-10-0-140-205 kubenswrapper[2560]: E0729 18:34:28.783843 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/13fd7656-9a1f-4221-a5ed-6ebecf3664c9-original-pull-secret podName:13fd7656-9a1f-4221-a5ed-6ebecf3664c9 nodeName:}" failed. No retries permitted until 2026-07-29 18:34:30.783822273 +0000 UTC m=+6.009491129 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/13fd7656-9a1f-4221-a5ed-6ebecf3664c9-original-pull-secret") pod "global-pull-secret-syncer-zbwwp" (UID: "13fd7656-9a1f-4221-a5ed-6ebecf3664c9") : object "kube-system"/"original-pull-secret" not registered Jul 29 18:34:28.788599 ip-10-0-140-205 kubenswrapper[2560]: E0729 18:34:28.784490 2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 18:34:28.788599 ip-10-0-140-205 kubenswrapper[2560]: E0729 18:34:28.784559 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/70307ff7-a0ca-4a2b-95be-94a5dde5d96e-metrics-certs podName:70307ff7-a0ca-4a2b-95be-94a5dde5d96e nodeName:}" failed. No retries permitted until 2026-07-29 18:34:30.784542007 +0000 UTC m=+6.010210860 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/70307ff7-a0ca-4a2b-95be-94a5dde5d96e-metrics-certs") pod "network-metrics-daemon-mrshg" (UID: "70307ff7-a0ca-4a2b-95be-94a5dde5d96e") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 18:34:28.881513 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:28.881368 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phzf8\" (UniqueName: \"kubernetes.io/projected/88336825-e84e-4595-9f68-f4d6b4e6edda-kube-api-access-phzf8\") pod \"network-check-target-8qdhk\" (UID: \"88336825-e84e-4595-9f68-f4d6b4e6edda\") " pod="openshift-network-diagnostics/network-check-target-8qdhk" Jul 29 18:34:28.881711 ip-10-0-140-205 kubenswrapper[2560]: E0729 18:34:28.881520 2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 29 18:34:28.881711 ip-10-0-140-205 kubenswrapper[2560]: E0729 18:34:28.881549 2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 29 18:34:28.881711 ip-10-0-140-205 kubenswrapper[2560]: E0729 18:34:28.881564 2560 projected.go:196] Error preparing data for projected volume kube-api-access-phzf8 for pod openshift-network-diagnostics/network-check-target-8qdhk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 18:34:28.881711 ip-10-0-140-205 kubenswrapper[2560]: E0729 18:34:28.881628 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/88336825-e84e-4595-9f68-f4d6b4e6edda-kube-api-access-phzf8 podName:88336825-e84e-4595-9f68-f4d6b4e6edda nodeName:}" failed. No retries permitted until 2026-07-29 18:34:30.881609082 +0000 UTC m=+6.107277945 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-phzf8" (UniqueName: "kubernetes.io/projected/88336825-e84e-4595-9f68-f4d6b4e6edda-kube-api-access-phzf8") pod "network-check-target-8qdhk" (UID: "88336825-e84e-4595-9f68-f4d6b4e6edda") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 18:34:29.204833 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:29.204722 2560 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-28 18:29:27 +0000 UTC" deadline="2028-02-27 02:37:56.533387992 +0000 UTC" Jul 29 18:34:29.204833 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:29.204785 2560 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13856h3m27.328607591s" Jul 29 18:34:29.349573 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:29.349540 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8qdhk" Jul 29 18:34:29.349755 ip-10-0-140-205 kubenswrapper[2560]: E0729 18:34:29.349688 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8qdhk" podUID="88336825-e84e-4595-9f68-f4d6b4e6edda" Jul 29 18:34:30.348750 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:30.348114 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mrshg" Jul 29 18:34:30.348750 ip-10-0-140-205 kubenswrapper[2560]: E0729 18:34:30.348249 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mrshg" podUID="70307ff7-a0ca-4a2b-95be-94a5dde5d96e" Jul 29 18:34:30.348750 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:30.348614 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zbwwp" Jul 29 18:34:30.348750 ip-10-0-140-205 kubenswrapper[2560]: E0729 18:34:30.348704 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zbwwp" podUID="13fd7656-9a1f-4221-a5ed-6ebecf3664c9" Jul 29 18:34:30.801867 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:30.801827 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/13fd7656-9a1f-4221-a5ed-6ebecf3664c9-original-pull-secret\") pod \"global-pull-secret-syncer-zbwwp\" (UID: \"13fd7656-9a1f-4221-a5ed-6ebecf3664c9\") " pod="kube-system/global-pull-secret-syncer-zbwwp" Jul 29 18:34:30.802039 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:30.801895 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70307ff7-a0ca-4a2b-95be-94a5dde5d96e-metrics-certs\") pod \"network-metrics-daemon-mrshg\" (UID: \"70307ff7-a0ca-4a2b-95be-94a5dde5d96e\") " pod="openshift-multus/network-metrics-daemon-mrshg" Jul 29 18:34:30.802039 ip-10-0-140-205 kubenswrapper[2560]: E0729 18:34:30.802007 2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 18:34:30.802147 ip-10-0-140-205 kubenswrapper[2560]: E0729 18:34:30.802065 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/70307ff7-a0ca-4a2b-95be-94a5dde5d96e-metrics-certs podName:70307ff7-a0ca-4a2b-95be-94a5dde5d96e nodeName:}" failed. No retries permitted until 2026-07-29 18:34:34.80204695 +0000 UTC m=+10.027715801 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/70307ff7-a0ca-4a2b-95be-94a5dde5d96e-metrics-certs") pod "network-metrics-daemon-mrshg" (UID: "70307ff7-a0ca-4a2b-95be-94a5dde5d96e") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 18:34:30.802559 ip-10-0-140-205 kubenswrapper[2560]: E0729 18:34:30.802538 2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 29 18:34:30.802677 ip-10-0-140-205 kubenswrapper[2560]: E0729 18:34:30.802593 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/13fd7656-9a1f-4221-a5ed-6ebecf3664c9-original-pull-secret podName:13fd7656-9a1f-4221-a5ed-6ebecf3664c9 nodeName:}" failed. No retries permitted until 2026-07-29 18:34:34.802578611 +0000 UTC m=+10.028247477 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/13fd7656-9a1f-4221-a5ed-6ebecf3664c9-original-pull-secret") pod "global-pull-secret-syncer-zbwwp" (UID: "13fd7656-9a1f-4221-a5ed-6ebecf3664c9") : object "kube-system"/"original-pull-secret" not registered Jul 29 18:34:30.905193 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:30.904626 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phzf8\" (UniqueName: \"kubernetes.io/projected/88336825-e84e-4595-9f68-f4d6b4e6edda-kube-api-access-phzf8\") pod \"network-check-target-8qdhk\" (UID: \"88336825-e84e-4595-9f68-f4d6b4e6edda\") " pod="openshift-network-diagnostics/network-check-target-8qdhk" Jul 29 18:34:30.905193 ip-10-0-140-205 kubenswrapper[2560]: E0729 18:34:30.904789 2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 29 18:34:30.905193 ip-10-0-140-205 kubenswrapper[2560]: E0729 18:34:30.904810 2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 29 18:34:30.905193 ip-10-0-140-205 kubenswrapper[2560]: E0729 18:34:30.904821 2560 projected.go:196] Error preparing data for projected volume kube-api-access-phzf8 for pod openshift-network-diagnostics/network-check-target-8qdhk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 18:34:30.905193 ip-10-0-140-205 kubenswrapper[2560]: E0729 18:34:30.904873 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/88336825-e84e-4595-9f68-f4d6b4e6edda-kube-api-access-phzf8 podName:88336825-e84e-4595-9f68-f4d6b4e6edda nodeName:}" failed. No retries permitted until 2026-07-29 18:34:34.90485394 +0000 UTC m=+10.130522790 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-phzf8" (UniqueName: "kubernetes.io/projected/88336825-e84e-4595-9f68-f4d6b4e6edda-kube-api-access-phzf8") pod "network-check-target-8qdhk" (UID: "88336825-e84e-4595-9f68-f4d6b4e6edda") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 18:34:31.348405 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:31.347902 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8qdhk" Jul 29 18:34:31.348405 ip-10-0-140-205 kubenswrapper[2560]: E0729 18:34:31.348036 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8qdhk" podUID="88336825-e84e-4595-9f68-f4d6b4e6edda" Jul 29 18:34:32.348714 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:32.347996 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mrshg" Jul 29 18:34:32.348714 ip-10-0-140-205 kubenswrapper[2560]: E0729 18:34:32.348148 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mrshg" podUID="70307ff7-a0ca-4a2b-95be-94a5dde5d96e" Jul 29 18:34:32.348714 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:32.348543 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zbwwp" Jul 29 18:34:32.348714 ip-10-0-140-205 kubenswrapper[2560]: E0729 18:34:32.348643 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zbwwp" podUID="13fd7656-9a1f-4221-a5ed-6ebecf3664c9" Jul 29 18:34:33.357479 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:33.357210 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8qdhk" Jul 29 18:34:33.357825 ip-10-0-140-205 kubenswrapper[2560]: E0729 18:34:33.357679 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8qdhk" podUID="88336825-e84e-4595-9f68-f4d6b4e6edda" Jul 29 18:34:34.348504 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:34.348475 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mrshg" Jul 29 18:34:34.348659 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:34.348517 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zbwwp" Jul 29 18:34:34.348659 ip-10-0-140-205 kubenswrapper[2560]: E0729 18:34:34.348618 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mrshg" podUID="70307ff7-a0ca-4a2b-95be-94a5dde5d96e" Jul 29 18:34:34.348984 ip-10-0-140-205 kubenswrapper[2560]: E0729 18:34:34.348961 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zbwwp" podUID="13fd7656-9a1f-4221-a5ed-6ebecf3664c9" Jul 29 18:34:34.839066 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:34.839027 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/13fd7656-9a1f-4221-a5ed-6ebecf3664c9-original-pull-secret\") pod \"global-pull-secret-syncer-zbwwp\" (UID: \"13fd7656-9a1f-4221-a5ed-6ebecf3664c9\") " pod="kube-system/global-pull-secret-syncer-zbwwp" Jul 29 18:34:34.839486 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:34.839108 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70307ff7-a0ca-4a2b-95be-94a5dde5d96e-metrics-certs\") pod \"network-metrics-daemon-mrshg\" (UID: \"70307ff7-a0ca-4a2b-95be-94a5dde5d96e\") " pod="openshift-multus/network-metrics-daemon-mrshg" Jul 29 18:34:34.839486 ip-10-0-140-205 kubenswrapper[2560]: E0729 18:34:34.839236 2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 18:34:34.839486 ip-10-0-140-205 kubenswrapper[2560]: E0729 18:34:34.839298 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/70307ff7-a0ca-4a2b-95be-94a5dde5d96e-metrics-certs podName:70307ff7-a0ca-4a2b-95be-94a5dde5d96e nodeName:}" failed. No retries permitted until 2026-07-29 18:34:42.83927936 +0000 UTC m=+18.064948216 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/70307ff7-a0ca-4a2b-95be-94a5dde5d96e-metrics-certs") pod "network-metrics-daemon-mrshg" (UID: "70307ff7-a0ca-4a2b-95be-94a5dde5d96e") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 18:34:34.839728 ip-10-0-140-205 kubenswrapper[2560]: E0729 18:34:34.839702 2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 29 18:34:34.839858 ip-10-0-140-205 kubenswrapper[2560]: E0729 18:34:34.839763 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/13fd7656-9a1f-4221-a5ed-6ebecf3664c9-original-pull-secret podName:13fd7656-9a1f-4221-a5ed-6ebecf3664c9 nodeName:}" failed. No retries permitted until 2026-07-29 18:34:42.839746638 +0000 UTC m=+18.065415492 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/13fd7656-9a1f-4221-a5ed-6ebecf3664c9-original-pull-secret") pod "global-pull-secret-syncer-zbwwp" (UID: "13fd7656-9a1f-4221-a5ed-6ebecf3664c9") : object "kube-system"/"original-pull-secret" not registered Jul 29 18:34:34.940538 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:34.939868 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phzf8\" (UniqueName: \"kubernetes.io/projected/88336825-e84e-4595-9f68-f4d6b4e6edda-kube-api-access-phzf8\") pod \"network-check-target-8qdhk\" (UID: \"88336825-e84e-4595-9f68-f4d6b4e6edda\") " pod="openshift-network-diagnostics/network-check-target-8qdhk" Jul 29 18:34:34.940538 ip-10-0-140-205 kubenswrapper[2560]: E0729 18:34:34.940057 2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 29 18:34:34.940538 ip-10-0-140-205 kubenswrapper[2560]: E0729 18:34:34.940076 2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 29 18:34:34.940538 ip-10-0-140-205 kubenswrapper[2560]: E0729 18:34:34.940089 2560 projected.go:196] Error preparing data for projected volume kube-api-access-phzf8 for pod openshift-network-diagnostics/network-check-target-8qdhk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 18:34:34.940538 ip-10-0-140-205 kubenswrapper[2560]: E0729 18:34:34.940145 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/88336825-e84e-4595-9f68-f4d6b4e6edda-kube-api-access-phzf8 podName:88336825-e84e-4595-9f68-f4d6b4e6edda nodeName:}" failed. No retries permitted until 2026-07-29 18:34:42.94012796 +0000 UTC m=+18.165796825 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-phzf8" (UniqueName: "kubernetes.io/projected/88336825-e84e-4595-9f68-f4d6b4e6edda-kube-api-access-phzf8") pod "network-check-target-8qdhk" (UID: "88336825-e84e-4595-9f68-f4d6b4e6edda") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 18:34:35.348829 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:35.348799 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8qdhk" Jul 29 18:34:35.349009 ip-10-0-140-205 kubenswrapper[2560]: E0729 18:34:35.348897 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8qdhk" podUID="88336825-e84e-4595-9f68-f4d6b4e6edda" Jul 29 18:34:36.347783 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:36.347750 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zbwwp" Jul 29 18:34:36.348241 ip-10-0-140-205 kubenswrapper[2560]: E0729 18:34:36.347878 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zbwwp" podUID="13fd7656-9a1f-4221-a5ed-6ebecf3664c9" Jul 29 18:34:36.348241 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:36.347940 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mrshg" Jul 29 18:34:36.348241 ip-10-0-140-205 kubenswrapper[2560]: E0729 18:34:36.348021 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mrshg" podUID="70307ff7-a0ca-4a2b-95be-94a5dde5d96e" Jul 29 18:34:37.347752 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:37.347718 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8qdhk" Jul 29 18:34:37.347932 ip-10-0-140-205 kubenswrapper[2560]: E0729 18:34:37.347852 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8qdhk" podUID="88336825-e84e-4595-9f68-f4d6b4e6edda" Jul 29 18:34:38.348638 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:38.348599 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mrshg" Jul 29 18:34:38.349067 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:38.348609 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zbwwp" Jul 29 18:34:38.349067 ip-10-0-140-205 kubenswrapper[2560]: E0729 18:34:38.348752 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mrshg" podUID="70307ff7-a0ca-4a2b-95be-94a5dde5d96e" Jul 29 18:34:38.349067 ip-10-0-140-205 kubenswrapper[2560]: E0729 18:34:38.348856 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zbwwp" podUID="13fd7656-9a1f-4221-a5ed-6ebecf3664c9" Jul 29 18:34:39.347875 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:39.347842 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8qdhk" Jul 29 18:34:39.348062 ip-10-0-140-205 kubenswrapper[2560]: E0729 18:34:39.347972 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8qdhk" podUID="88336825-e84e-4595-9f68-f4d6b4e6edda" Jul 29 18:34:40.348237 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:40.348202 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zbwwp" Jul 29 18:34:40.348706 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:40.348203 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mrshg" Jul 29 18:34:40.348706 ip-10-0-140-205 kubenswrapper[2560]: E0729 18:34:40.348336 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zbwwp" podUID="13fd7656-9a1f-4221-a5ed-6ebecf3664c9" Jul 29 18:34:40.348706 ip-10-0-140-205 kubenswrapper[2560]: E0729 18:34:40.348390 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mrshg" podUID="70307ff7-a0ca-4a2b-95be-94a5dde5d96e" Jul 29 18:34:41.348550 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:41.348511 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8qdhk" Jul 29 18:34:41.349006 ip-10-0-140-205 kubenswrapper[2560]: E0729 18:34:41.348642 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8qdhk" podUID="88336825-e84e-4595-9f68-f4d6b4e6edda" Jul 29 18:34:42.348333 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:42.348304 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zbwwp" Jul 29 18:34:42.348554 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:42.348307 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mrshg" Jul 29 18:34:42.348554 ip-10-0-140-205 kubenswrapper[2560]: E0729 18:34:42.348441 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zbwwp" podUID="13fd7656-9a1f-4221-a5ed-6ebecf3664c9" Jul 29 18:34:42.348554 ip-10-0-140-205 kubenswrapper[2560]: E0729 18:34:42.348490 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mrshg" podUID="70307ff7-a0ca-4a2b-95be-94a5dde5d96e" Jul 29 18:34:42.900002 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:42.899968 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/13fd7656-9a1f-4221-a5ed-6ebecf3664c9-original-pull-secret\") pod \"global-pull-secret-syncer-zbwwp\" (UID: \"13fd7656-9a1f-4221-a5ed-6ebecf3664c9\") " pod="kube-system/global-pull-secret-syncer-zbwwp" Jul 29 18:34:42.900187 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:42.900035 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70307ff7-a0ca-4a2b-95be-94a5dde5d96e-metrics-certs\") pod \"network-metrics-daemon-mrshg\" (UID: \"70307ff7-a0ca-4a2b-95be-94a5dde5d96e\") " pod="openshift-multus/network-metrics-daemon-mrshg" Jul 29 18:34:42.900187 ip-10-0-140-205 kubenswrapper[2560]: E0729 18:34:42.900146 2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 18:34:42.900187 ip-10-0-140-205 kubenswrapper[2560]: E0729 18:34:42.900149 2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 29 18:34:42.900333 ip-10-0-140-205 kubenswrapper[2560]: E0729 18:34:42.900203 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/70307ff7-a0ca-4a2b-95be-94a5dde5d96e-metrics-certs podName:70307ff7-a0ca-4a2b-95be-94a5dde5d96e nodeName:}" failed. No retries permitted until 2026-07-29 18:34:58.900183605 +0000 UTC m=+34.125852459 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/70307ff7-a0ca-4a2b-95be-94a5dde5d96e-metrics-certs") pod "network-metrics-daemon-mrshg" (UID: "70307ff7-a0ca-4a2b-95be-94a5dde5d96e") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 18:34:42.900333 ip-10-0-140-205 kubenswrapper[2560]: E0729 18:34:42.900221 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/13fd7656-9a1f-4221-a5ed-6ebecf3664c9-original-pull-secret podName:13fd7656-9a1f-4221-a5ed-6ebecf3664c9 nodeName:}" failed. No retries permitted until 2026-07-29 18:34:58.900212614 +0000 UTC m=+34.125881465 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/13fd7656-9a1f-4221-a5ed-6ebecf3664c9-original-pull-secret") pod "global-pull-secret-syncer-zbwwp" (UID: "13fd7656-9a1f-4221-a5ed-6ebecf3664c9") : object "kube-system"/"original-pull-secret" not registered Jul 29 18:34:43.001296 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:43.001268 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phzf8\" (UniqueName: \"kubernetes.io/projected/88336825-e84e-4595-9f68-f4d6b4e6edda-kube-api-access-phzf8\") pod \"network-check-target-8qdhk\" (UID: \"88336825-e84e-4595-9f68-f4d6b4e6edda\") " pod="openshift-network-diagnostics/network-check-target-8qdhk" Jul 29 18:34:43.001479 ip-10-0-140-205 kubenswrapper[2560]: E0729 18:34:43.001417 2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 29 18:34:43.001479 ip-10-0-140-205 kubenswrapper[2560]: E0729 18:34:43.001435 2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 29 18:34:43.001479 ip-10-0-140-205 kubenswrapper[2560]: E0729 18:34:43.001443 2560 projected.go:196] Error preparing data for projected volume kube-api-access-phzf8 for pod openshift-network-diagnostics/network-check-target-8qdhk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 18:34:43.001641 ip-10-0-140-205 kubenswrapper[2560]: E0729 18:34:43.001490 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/88336825-e84e-4595-9f68-f4d6b4e6edda-kube-api-access-phzf8 podName:88336825-e84e-4595-9f68-f4d6b4e6edda nodeName:}" failed. No retries permitted until 2026-07-29 18:34:59.001476119 +0000 UTC m=+34.227144970 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-phzf8" (UniqueName: "kubernetes.io/projected/88336825-e84e-4595-9f68-f4d6b4e6edda-kube-api-access-phzf8") pod "network-check-target-8qdhk" (UID: "88336825-e84e-4595-9f68-f4d6b4e6edda") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 18:34:43.347756 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:43.347718 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8qdhk" Jul 29 18:34:43.347938 ip-10-0-140-205 kubenswrapper[2560]: E0729 18:34:43.347843 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8qdhk" podUID="88336825-e84e-4595-9f68-f4d6b4e6edda" Jul 29 18:34:44.347662 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:44.347622 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mrshg" Jul 29 18:34:44.348079 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:44.347668 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zbwwp" Jul 29 18:34:44.348079 ip-10-0-140-205 kubenswrapper[2560]: E0729 18:34:44.347775 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mrshg" podUID="70307ff7-a0ca-4a2b-95be-94a5dde5d96e" Jul 29 18:34:44.348079 ip-10-0-140-205 kubenswrapper[2560]: E0729 18:34:44.347908 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zbwwp" podUID="13fd7656-9a1f-4221-a5ed-6ebecf3664c9" Jul 29 18:34:45.348564 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:45.348530 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8qdhk" Jul 29 18:34:45.349062 ip-10-0-140-205 kubenswrapper[2560]: E0729 18:34:45.348955 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8qdhk" podUID="88336825-e84e-4595-9f68-f4d6b4e6edda" Jul 29 18:34:46.348099 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:46.348071 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mrshg" Jul 29 18:34:46.348274 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:46.348071 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zbwwp" Jul 29 18:34:46.348274 ip-10-0-140-205 kubenswrapper[2560]: E0729 18:34:46.348167 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mrshg" podUID="70307ff7-a0ca-4a2b-95be-94a5dde5d96e" Jul 29 18:34:46.348274 ip-10-0-140-205 kubenswrapper[2560]: E0729 18:34:46.348255 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zbwwp" podUID="13fd7656-9a1f-4221-a5ed-6ebecf3664c9" Jul 29 18:34:47.348217 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:47.348190 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8qdhk" Jul 29 18:34:47.348609 ip-10-0-140-205 kubenswrapper[2560]: E0729 18:34:47.348300 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8qdhk" podUID="88336825-e84e-4595-9f68-f4d6b4e6edda" Jul 29 18:34:47.513649 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:47.513616 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-v4n5x" event={"ID":"1727fb40-fbed-4984-91d4-a5890d0f4103","Type":"ContainerStarted","Data":"067364ecc5649aff4af055ac9782a353d044a0e6f37dec24e6ba279682bf08d2"} Jul 29 18:34:47.517106 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:47.517083 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-84dhd" event={"ID":"c46b0e07-5bf9-469e-bde5-bc4708ce64f2","Type":"ContainerStarted","Data":"0e343f65795b7c43b9291971b9cb2b70bcc7b46767ddc135fd6e87ca19728ec7"} Jul 29 18:34:47.518540 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:47.518520 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-205.ec2.internal" event={"ID":"89287a5239330a896427ad6104c0beff","Type":"ContainerStarted","Data":"800da12bfc2e678f2185999bd23b5feb0c128d24475e65e771d5815a1ff997d9"} Jul 29 18:34:47.537486 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:47.537446 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-84dhd" podStartSLOduration=2.717451002 podStartE2EDuration="22.537434074s" totalImagesPullingTime="19.819983072s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 18:34:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 18:34:47.537010821 +0000 UTC m=+22.762679694" watchObservedRunningTime="2026-07-29 18:34:47.537434074 +0000 UTC m=+22.763102975" Jul 29 18:34:47.558212 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:47.558159 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-205.ec2.internal" podStartSLOduration=21.558141433 podStartE2EDuration="21.558141433s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 18:34:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 18:34:47.557564043 +0000 UTC m=+22.783232918" watchObservedRunningTime="2026-07-29 18:34:47.558141433 +0000 UTC m=+22.783810306" Jul 29 18:34:48.348670 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:48.348393 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mrshg" Jul 29 18:34:48.349185 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:48.348427 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zbwwp" Jul 29 18:34:48.349185 ip-10-0-140-205 kubenswrapper[2560]: E0729 18:34:48.348746 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mrshg" podUID="70307ff7-a0ca-4a2b-95be-94a5dde5d96e" Jul 29 18:34:48.349185 ip-10-0-140-205 kubenswrapper[2560]: E0729 18:34:48.348823 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zbwwp" podUID="13fd7656-9a1f-4221-a5ed-6ebecf3664c9" Jul 29 18:34:48.522525 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:48.522488 2560 generic.go:356] "Generic (PLEG): container finished" podID="0d56f34abf69a77b3a700a4e3019bb67" containerID="f7d59d1f14f696bef4e59a886ff49fae104b6afd18a6ab81d0f2e73dd5c29538" exitCode=0 Jul 29 18:34:48.522677 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:48.522575 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-205.ec2.internal" event={"ID":"0d56f34abf69a77b3a700a4e3019bb67","Type":"ContainerDied","Data":"f7d59d1f14f696bef4e59a886ff49fae104b6afd18a6ab81d0f2e73dd5c29538"} Jul 29 18:34:48.524093 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:48.524069 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-249zt" event={"ID":"aeadd3c9-a1ee-45a2-a683-3cfb51804716","Type":"ContainerStarted","Data":"5670c93433b44531632a6981507f6c427393884ee8342b3a3b137e6d122fe60e"} Jul 29 18:34:48.525668 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:48.525573 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-nvp4r" event={"ID":"96c83b74-1875-4a28-8e0e-1470f61a31c9","Type":"ContainerStarted","Data":"37952b5352b522cd2b72ff52d98976870a7163b1ffd19291b3657af1879d15b5"} Jul 29 18:34:48.528480 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:48.528465 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-v4n5x_1727fb40-fbed-4984-91d4-a5890d0f4103/ovn-acl-logging/0.log" Jul 29 18:34:48.528760 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:48.528737 2560 generic.go:356] "Generic (PLEG): container finished" podID="1727fb40-fbed-4984-91d4-a5890d0f4103" containerID="cf3d4df0e70135cfb9bbf2e8f7adbedae208be5c0c1fd41a8906cc792eef1c7e" exitCode=1 Jul 29 18:34:48.528847 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:48.528801 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-v4n5x" event={"ID":"1727fb40-fbed-4984-91d4-a5890d0f4103","Type":"ContainerStarted","Data":"119c0d528510399467b470e25a820be7bd853c956dc7a64d51a9c51460764502"} Jul 29 18:34:48.528897 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:48.528856 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-v4n5x" event={"ID":"1727fb40-fbed-4984-91d4-a5890d0f4103","Type":"ContainerStarted","Data":"ec6fe09f990c57a1b5334561384b9bd2cc31643498dff98d2e60ab19f0083424"} Jul 29 18:34:48.528945 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:48.528894 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-v4n5x" event={"ID":"1727fb40-fbed-4984-91d4-a5890d0f4103","Type":"ContainerStarted","Data":"59f976847773f45475cab4bab5bd6b5b46c56dc262609603ec19c8d2292e7ac3"} Jul 29 18:34:48.528945 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:48.528923 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-v4n5x" event={"ID":"1727fb40-fbed-4984-91d4-a5890d0f4103","Type":"ContainerStarted","Data":"016b1927aa430899264c153a896835710dc8ac6f50877b3a8751f0cc9d0e5677"} Jul 29 18:34:48.529027 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:48.528957 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-v4n5x" event={"ID":"1727fb40-fbed-4984-91d4-a5890d0f4103","Type":"ContainerDied","Data":"cf3d4df0e70135cfb9bbf2e8f7adbedae208be5c0c1fd41a8906cc792eef1c7e"} Jul 29 18:34:48.530196 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:48.530162 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-69g8w" event={"ID":"07b2a3b0-456d-4696-92a3-968f3cf7d956","Type":"ContainerStarted","Data":"fef810d51d3323afd92348a8377ee852f5df0ba2c24323b94ca63b37a759bdbb"} Jul 29 18:34:48.531501 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:48.531481 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-nnlkl" event={"ID":"2fbfc68e-382a-4371-8291-ce71b8eec8ce","Type":"ContainerStarted","Data":"b5736a8e63db79ee392573fe655eeaa60a959e67a999a9c9139973b1d9f5d71d"} Jul 29 18:34:48.532801 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:48.532783 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-z7pg7" event={"ID":"d5b867d8-3ad7-4aef-a0d8-6e8938ae7c68","Type":"ContainerStarted","Data":"a3013aac7c314087c1622269af9999cb9363f126821efb0b91a0b570eabf61b1"} Jul 29 18:34:48.534091 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:48.534072 2560 generic.go:356] "Generic (PLEG): container finished" podID="fb2816f1-d921-4e68-908b-ecc463ffb899" containerID="2e299a6eee1cfdbc9e40c0bf040fd360d3d199d2e9480a1ca9c898a79693ef09" exitCode=0 Jul 29 18:34:48.534164 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:48.534110 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xvbfs" event={"ID":"fb2816f1-d921-4e68-908b-ecc463ffb899","Type":"ContainerDied","Data":"2e299a6eee1cfdbc9e40c0bf040fd360d3d199d2e9480a1ca9c898a79693ef09"} Jul 29 18:34:48.554148 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:48.554100 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-nvp4r" podStartSLOduration=3.767794669 podStartE2EDuration="23.554088287s" totalImagesPullingTime="19.786293618s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 18:34:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 18:34:48.553625492 +0000 UTC m=+23.779294365" watchObservedRunningTime="2026-07-29 18:34:48.554088287 +0000 UTC m=+23.779757159" Jul 29 18:34:48.598877 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:48.598829 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-z7pg7" podStartSLOduration=3.53077947 podStartE2EDuration="23.598816059s" totalImagesPullingTime="20.068036589s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 18:34:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 18:34:48.598702195 +0000 UTC m=+23.824371068" watchObservedRunningTime="2026-07-29 18:34:48.598816059 +0000 UTC m=+23.824484931" Jul 29 18:34:48.630199 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:48.630155 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-69g8w" podStartSLOduration=3.9131373529999998 podStartE2EDuration="23.630138349s" totalImagesPullingTime="19.717000996s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 18:34:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 18:34:48.630098467 +0000 UTC m=+23.855767338" watchObservedRunningTime="2026-07-29 18:34:48.630138349 +0000 UTC m=+23.855807220" Jul 29 18:34:48.630553 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:48.630533 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-nnlkl" podStartSLOduration=3.828281939 podStartE2EDuration="23.63052726s" totalImagesPullingTime="19.802245321s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 18:34:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 18:34:48.61257351 +0000 UTC m=+23.838242383" watchObservedRunningTime="2026-07-29 18:34:48.63052726 +0000 UTC m=+23.856196132" Jul 29 18:34:48.760918 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:48.760870 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:34:48.800689 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:48.800616 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-nnlkl_2fbfc68e-382a-4371-8291-ce71b8eec8ce/dns-node-resolver/0.log" Jul 29 18:34:48.836540 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:48.836512 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:34:48.933854 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:48.933813 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:34:48.993884 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:48.993854 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:34:49.347924 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:49.347886 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8qdhk" Jul 29 18:34:49.348226 ip-10-0-140-205 kubenswrapper[2560]: E0729 18:34:49.348010 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8qdhk" podUID="88336825-e84e-4595-9f68-f4d6b4e6edda" Jul 29 18:34:49.496993 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:49.496972 2560 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 29 18:34:49.538337 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:49.538307 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-nxn8k" event={"ID":"90d823ce-32c1-4cf9-8165-161155b5c5d1","Type":"ContainerStarted","Data":"61a900311dcc4b31f91f7c9f2b33b8327f1759ce607f568cb14ccc92cbcde94a"} Jul 29 18:34:49.540490 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:49.540465 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-205.ec2.internal" event={"ID":"0d56f34abf69a77b3a700a4e3019bb67","Type":"ContainerStarted","Data":"60f5682d3217e07846626622e9d5af9180ec4bca065cd79376cf1e45ec6236ad"} Jul 29 18:34:49.542678 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:49.542647 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-249zt" event={"ID":"aeadd3c9-a1ee-45a2-a683-3cfb51804716","Type":"ContainerStarted","Data":"63f8141f87d5893e01176aa8a98068148d1f66f9b62d1e1fa68b76e59ee956c4"} Jul 29 18:34:49.567287 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:49.567231 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-205.ec2.internal" podStartSLOduration=23.567213727 podStartE2EDuration="23.567213727s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 18:34:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 18:34:49.566730207 +0000 UTC m=+24.792399080" watchObservedRunningTime="2026-07-29 18:34:49.567213727 +0000 UTC m=+24.792882601" Jul 29 18:34:49.567439 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:49.567404 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-nxn8k" podStartSLOduration=4.740217633 podStartE2EDuration="24.567395412s" totalImagesPullingTime="19.827177779s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 18:34:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 18:34:49.553316471 +0000 UTC m=+24.778985344" watchObservedRunningTime="2026-07-29 18:34:49.567395412 +0000 UTC m=+24.793064286" Jul 29 18:34:49.672580 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:49.672482 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:34:49.772878 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:49.772848 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:34:50.197168 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:50.197140 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-nvp4r_96c83b74-1875-4a28-8e0e-1470f61a31c9/node-ca/0.log" Jul 29 18:34:50.221537 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:50.221419 2560 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-29T18:34:49.49698929Z","UUID":"e8b5934a-3ef9-4cd4-9cd7-7681411aec15","Handler":null,"Name":"","Endpoint":""} Jul 29 18:34:50.223339 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:50.223313 2560 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 29 18:34:50.223539 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:50.223374 2560 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 29 18:34:50.347889 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:50.347811 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mrshg" Jul 29 18:34:50.348080 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:50.347824 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zbwwp" Jul 29 18:34:50.348080 ip-10-0-140-205 kubenswrapper[2560]: E0729 18:34:50.347932 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mrshg" podUID="70307ff7-a0ca-4a2b-95be-94a5dde5d96e" Jul 29 18:34:50.348080 ip-10-0-140-205 kubenswrapper[2560]: E0729 18:34:50.348013 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zbwwp" podUID="13fd7656-9a1f-4221-a5ed-6ebecf3664c9" Jul 29 18:34:50.550568 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:50.550536 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-v4n5x_1727fb40-fbed-4984-91d4-a5890d0f4103/ovn-acl-logging/0.log" Jul 29 18:34:50.551037 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:50.551012 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-v4n5x" event={"ID":"1727fb40-fbed-4984-91d4-a5890d0f4103","Type":"ContainerStarted","Data":"ad1934ad89f84f49bbdbe7953915dc6cc905ac584e4ef3651c695e5692dbc021"} Jul 29 18:34:50.616986 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:50.616959 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:34:50.720467 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:50.720437 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:34:50.820455 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:50.820417 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:34:51.348336 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:51.348309 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8qdhk" Jul 29 18:34:51.348568 ip-10-0-140-205 kubenswrapper[2560]: E0729 18:34:51.348434 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8qdhk" podUID="88336825-e84e-4595-9f68-f4d6b4e6edda" Jul 29 18:34:51.461907 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:51.461873 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:34:51.555130 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:51.555096 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-249zt" event={"ID":"aeadd3c9-a1ee-45a2-a683-3cfb51804716","Type":"ContainerStarted","Data":"9c9a8ecae4d7dcd85e371c985643bf5cc713238fed0764cd9f3a4103c7dc0be1"} Jul 29 18:34:51.577385 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:51.577317 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-249zt" podStartSLOduration=4.569208626 podStartE2EDuration="26.577299433s" totalImagesPullingTime="22.008090807s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 18:34:25 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 18:34:51.576951344 +0000 UTC m=+26.802620216" watchObservedRunningTime="2026-07-29 18:34:51.577299433 +0000 UTC m=+26.802968306" Jul 29 18:34:52.116881 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:52.116844 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:34:52.296480 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:52.296447 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:34:52.348460 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:52.348429 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mrshg" Jul 29 18:34:52.348460 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:52.348448 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zbwwp" Jul 29 18:34:52.348652 ip-10-0-140-205 kubenswrapper[2560]: E0729 18:34:52.348560 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zbwwp" podUID="13fd7656-9a1f-4221-a5ed-6ebecf3664c9" Jul 29 18:34:52.348719 ip-10-0-140-205 kubenswrapper[2560]: E0729 18:34:52.348696 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mrshg" podUID="70307ff7-a0ca-4a2b-95be-94a5dde5d96e" Jul 29 18:34:52.430084 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:52.430011 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:34:53.162402 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:53.161695 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-69g8w" Jul 29 18:34:53.162402 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:53.162320 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-69g8w" Jul 29 18:34:53.347977 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:53.347943 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8qdhk" Jul 29 18:34:53.348122 ip-10-0-140-205 kubenswrapper[2560]: E0729 18:34:53.348090 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8qdhk" podUID="88336825-e84e-4595-9f68-f4d6b4e6edda" Jul 29 18:34:53.401667 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:53.401634 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:34:53.462551 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:53.462524 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:34:53.564226 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:53.564168 2560 generic.go:356] "Generic (PLEG): container finished" podID="fb2816f1-d921-4e68-908b-ecc463ffb899" containerID="08d2b7031564d17466133c7606a20804ad5d94bd73d15dc86380d4db928b2d6d" exitCode=0 Jul 29 18:34:53.564369 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:53.564254 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xvbfs" event={"ID":"fb2816f1-d921-4e68-908b-ecc463ffb899","Type":"ContainerDied","Data":"08d2b7031564d17466133c7606a20804ad5d94bd73d15dc86380d4db928b2d6d"} Jul 29 18:34:53.567913 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:53.567895 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-v4n5x_1727fb40-fbed-4984-91d4-a5890d0f4103/ovn-acl-logging/0.log" Jul 29 18:34:53.568253 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:53.568233 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-v4n5x" event={"ID":"1727fb40-fbed-4984-91d4-a5890d0f4103","Type":"ContainerStarted","Data":"8eb3280b02f85f51c5b6bc3654f9ff316fe7e4fab9e6e8ab601173b78327a79b"} Jul 29 18:34:53.568470 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:53.568450 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-69g8w" Jul 29 18:34:53.568659 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:53.568647 2560 scope.go:117] "RemoveContainer" containerID="cf3d4df0e70135cfb9bbf2e8f7adbedae208be5c0c1fd41a8906cc792eef1c7e" Jul 29 18:34:53.568910 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:53.568893 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-69g8w" Jul 29 18:34:53.778867 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:53.778709 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-v4n5x" Jul 29 18:34:53.819122 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:53.819052 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:34:54.012753 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:54.012720 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:34:54.348617 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:54.348582 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zbwwp" Jul 29 18:34:54.349211 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:54.348629 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mrshg" Jul 29 18:34:54.349211 ip-10-0-140-205 kubenswrapper[2560]: E0729 18:34:54.348698 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zbwwp" podUID="13fd7656-9a1f-4221-a5ed-6ebecf3664c9" Jul 29 18:34:54.349211 ip-10-0-140-205 kubenswrapper[2560]: E0729 18:34:54.348827 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mrshg" podUID="70307ff7-a0ca-4a2b-95be-94a5dde5d96e" Jul 29 18:34:54.423506 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:54.423481 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:34:54.573656 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:54.573621 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-v4n5x_1727fb40-fbed-4984-91d4-a5890d0f4103/ovn-acl-logging/0.log" Jul 29 18:34:54.574106 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:54.574080 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-v4n5x" event={"ID":"1727fb40-fbed-4984-91d4-a5890d0f4103","Type":"ContainerStarted","Data":"6c4ee594f963fee48744f9c7a8171a724d4699ca172830c82eb1a51af53fd307"} Jul 29 18:34:54.574194 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:54.574151 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-v4n5x" Jul 29 18:34:54.574323 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:54.574309 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-v4n5x" Jul 29 18:34:54.589849 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:54.589828 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-v4n5x" Jul 29 18:34:54.589944 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:54.589923 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-v4n5x" Jul 29 18:34:54.605197 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:54.605110 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-v4n5x" podStartSLOduration=9.784890987 podStartE2EDuration="29.605096867s" totalImagesPullingTime="19.82020588s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 18:34:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 18:34:54.603802252 +0000 UTC m=+29.829471124" watchObservedRunningTime="2026-07-29 18:34:54.605096867 +0000 UTC m=+29.830765747" Jul 29 18:34:54.622275 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:54.622258 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:34:54.999744 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:54.999721 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:34:55.217638 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:55.217558 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:34:55.349207 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:55.349182 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8qdhk" Jul 29 18:34:55.349567 ip-10-0-140-205 kubenswrapper[2560]: E0729 18:34:55.349254 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8qdhk" podUID="88336825-e84e-4595-9f68-f4d6b4e6edda" Jul 29 18:34:55.415404 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:55.415382 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:34:55.577933 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:55.577902 2560 generic.go:356] "Generic (PLEG): container finished" podID="fb2816f1-d921-4e68-908b-ecc463ffb899" containerID="b22b51752bea48cf5791fafa184b14f4c802451a7263fac3948274a7ad1e8f0f" exitCode=0 Jul 29 18:34:55.578104 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:55.577982 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xvbfs" event={"ID":"fb2816f1-d921-4e68-908b-ecc463ffb899","Type":"ContainerDied","Data":"b22b51752bea48cf5791fafa184b14f4c802451a7263fac3948274a7ad1e8f0f"} Jul 29 18:34:55.810140 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:55.810109 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:34:55.932670 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:55.932586 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:34:56.032834 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:56.032797 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:34:56.348161 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:56.348128 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mrshg" Jul 29 18:34:56.348161 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:56.348150 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zbwwp" Jul 29 18:34:56.348436 ip-10-0-140-205 kubenswrapper[2560]: E0729 18:34:56.348263 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mrshg" podUID="70307ff7-a0ca-4a2b-95be-94a5dde5d96e" Jul 29 18:34:56.348502 ip-10-0-140-205 kubenswrapper[2560]: E0729 18:34:56.348375 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zbwwp" podUID="13fd7656-9a1f-4221-a5ed-6ebecf3664c9" Jul 29 18:34:56.411184 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:56.411160 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:34:56.472107 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:56.472079 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:34:56.811488 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:56.811457 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:34:57.220501 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:57.220469 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:34:57.348002 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:57.347970 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8qdhk" Jul 29 18:34:57.348150 ip-10-0-140-205 kubenswrapper[2560]: E0729 18:34:57.348085 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8qdhk" podUID="88336825-e84e-4595-9f68-f4d6b4e6edda" Jul 29 18:34:57.583849 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:57.583814 2560 generic.go:356] "Generic (PLEG): container finished" podID="fb2816f1-d921-4e68-908b-ecc463ffb899" containerID="a188dc805134cfc0d7025fbb6dd7e207ee1c0c348bd8824fb49dda039ba538fb" exitCode=0 Jul 29 18:34:57.584319 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:57.583883 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xvbfs" event={"ID":"fb2816f1-d921-4e68-908b-ecc463ffb899","Type":"ContainerDied","Data":"a188dc805134cfc0d7025fbb6dd7e207ee1c0c348bd8824fb49dda039ba538fb"} Jul 29 18:34:57.616238 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:57.616211 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:34:57.810515 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:57.810485 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:34:58.348395 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:58.348362 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mrshg" Jul 29 18:34:58.348395 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:58.348393 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zbwwp" Jul 29 18:34:58.348644 ip-10-0-140-205 kubenswrapper[2560]: E0729 18:34:58.348495 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mrshg" podUID="70307ff7-a0ca-4a2b-95be-94a5dde5d96e" Jul 29 18:34:58.348698 ip-10-0-140-205 kubenswrapper[2560]: E0729 18:34:58.348640 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zbwwp" podUID="13fd7656-9a1f-4221-a5ed-6ebecf3664c9" Jul 29 18:34:58.416544 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:58.416504 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:34:58.517665 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:58.517612 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:34:58.813604 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:58.813580 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:34:58.921854 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:58.921827 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/13fd7656-9a1f-4221-a5ed-6ebecf3664c9-original-pull-secret\") pod \"global-pull-secret-syncer-zbwwp\" (UID: \"13fd7656-9a1f-4221-a5ed-6ebecf3664c9\") " pod="kube-system/global-pull-secret-syncer-zbwwp" Jul 29 18:34:58.922037 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:58.921880 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70307ff7-a0ca-4a2b-95be-94a5dde5d96e-metrics-certs\") pod \"network-metrics-daemon-mrshg\" (UID: \"70307ff7-a0ca-4a2b-95be-94a5dde5d96e\") " pod="openshift-multus/network-metrics-daemon-mrshg" Jul 29 18:34:58.922037 ip-10-0-140-205 kubenswrapper[2560]: E0729 18:34:58.921981 2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 18:34:58.922037 ip-10-0-140-205 kubenswrapper[2560]: E0729 18:34:58.921986 2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 29 18:34:58.922187 ip-10-0-140-205 kubenswrapper[2560]: E0729 18:34:58.922042 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/70307ff7-a0ca-4a2b-95be-94a5dde5d96e-metrics-certs podName:70307ff7-a0ca-4a2b-95be-94a5dde5d96e nodeName:}" failed. No retries permitted until 2026-07-29 18:35:30.92202547 +0000 UTC m=+66.147694447 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/70307ff7-a0ca-4a2b-95be-94a5dde5d96e-metrics-certs") pod "network-metrics-daemon-mrshg" (UID: "70307ff7-a0ca-4a2b-95be-94a5dde5d96e") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 18:34:58.922187 ip-10-0-140-205 kubenswrapper[2560]: E0729 18:34:58.922058 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/13fd7656-9a1f-4221-a5ed-6ebecf3664c9-original-pull-secret podName:13fd7656-9a1f-4221-a5ed-6ebecf3664c9 nodeName:}" failed. No retries permitted until 2026-07-29 18:35:30.922051765 +0000 UTC m=+66.147720616 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/13fd7656-9a1f-4221-a5ed-6ebecf3664c9-original-pull-secret") pod "global-pull-secret-syncer-zbwwp" (UID: "13fd7656-9a1f-4221-a5ed-6ebecf3664c9") : object "kube-system"/"original-pull-secret" not registered Jul 29 18:34:59.022983 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:59.022950 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phzf8\" (UniqueName: \"kubernetes.io/projected/88336825-e84e-4595-9f68-f4d6b4e6edda-kube-api-access-phzf8\") pod \"network-check-target-8qdhk\" (UID: \"88336825-e84e-4595-9f68-f4d6b4e6edda\") " pod="openshift-network-diagnostics/network-check-target-8qdhk" Jul 29 18:34:59.023184 ip-10-0-140-205 kubenswrapper[2560]: E0729 18:34:59.023137 2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 29 18:34:59.023184 ip-10-0-140-205 kubenswrapper[2560]: E0729 18:34:59.023164 2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 29 18:34:59.023184 ip-10-0-140-205 kubenswrapper[2560]: E0729 18:34:59.023176 2560 projected.go:196] Error preparing data for projected volume kube-api-access-phzf8 for pod openshift-network-diagnostics/network-check-target-8qdhk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 18:34:59.023334 ip-10-0-140-205 kubenswrapper[2560]: E0729 18:34:59.023226 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/88336825-e84e-4595-9f68-f4d6b4e6edda-kube-api-access-phzf8 podName:88336825-e84e-4595-9f68-f4d6b4e6edda nodeName:}" failed. No retries permitted until 2026-07-29 18:35:31.023211397 +0000 UTC m=+66.248880248 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-phzf8" (UniqueName: "kubernetes.io/projected/88336825-e84e-4595-9f68-f4d6b4e6edda-kube-api-access-phzf8") pod "network-check-target-8qdhk" (UID: "88336825-e84e-4595-9f68-f4d6b4e6edda") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 18:34:59.208921 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:59.208845 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:34:59.351784 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:59.351751 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8qdhk" Jul 29 18:34:59.351938 ip-10-0-140-205 kubenswrapper[2560]: E0729 18:34:59.351870 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8qdhk" podUID="88336825-e84e-4595-9f68-f4d6b4e6edda" Jul 29 18:34:59.611808 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:59.611770 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:34:59.843152 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:59.843122 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:34:59.897414 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:34:59.897324 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:35:00.348562 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:00.348520 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zbwwp" Jul 29 18:35:00.348745 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:00.348571 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mrshg" Jul 29 18:35:00.348745 ip-10-0-140-205 kubenswrapper[2560]: E0729 18:35:00.348682 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zbwwp" podUID="13fd7656-9a1f-4221-a5ed-6ebecf3664c9" Jul 29 18:35:00.348745 ip-10-0-140-205 kubenswrapper[2560]: E0729 18:35:00.348709 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mrshg" podUID="70307ff7-a0ca-4a2b-95be-94a5dde5d96e" Jul 29 18:35:00.936876 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:00.936840 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:35:01.037476 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:01.037446 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:35:01.249134 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:01.249107 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:35:01.348617 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:01.348583 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8qdhk" Jul 29 18:35:01.348800 ip-10-0-140-205 kubenswrapper[2560]: E0729 18:35:01.348707 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8qdhk" podUID="88336825-e84e-4595-9f68-f4d6b4e6edda" Jul 29 18:35:01.392487 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:01.392448 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:35:01.494465 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:01.494418 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:35:01.837521 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:01.837492 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:35:02.018710 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:02.018676 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:35:02.216675 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:02.216622 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:35:02.317724 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:02.317602 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:35:02.347655 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:02.347625 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zbwwp" Jul 29 18:35:02.347801 ip-10-0-140-205 kubenswrapper[2560]: E0729 18:35:02.347740 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zbwwp" podUID="13fd7656-9a1f-4221-a5ed-6ebecf3664c9" Jul 29 18:35:02.347855 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:02.347797 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mrshg" Jul 29 18:35:02.347925 ip-10-0-140-205 kubenswrapper[2560]: E0729 18:35:02.347905 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mrshg" podUID="70307ff7-a0ca-4a2b-95be-94a5dde5d96e" Jul 29 18:35:02.419797 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:02.419761 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:35:02.598050 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:02.597955 2560 generic.go:356] "Generic (PLEG): container finished" podID="fb2816f1-d921-4e68-908b-ecc463ffb899" containerID="40597513503bd9965d41b27be1928705ecd3ee53731b3856f815dc3b8e0e769a" exitCode=0 Jul 29 18:35:02.598050 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:02.598001 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xvbfs" event={"ID":"fb2816f1-d921-4e68-908b-ecc463ffb899","Type":"ContainerDied","Data":"40597513503bd9965d41b27be1928705ecd3ee53731b3856f815dc3b8e0e769a"} Jul 29 18:35:02.824800 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:02.824764 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:35:03.348461 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:03.348433 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8qdhk" Jul 29 18:35:03.348819 ip-10-0-140-205 kubenswrapper[2560]: E0729 18:35:03.348534 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8qdhk" podUID="88336825-e84e-4595-9f68-f4d6b4e6edda" Jul 29 18:35:03.476771 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:03.476736 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:35:03.604060 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:03.603975 2560 generic.go:356] "Generic (PLEG): container finished" podID="fb2816f1-d921-4e68-908b-ecc463ffb899" containerID="0941dad57f23c259fa753686ddbde1bca135d6b99c13f3bb9fb7bf8d531a8dbe" exitCode=0 Jul 29 18:35:03.604060 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:03.604030 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xvbfs" event={"ID":"fb2816f1-d921-4e68-908b-ecc463ffb899","Type":"ContainerDied","Data":"0941dad57f23c259fa753686ddbde1bca135d6b99c13f3bb9fb7bf8d531a8dbe"} Jul 29 18:35:04.348570 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:04.348535 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mrshg" Jul 29 18:35:04.348949 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:04.348580 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zbwwp" Jul 29 18:35:04.348949 ip-10-0-140-205 kubenswrapper[2560]: E0729 18:35:04.348642 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mrshg" podUID="70307ff7-a0ca-4a2b-95be-94a5dde5d96e" Jul 29 18:35:04.348949 ip-10-0-140-205 kubenswrapper[2560]: E0729 18:35:04.348755 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zbwwp" podUID="13fd7656-9a1f-4221-a5ed-6ebecf3664c9" Jul 29 18:35:04.597239 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:04.597210 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:35:04.609712 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:04.609640 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xvbfs" event={"ID":"fb2816f1-d921-4e68-908b-ecc463ffb899","Type":"ContainerStarted","Data":"d54ce2c74a09ff214bcf768f40bc422fecfd228f7598753af763cc44ce0b69a0"} Jul 29 18:35:04.632674 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:04.632633 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-xvbfs" podStartSLOduration=8.672621323 podStartE2EDuration="39.632619651s" totalImagesPullingTime="30.959998328s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 18:34:25 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 18:35:04.631395032 +0000 UTC m=+39.857063897" watchObservedRunningTime="2026-07-29 18:35:04.632619651 +0000 UTC m=+39.858288524" Jul 29 18:35:04.653672 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:04.653650 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:35:04.705314 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:04.705292 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:35:04.883469 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:04.883395 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:35:05.063384 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:05.063338 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:35:05.276658 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:05.276631 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:35:05.337671 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:05.337643 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:35:05.348445 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:05.348429 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8qdhk" Jul 29 18:35:05.348530 ip-10-0-140-205 kubenswrapper[2560]: E0729 18:35:05.348503 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8qdhk" podUID="88336825-e84e-4595-9f68-f4d6b4e6edda" Jul 29 18:35:05.460415 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:05.460376 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:35:05.563278 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:05.563210 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:35:06.347913 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:06.347882 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mrshg" Jul 29 18:35:06.348065 ip-10-0-140-205 kubenswrapper[2560]: E0729 18:35:06.347986 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mrshg" podUID="70307ff7-a0ca-4a2b-95be-94a5dde5d96e" Jul 29 18:35:06.348065 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:06.348042 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zbwwp" Jul 29 18:35:06.348136 ip-10-0-140-205 kubenswrapper[2560]: E0729 18:35:06.348117 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zbwwp" podUID="13fd7656-9a1f-4221-a5ed-6ebecf3664c9" Jul 29 18:35:06.478625 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:06.478596 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:35:06.675610 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:06.675528 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:35:06.776094 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:06.776065 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:35:07.305090 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:07.305059 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:35:07.347967 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:07.347938 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8qdhk" Jul 29 18:35:07.348125 ip-10-0-140-205 kubenswrapper[2560]: E0729 18:35:07.348102 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8qdhk" podUID="88336825-e84e-4595-9f68-f4d6b4e6edda" Jul 29 18:35:07.372191 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:07.372164 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:35:08.348381 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:08.348332 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mrshg" Jul 29 18:35:08.348743 ip-10-0-140-205 kubenswrapper[2560]: E0729 18:35:08.348459 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mrshg" podUID="70307ff7-a0ca-4a2b-95be-94a5dde5d96e" Jul 29 18:35:08.348743 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:08.348513 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zbwwp" Jul 29 18:35:08.348743 ip-10-0-140-205 kubenswrapper[2560]: E0729 18:35:08.348597 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zbwwp" podUID="13fd7656-9a1f-4221-a5ed-6ebecf3664c9" Jul 29 18:35:09.348393 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:09.348365 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8qdhk" Jul 29 18:35:09.348784 ip-10-0-140-205 kubenswrapper[2560]: E0729 18:35:09.348473 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8qdhk" podUID="88336825-e84e-4595-9f68-f4d6b4e6edda" Jul 29 18:35:10.348274 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:10.348237 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mrshg" Jul 29 18:35:10.348274 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:10.348282 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zbwwp" Jul 29 18:35:10.348758 ip-10-0-140-205 kubenswrapper[2560]: E0729 18:35:10.348382 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mrshg" podUID="70307ff7-a0ca-4a2b-95be-94a5dde5d96e" Jul 29 18:35:10.348758 ip-10-0-140-205 kubenswrapper[2560]: E0729 18:35:10.348420 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zbwwp" podUID="13fd7656-9a1f-4221-a5ed-6ebecf3664c9" Jul 29 18:35:10.712792 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:10.712719 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:35:11.348308 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:11.348276 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8qdhk" Jul 29 18:35:11.348555 ip-10-0-140-205 kubenswrapper[2560]: E0729 18:35:11.348390 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8qdhk" podUID="88336825-e84e-4595-9f68-f4d6b4e6edda" Jul 29 18:35:11.552318 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:11.552287 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:35:11.618335 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:11.618269 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:35:12.294514 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:12.294477 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:35:12.347898 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:12.347871 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zbwwp" Jul 29 18:35:12.348002 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:12.347875 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mrshg" Jul 29 18:35:12.348002 ip-10-0-140-205 kubenswrapper[2560]: E0729 18:35:12.347969 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zbwwp" podUID="13fd7656-9a1f-4221-a5ed-6ebecf3664c9" Jul 29 18:35:12.348071 ip-10-0-140-205 kubenswrapper[2560]: E0729 18:35:12.348048 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mrshg" podUID="70307ff7-a0ca-4a2b-95be-94a5dde5d96e" Jul 29 18:35:12.395887 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:12.395854 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:35:12.498234 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:12.498203 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:35:12.601145 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:12.601072 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:35:12.828292 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:12.828257 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:35:12.983722 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:12.983648 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:35:13.160095 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:13.160066 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:35:13.285259 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:13.285235 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:35:13.348538 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:13.348513 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8qdhk" Jul 29 18:35:13.348692 ip-10-0-140-205 kubenswrapper[2560]: E0729 18:35:13.348614 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8qdhk" podUID="88336825-e84e-4595-9f68-f4d6b4e6edda" Jul 29 18:35:13.932751 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:13.932591 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:35:13.998945 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:13.998905 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:35:14.067652 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:14.067618 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-zbwwp"] Jul 29 18:35:14.067778 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:14.067726 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zbwwp" Jul 29 18:35:14.067864 ip-10-0-140-205 kubenswrapper[2560]: E0729 18:35:14.067840 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zbwwp" podUID="13fd7656-9a1f-4221-a5ed-6ebecf3664c9" Jul 29 18:35:14.071618 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:14.071224 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-8qdhk"] Jul 29 18:35:14.072273 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:14.072209 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-mrshg"] Jul 29 18:35:14.072409 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:14.072328 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mrshg" Jul 29 18:35:14.072474 ip-10-0-140-205 kubenswrapper[2560]: E0729 18:35:14.072443 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mrshg" podUID="70307ff7-a0ca-4a2b-95be-94a5dde5d96e" Jul 29 18:35:14.072645 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:14.072629 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8qdhk" Jul 29 18:35:14.072723 ip-10-0-140-205 kubenswrapper[2560]: E0729 18:35:14.072711 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8qdhk" podUID="88336825-e84e-4595-9f68-f4d6b4e6edda" Jul 29 18:35:15.348788 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:15.348763 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mrshg" Jul 29 18:35:15.349429 ip-10-0-140-205 kubenswrapper[2560]: E0729 18:35:15.348898 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mrshg" podUID="70307ff7-a0ca-4a2b-95be-94a5dde5d96e" Jul 29 18:35:15.349429 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:15.348955 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8qdhk" Jul 29 18:35:15.349429 ip-10-0-140-205 kubenswrapper[2560]: E0729 18:35:15.349069 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8qdhk" podUID="88336825-e84e-4595-9f68-f4d6b4e6edda" Jul 29 18:35:16.348103 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:16.348066 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zbwwp" Jul 29 18:35:16.348279 ip-10-0-140-205 kubenswrapper[2560]: E0729 18:35:16.348181 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zbwwp" podUID="13fd7656-9a1f-4221-a5ed-6ebecf3664c9" Jul 29 18:35:17.348104 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:17.348068 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mrshg" Jul 29 18:35:17.348648 ip-10-0-140-205 kubenswrapper[2560]: E0729 18:35:17.348186 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mrshg" podUID="70307ff7-a0ca-4a2b-95be-94a5dde5d96e" Jul 29 18:35:17.348648 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:17.348256 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8qdhk" Jul 29 18:35:17.348648 ip-10-0-140-205 kubenswrapper[2560]: E0729 18:35:17.348391 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8qdhk" podUID="88336825-e84e-4595-9f68-f4d6b4e6edda" Jul 29 18:35:18.071785 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:18.071761 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-205.ec2.internal" event="NodeReady" Jul 29 18:35:18.071903 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:18.071869 2560 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 29 18:35:18.116896 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:18.116862 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-f9stl"] Jul 29 18:35:18.121990 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:18.121968 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-f9stl" Jul 29 18:35:18.122763 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:18.122742 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-nr2lw"] Jul 29 18:35:18.126633 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:18.124833 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 29 18:35:18.126633 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:18.124913 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-84hqk\"" Jul 29 18:35:18.126633 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:18.124942 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 29 18:35:18.126633 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:18.125996 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 29 18:35:18.128402 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:18.128381 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-nr2lw" Jul 29 18:35:18.130539 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:18.130519 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 29 18:35:18.130868 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:18.130842 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 29 18:35:18.130967 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:18.130890 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 29 18:35:18.130967 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:18.130920 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 29 18:35:18.130967 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:18.130920 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-9grlj\"" Jul 29 18:35:18.132783 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:18.132765 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-f9stl"] Jul 29 18:35:18.136450 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:18.136426 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-nr2lw"] Jul 29 18:35:18.139544 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:18.139522 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-gls76"] Jul 29 18:35:18.146016 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:18.146000 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-gls76" Jul 29 18:35:18.148219 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:18.148202 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 29 18:35:18.148305 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:18.148250 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 29 18:35:18.148731 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:18.148485 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-2xgwr\"" Jul 29 18:35:18.156640 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:18.156617 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-gls76"] Jul 29 18:35:18.170828 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:18.170807 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/36f140ba-4691-4e5e-b58f-9305091a175e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nr2lw\" (UID: \"36f140ba-4691-4e5e-b58f-9305091a175e\") " pod="openshift-insights/insights-runtime-extractor-nr2lw" Jul 29 18:35:18.170913 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:18.170837 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/36f140ba-4691-4e5e-b58f-9305091a175e-data-volume\") pod \"insights-runtime-extractor-nr2lw\" (UID: \"36f140ba-4691-4e5e-b58f-9305091a175e\") " pod="openshift-insights/insights-runtime-extractor-nr2lw" Jul 29 18:35:18.170913 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:18.170858 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8q6v\" (UniqueName: \"kubernetes.io/projected/36f140ba-4691-4e5e-b58f-9305091a175e-kube-api-access-k8q6v\") pod \"insights-runtime-extractor-nr2lw\" (UID: \"36f140ba-4691-4e5e-b58f-9305091a175e\") " pod="openshift-insights/insights-runtime-extractor-nr2lw" Jul 29 18:35:18.170992 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:18.170911 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/4e3cd5b3-9c9f-4282-a3ee-9ebb947273a9-config-volume\") pod \"dns-default-gls76\" (UID: \"4e3cd5b3-9c9f-4282-a3ee-9ebb947273a9\") " pod="openshift-dns/dns-default-gls76" Jul 29 18:35:18.170992 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:18.170930 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/36f140ba-4691-4e5e-b58f-9305091a175e-crio-socket\") pod \"insights-runtime-extractor-nr2lw\" (UID: \"36f140ba-4691-4e5e-b58f-9305091a175e\") " pod="openshift-insights/insights-runtime-extractor-nr2lw" Jul 29 18:35:18.171055 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:18.170994 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/615e2b70-a2a4-4564-ab30-eb2ca3869414-cert\") pod \"ingress-canary-f9stl\" (UID: \"615e2b70-a2a4-4564-ab30-eb2ca3869414\") " pod="openshift-ingress-canary/ingress-canary-f9stl" Jul 29 18:35:18.171055 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:18.171023 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4e3cd5b3-9c9f-4282-a3ee-9ebb947273a9-metrics-tls\") pod \"dns-default-gls76\" (UID: \"4e3cd5b3-9c9f-4282-a3ee-9ebb947273a9\") " pod="openshift-dns/dns-default-gls76" Jul 29 18:35:18.171055 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:18.171041 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4wzhp\" (UniqueName: \"kubernetes.io/projected/615e2b70-a2a4-4564-ab30-eb2ca3869414-kube-api-access-4wzhp\") pod \"ingress-canary-f9stl\" (UID: \"615e2b70-a2a4-4564-ab30-eb2ca3869414\") " pod="openshift-ingress-canary/ingress-canary-f9stl" Jul 29 18:35:18.171140 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:18.171057 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4p22t\" (UniqueName: \"kubernetes.io/projected/4e3cd5b3-9c9f-4282-a3ee-9ebb947273a9-kube-api-access-4p22t\") pod \"dns-default-gls76\" (UID: \"4e3cd5b3-9c9f-4282-a3ee-9ebb947273a9\") " pod="openshift-dns/dns-default-gls76" Jul 29 18:35:18.171140 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:18.171078 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e3cd5b3-9c9f-4282-a3ee-9ebb947273a9-tmp-dir\") pod \"dns-default-gls76\" (UID: \"4e3cd5b3-9c9f-4282-a3ee-9ebb947273a9\") " pod="openshift-dns/dns-default-gls76" Jul 29 18:35:18.171140 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:18.171100 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/36f140ba-4691-4e5e-b58f-9305091a175e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nr2lw\" (UID: \"36f140ba-4691-4e5e-b58f-9305091a175e\") " pod="openshift-insights/insights-runtime-extractor-nr2lw" Jul 29 18:35:18.257282 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:18.257252 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:35:18.271360 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:18.271320 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/4e3cd5b3-9c9f-4282-a3ee-9ebb947273a9-config-volume\") pod \"dns-default-gls76\" (UID: \"4e3cd5b3-9c9f-4282-a3ee-9ebb947273a9\") " pod="openshift-dns/dns-default-gls76" Jul 29 18:35:18.271467 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:18.271368 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/36f140ba-4691-4e5e-b58f-9305091a175e-crio-socket\") pod \"insights-runtime-extractor-nr2lw\" (UID: \"36f140ba-4691-4e5e-b58f-9305091a175e\") " pod="openshift-insights/insights-runtime-extractor-nr2lw" Jul 29 18:35:18.271467 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:18.271410 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/615e2b70-a2a4-4564-ab30-eb2ca3869414-cert\") pod \"ingress-canary-f9stl\" (UID: \"615e2b70-a2a4-4564-ab30-eb2ca3869414\") " pod="openshift-ingress-canary/ingress-canary-f9stl" Jul 29 18:35:18.271467 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:18.271425 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4e3cd5b3-9c9f-4282-a3ee-9ebb947273a9-metrics-tls\") pod \"dns-default-gls76\" (UID: \"4e3cd5b3-9c9f-4282-a3ee-9ebb947273a9\") " pod="openshift-dns/dns-default-gls76" Jul 29 18:35:18.271467 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:18.271441 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wzhp\" (UniqueName: \"kubernetes.io/projected/615e2b70-a2a4-4564-ab30-eb2ca3869414-kube-api-access-4wzhp\") pod \"ingress-canary-f9stl\" (UID: \"615e2b70-a2a4-4564-ab30-eb2ca3869414\") " pod="openshift-ingress-canary/ingress-canary-f9stl" Jul 29 18:35:18.271467 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:18.271455 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4p22t\" (UniqueName: \"kubernetes.io/projected/4e3cd5b3-9c9f-4282-a3ee-9ebb947273a9-kube-api-access-4p22t\") pod \"dns-default-gls76\" (UID: \"4e3cd5b3-9c9f-4282-a3ee-9ebb947273a9\") " pod="openshift-dns/dns-default-gls76" Jul 29 18:35:18.271688 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:18.271473 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e3cd5b3-9c9f-4282-a3ee-9ebb947273a9-tmp-dir\") pod \"dns-default-gls76\" (UID: \"4e3cd5b3-9c9f-4282-a3ee-9ebb947273a9\") " pod="openshift-dns/dns-default-gls76" Jul 29 18:35:18.271688 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:18.271547 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/36f140ba-4691-4e5e-b58f-9305091a175e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nr2lw\" (UID: \"36f140ba-4691-4e5e-b58f-9305091a175e\") " pod="openshift-insights/insights-runtime-extractor-nr2lw" Jul 29 18:35:18.271688 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:18.271589 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/36f140ba-4691-4e5e-b58f-9305091a175e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nr2lw\" (UID: \"36f140ba-4691-4e5e-b58f-9305091a175e\") " pod="openshift-insights/insights-runtime-extractor-nr2lw" Jul 29 18:35:18.271688 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:18.271621 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/36f140ba-4691-4e5e-b58f-9305091a175e-data-volume\") pod \"insights-runtime-extractor-nr2lw\" (UID: \"36f140ba-4691-4e5e-b58f-9305091a175e\") " pod="openshift-insights/insights-runtime-extractor-nr2lw" Jul 29 18:35:18.271688 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:18.271652 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8q6v\" (UniqueName: \"kubernetes.io/projected/36f140ba-4691-4e5e-b58f-9305091a175e-kube-api-access-k8q6v\") pod \"insights-runtime-extractor-nr2lw\" (UID: \"36f140ba-4691-4e5e-b58f-9305091a175e\") " pod="openshift-insights/insights-runtime-extractor-nr2lw" Jul 29 18:35:18.271688 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:18.271660 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/36f140ba-4691-4e5e-b58f-9305091a175e-crio-socket\") pod \"insights-runtime-extractor-nr2lw\" (UID: \"36f140ba-4691-4e5e-b58f-9305091a175e\") " pod="openshift-insights/insights-runtime-extractor-nr2lw" Jul 29 18:35:18.272005 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:18.271969 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/4e3cd5b3-9c9f-4282-a3ee-9ebb947273a9-config-volume\") pod \"dns-default-gls76\" (UID: \"4e3cd5b3-9c9f-4282-a3ee-9ebb947273a9\") " pod="openshift-dns/dns-default-gls76" Jul 29 18:35:18.272092 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:18.271971 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e3cd5b3-9c9f-4282-a3ee-9ebb947273a9-tmp-dir\") pod \"dns-default-gls76\" (UID: \"4e3cd5b3-9c9f-4282-a3ee-9ebb947273a9\") " pod="openshift-dns/dns-default-gls76" Jul 29 18:35:18.272262 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:18.272241 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/36f140ba-4691-4e5e-b58f-9305091a175e-data-volume\") pod \"insights-runtime-extractor-nr2lw\" (UID: \"36f140ba-4691-4e5e-b58f-9305091a175e\") " pod="openshift-insights/insights-runtime-extractor-nr2lw" Jul 29 18:35:18.272437 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:18.272420 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/36f140ba-4691-4e5e-b58f-9305091a175e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nr2lw\" (UID: \"36f140ba-4691-4e5e-b58f-9305091a175e\") " pod="openshift-insights/insights-runtime-extractor-nr2lw" Jul 29 18:35:18.275594 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:18.275570 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/36f140ba-4691-4e5e-b58f-9305091a175e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nr2lw\" (UID: \"36f140ba-4691-4e5e-b58f-9305091a175e\") " pod="openshift-insights/insights-runtime-extractor-nr2lw" Jul 29 18:35:18.275682 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:18.275614 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4e3cd5b3-9c9f-4282-a3ee-9ebb947273a9-metrics-tls\") pod \"dns-default-gls76\" (UID: \"4e3cd5b3-9c9f-4282-a3ee-9ebb947273a9\") " pod="openshift-dns/dns-default-gls76" Jul 29 18:35:18.275800 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:18.275786 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/615e2b70-a2a4-4564-ab30-eb2ca3869414-cert\") pod \"ingress-canary-f9stl\" (UID: \"615e2b70-a2a4-4564-ab30-eb2ca3869414\") " pod="openshift-ingress-canary/ingress-canary-f9stl" Jul 29 18:35:18.280874 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:18.280850 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8q6v\" (UniqueName: \"kubernetes.io/projected/36f140ba-4691-4e5e-b58f-9305091a175e-kube-api-access-k8q6v\") pod \"insights-runtime-extractor-nr2lw\" (UID: \"36f140ba-4691-4e5e-b58f-9305091a175e\") " pod="openshift-insights/insights-runtime-extractor-nr2lw" Jul 29 18:35:18.281184 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:18.281162 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4p22t\" (UniqueName: \"kubernetes.io/projected/4e3cd5b3-9c9f-4282-a3ee-9ebb947273a9-kube-api-access-4p22t\") pod \"dns-default-gls76\" (UID: \"4e3cd5b3-9c9f-4282-a3ee-9ebb947273a9\") " pod="openshift-dns/dns-default-gls76" Jul 29 18:35:18.281306 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:18.281288 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wzhp\" (UniqueName: \"kubernetes.io/projected/615e2b70-a2a4-4564-ab30-eb2ca3869414-kube-api-access-4wzhp\") pod \"ingress-canary-f9stl\" (UID: \"615e2b70-a2a4-4564-ab30-eb2ca3869414\") " pod="openshift-ingress-canary/ingress-canary-f9stl" Jul 29 18:35:18.319673 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:18.319648 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:35:18.347643 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:18.347623 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zbwwp" Jul 29 18:35:18.350133 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:18.350115 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 29 18:35:18.436010 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:18.435976 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-f9stl" Jul 29 18:35:18.442119 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:18.442101 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-nr2lw" Jul 29 18:35:18.453226 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:18.453203 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-gls76" Jul 29 18:35:18.646297 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:18.645990 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-nr2lw"] Jul 29 18:35:18.650853 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:18.650827 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-gls76"] Jul 29 18:35:18.654129 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:35:18.654097 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e3cd5b3_9c9f_4282_a3ee_9ebb947273a9.slice/crio-d5f878391e0507fdcda558391f2be17383cbbd258d608d1380b243374ef16aa4 WatchSource:0}: Error finding container d5f878391e0507fdcda558391f2be17383cbbd258d608d1380b243374ef16aa4: Status 404 returned error can't find the container with id d5f878391e0507fdcda558391f2be17383cbbd258d608d1380b243374ef16aa4 Jul 29 18:35:18.654302 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:18.654283 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-f9stl"] Jul 29 18:35:18.710249 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:18.710181 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:35:18.766756 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:18.766639 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:35:18.962764 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:18.962716 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:35:19.023603 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:19.023488 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:35:19.349591 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:19.349483 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8qdhk" Jul 29 18:35:19.350117 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:19.349816 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mrshg" Jul 29 18:35:19.354844 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:19.353147 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-hfcvr\"" Jul 29 18:35:19.354844 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:19.353402 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 29 18:35:19.354844 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:19.353596 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-mfhf2\"" Jul 29 18:35:19.354844 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:19.353799 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 29 18:35:19.354844 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:19.353992 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 29 18:35:19.644010 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:19.643874 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-gls76" event={"ID":"4e3cd5b3-9c9f-4282-a3ee-9ebb947273a9","Type":"ContainerStarted","Data":"d5f878391e0507fdcda558391f2be17383cbbd258d608d1380b243374ef16aa4"} Jul 29 18:35:19.646716 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:19.646686 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-nr2lw" event={"ID":"36f140ba-4691-4e5e-b58f-9305091a175e","Type":"ContainerStarted","Data":"c8d97ab3e2a585ce7aa25b3510c98b8068fbab79d7f07ff0ba7858319766a803"} Jul 29 18:35:19.655171 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:19.655116 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-f9stl" event={"ID":"615e2b70-a2a4-4564-ab30-eb2ca3869414","Type":"ContainerStarted","Data":"4b66554ab953647f0c2aa478001bc559a1ed4a16d5d028512335633ffeeb9197"} Jul 29 18:35:20.904528 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:20.903777 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-765c45df68-wbklv"] Jul 29 18:35:20.912699 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:20.912677 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-765c45df68-wbklv" Jul 29 18:35:20.915197 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:20.915173 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 29 18:35:20.915312 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:20.915200 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 29 18:35:20.915391 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:20.915173 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 29 18:35:20.915994 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:20.915966 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-9hd9l\"" Jul 29 18:35:20.916108 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:20.915996 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 29 18:35:20.916395 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:20.916376 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 29 18:35:20.916590 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:20.916572 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 29 18:35:20.916770 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:20.916755 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 29 18:35:20.917974 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:20.917950 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-765c45df68-wbklv"] Jul 29 18:35:20.921546 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:20.921494 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 29 18:35:20.994299 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:20.994100 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/dcaf4f9c-b5cb-4f1d-83ed-72639860bee6-console-serving-cert\") pod \"console-765c45df68-wbklv\" (UID: \"dcaf4f9c-b5cb-4f1d-83ed-72639860bee6\") " pod="openshift-console/console-765c45df68-wbklv" Jul 29 18:35:20.994299 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:20.994136 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dcaf4f9c-b5cb-4f1d-83ed-72639860bee6-trusted-ca-bundle\") pod \"console-765c45df68-wbklv\" (UID: \"dcaf4f9c-b5cb-4f1d-83ed-72639860bee6\") " pod="openshift-console/console-765c45df68-wbklv" Jul 29 18:35:20.994299 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:20.994179 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/dcaf4f9c-b5cb-4f1d-83ed-72639860bee6-console-oauth-config\") pod \"console-765c45df68-wbklv\" (UID: \"dcaf4f9c-b5cb-4f1d-83ed-72639860bee6\") " pod="openshift-console/console-765c45df68-wbklv" Jul 29 18:35:20.994299 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:20.994203 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/dcaf4f9c-b5cb-4f1d-83ed-72639860bee6-console-config\") pod \"console-765c45df68-wbklv\" (UID: \"dcaf4f9c-b5cb-4f1d-83ed-72639860bee6\") " pod="openshift-console/console-765c45df68-wbklv" Jul 29 18:35:20.994299 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:20.994227 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/dcaf4f9c-b5cb-4f1d-83ed-72639860bee6-oauth-serving-cert\") pod \"console-765c45df68-wbklv\" (UID: \"dcaf4f9c-b5cb-4f1d-83ed-72639860bee6\") " pod="openshift-console/console-765c45df68-wbklv" Jul 29 18:35:20.994299 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:20.994269 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f6gfr\" (UniqueName: \"kubernetes.io/projected/dcaf4f9c-b5cb-4f1d-83ed-72639860bee6-kube-api-access-f6gfr\") pod \"console-765c45df68-wbklv\" (UID: \"dcaf4f9c-b5cb-4f1d-83ed-72639860bee6\") " pod="openshift-console/console-765c45df68-wbklv" Jul 29 18:35:20.994671 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:20.994367 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/dcaf4f9c-b5cb-4f1d-83ed-72639860bee6-service-ca\") pod \"console-765c45df68-wbklv\" (UID: \"dcaf4f9c-b5cb-4f1d-83ed-72639860bee6\") " pod="openshift-console/console-765c45df68-wbklv" Jul 29 18:35:21.095664 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:21.095625 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/dcaf4f9c-b5cb-4f1d-83ed-72639860bee6-console-oauth-config\") pod \"console-765c45df68-wbklv\" (UID: \"dcaf4f9c-b5cb-4f1d-83ed-72639860bee6\") " pod="openshift-console/console-765c45df68-wbklv" Jul 29 18:35:21.095664 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:21.095673 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/dcaf4f9c-b5cb-4f1d-83ed-72639860bee6-console-config\") pod \"console-765c45df68-wbklv\" (UID: \"dcaf4f9c-b5cb-4f1d-83ed-72639860bee6\") " pod="openshift-console/console-765c45df68-wbklv" Jul 29 18:35:21.095913 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:21.095698 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/dcaf4f9c-b5cb-4f1d-83ed-72639860bee6-oauth-serving-cert\") pod \"console-765c45df68-wbklv\" (UID: \"dcaf4f9c-b5cb-4f1d-83ed-72639860bee6\") " pod="openshift-console/console-765c45df68-wbklv" Jul 29 18:35:21.095913 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:21.095744 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6gfr\" (UniqueName: \"kubernetes.io/projected/dcaf4f9c-b5cb-4f1d-83ed-72639860bee6-kube-api-access-f6gfr\") pod \"console-765c45df68-wbklv\" (UID: \"dcaf4f9c-b5cb-4f1d-83ed-72639860bee6\") " pod="openshift-console/console-765c45df68-wbklv" Jul 29 18:35:21.095913 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:21.095811 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/dcaf4f9c-b5cb-4f1d-83ed-72639860bee6-service-ca\") pod \"console-765c45df68-wbklv\" (UID: \"dcaf4f9c-b5cb-4f1d-83ed-72639860bee6\") " pod="openshift-console/console-765c45df68-wbklv" Jul 29 18:35:21.095913 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:21.095848 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/dcaf4f9c-b5cb-4f1d-83ed-72639860bee6-console-serving-cert\") pod \"console-765c45df68-wbklv\" (UID: \"dcaf4f9c-b5cb-4f1d-83ed-72639860bee6\") " pod="openshift-console/console-765c45df68-wbklv" Jul 29 18:35:21.095913 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:21.095874 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dcaf4f9c-b5cb-4f1d-83ed-72639860bee6-trusted-ca-bundle\") pod \"console-765c45df68-wbklv\" (UID: \"dcaf4f9c-b5cb-4f1d-83ed-72639860bee6\") " pod="openshift-console/console-765c45df68-wbklv" Jul 29 18:35:21.096916 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:21.096892 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dcaf4f9c-b5cb-4f1d-83ed-72639860bee6-trusted-ca-bundle\") pod \"console-765c45df68-wbklv\" (UID: \"dcaf4f9c-b5cb-4f1d-83ed-72639860bee6\") " pod="openshift-console/console-765c45df68-wbklv" Jul 29 18:35:21.097039 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:21.096900 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/dcaf4f9c-b5cb-4f1d-83ed-72639860bee6-service-ca\") pod \"console-765c45df68-wbklv\" (UID: \"dcaf4f9c-b5cb-4f1d-83ed-72639860bee6\") " pod="openshift-console/console-765c45df68-wbklv" Jul 29 18:35:21.097150 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:21.097130 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/dcaf4f9c-b5cb-4f1d-83ed-72639860bee6-console-config\") pod \"console-765c45df68-wbklv\" (UID: \"dcaf4f9c-b5cb-4f1d-83ed-72639860bee6\") " pod="openshift-console/console-765c45df68-wbklv" Jul 29 18:35:21.097492 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:21.097465 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/dcaf4f9c-b5cb-4f1d-83ed-72639860bee6-oauth-serving-cert\") pod \"console-765c45df68-wbklv\" (UID: \"dcaf4f9c-b5cb-4f1d-83ed-72639860bee6\") " pod="openshift-console/console-765c45df68-wbklv" Jul 29 18:35:21.100828 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:21.100808 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/dcaf4f9c-b5cb-4f1d-83ed-72639860bee6-console-oauth-config\") pod \"console-765c45df68-wbklv\" (UID: \"dcaf4f9c-b5cb-4f1d-83ed-72639860bee6\") " pod="openshift-console/console-765c45df68-wbklv" Jul 29 18:35:21.100946 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:21.100920 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/dcaf4f9c-b5cb-4f1d-83ed-72639860bee6-console-serving-cert\") pod \"console-765c45df68-wbklv\" (UID: \"dcaf4f9c-b5cb-4f1d-83ed-72639860bee6\") " pod="openshift-console/console-765c45df68-wbklv" Jul 29 18:35:21.105624 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:21.105598 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6gfr\" (UniqueName: \"kubernetes.io/projected/dcaf4f9c-b5cb-4f1d-83ed-72639860bee6-kube-api-access-f6gfr\") pod \"console-765c45df68-wbklv\" (UID: \"dcaf4f9c-b5cb-4f1d-83ed-72639860bee6\") " pod="openshift-console/console-765c45df68-wbklv" Jul 29 18:35:21.226941 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:21.226851 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-765c45df68-wbklv" Jul 29 18:35:21.312691 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:21.312620 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:35:21.382273 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:21.382237 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:35:21.506005 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:21.505973 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:35:21.545036 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:21.545003 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-cxj67"] Jul 29 18:35:21.547224 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:21.547202 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-84d69f794d-cxj67" Jul 29 18:35:21.549538 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:21.549512 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Jul 29 18:35:21.549538 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:21.549525 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-q7b4c\"" Jul 29 18:35:21.550484 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:21.550463 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 29 18:35:21.550484 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:21.550479 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Jul 29 18:35:21.550633 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:21.550501 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 29 18:35:21.550633 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:21.550481 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 29 18:35:21.556981 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:21.556956 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-cxj67"] Jul 29 18:35:21.586103 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:21.586072 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:35:21.601300 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:21.601261 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rl9z5\" (UniqueName: \"kubernetes.io/projected/64b54a01-00fb-446a-9def-aaa7127c3fef-kube-api-access-rl9z5\") pod \"prometheus-operator-84d69f794d-cxj67\" (UID: \"64b54a01-00fb-446a-9def-aaa7127c3fef\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-cxj67" Jul 29 18:35:21.601300 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:21.601299 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/64b54a01-00fb-446a-9def-aaa7127c3fef-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-cxj67\" (UID: \"64b54a01-00fb-446a-9def-aaa7127c3fef\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-cxj67" Jul 29 18:35:21.601497 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:21.601375 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/64b54a01-00fb-446a-9def-aaa7127c3fef-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-cxj67\" (UID: \"64b54a01-00fb-446a-9def-aaa7127c3fef\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-cxj67" Jul 29 18:35:21.601497 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:21.601434 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/64b54a01-00fb-446a-9def-aaa7127c3fef-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-cxj67\" (UID: \"64b54a01-00fb-446a-9def-aaa7127c3fef\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-cxj67" Jul 29 18:35:21.701816 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:21.701787 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rl9z5\" (UniqueName: \"kubernetes.io/projected/64b54a01-00fb-446a-9def-aaa7127c3fef-kube-api-access-rl9z5\") pod \"prometheus-operator-84d69f794d-cxj67\" (UID: \"64b54a01-00fb-446a-9def-aaa7127c3fef\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-cxj67" Jul 29 18:35:21.701816 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:21.701821 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/64b54a01-00fb-446a-9def-aaa7127c3fef-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-cxj67\" (UID: \"64b54a01-00fb-446a-9def-aaa7127c3fef\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-cxj67" Jul 29 18:35:21.702044 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:21.701840 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/64b54a01-00fb-446a-9def-aaa7127c3fef-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-cxj67\" (UID: \"64b54a01-00fb-446a-9def-aaa7127c3fef\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-cxj67" Jul 29 18:35:21.702044 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:21.701887 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/64b54a01-00fb-446a-9def-aaa7127c3fef-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-cxj67\" (UID: \"64b54a01-00fb-446a-9def-aaa7127c3fef\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-cxj67" Jul 29 18:35:21.702044 ip-10-0-140-205 kubenswrapper[2560]: E0729 18:35:21.701940 2560 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found Jul 29 18:35:21.702044 ip-10-0-140-205 kubenswrapper[2560]: E0729 18:35:21.702009 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/64b54a01-00fb-446a-9def-aaa7127c3fef-prometheus-operator-tls podName:64b54a01-00fb-446a-9def-aaa7127c3fef nodeName:}" failed. No retries permitted until 2026-07-29 18:35:22.201990835 +0000 UTC m=+57.427659705 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/64b54a01-00fb-446a-9def-aaa7127c3fef-prometheus-operator-tls") pod "prometheus-operator-84d69f794d-cxj67" (UID: "64b54a01-00fb-446a-9def-aaa7127c3fef") : secret "prometheus-operator-tls" not found Jul 29 18:35:21.702526 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:21.702504 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/64b54a01-00fb-446a-9def-aaa7127c3fef-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-cxj67\" (UID: \"64b54a01-00fb-446a-9def-aaa7127c3fef\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-cxj67" Jul 29 18:35:21.704416 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:21.704389 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/64b54a01-00fb-446a-9def-aaa7127c3fef-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-cxj67\" (UID: \"64b54a01-00fb-446a-9def-aaa7127c3fef\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-cxj67" Jul 29 18:35:21.710527 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:21.710507 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rl9z5\" (UniqueName: \"kubernetes.io/projected/64b54a01-00fb-446a-9def-aaa7127c3fef-kube-api-access-rl9z5\") pod \"prometheus-operator-84d69f794d-cxj67\" (UID: \"64b54a01-00fb-446a-9def-aaa7127c3fef\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-cxj67" Jul 29 18:35:21.904921 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:21.904833 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:35:21.964651 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:21.964617 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:35:22.106399 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:22.106341 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:35:22.172283 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:22.172202 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:35:22.206415 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:22.206376 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/64b54a01-00fb-446a-9def-aaa7127c3fef-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-cxj67\" (UID: \"64b54a01-00fb-446a-9def-aaa7127c3fef\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-cxj67" Jul 29 18:35:22.209030 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:22.209002 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/64b54a01-00fb-446a-9def-aaa7127c3fef-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-cxj67\" (UID: \"64b54a01-00fb-446a-9def-aaa7127c3fef\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-cxj67" Jul 29 18:35:22.458404 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:22.458308 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-84d69f794d-cxj67" Jul 29 18:35:24.508908 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:24.508830 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-cxj67"] Jul 29 18:35:24.515082 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:35:24.515041 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64b54a01_00fb_446a_9def_aaa7127c3fef.slice/crio-7c908564e30a36d4d476d39a734d68dc0d92c3fa3eadeee8d5851ce7eee13501 WatchSource:0}: Error finding container 7c908564e30a36d4d476d39a734d68dc0d92c3fa3eadeee8d5851ce7eee13501: Status 404 returned error can't find the container with id 7c908564e30a36d4d476d39a734d68dc0d92c3fa3eadeee8d5851ce7eee13501 Jul 29 18:35:24.531694 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:24.531668 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-765c45df68-wbklv"] Jul 29 18:35:24.536811 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:35:24.536719 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcaf4f9c_b5cb_4f1d_83ed_72639860bee6.slice/crio-a6d5b18e207a379466085b78728311e51b8bd7a32cab4ce82dd7eb4b4098f809 WatchSource:0}: Error finding container a6d5b18e207a379466085b78728311e51b8bd7a32cab4ce82dd7eb4b4098f809: Status 404 returned error can't find the container with id a6d5b18e207a379466085b78728311e51b8bd7a32cab4ce82dd7eb4b4098f809 Jul 29 18:35:24.669095 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:24.669064 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-765c45df68-wbklv" event={"ID":"dcaf4f9c-b5cb-4f1d-83ed-72639860bee6","Type":"ContainerStarted","Data":"a6d5b18e207a379466085b78728311e51b8bd7a32cab4ce82dd7eb4b4098f809"} Jul 29 18:35:24.670250 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:24.670220 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-cxj67" event={"ID":"64b54a01-00fb-446a-9def-aaa7127c3fef","Type":"ContainerStarted","Data":"7c908564e30a36d4d476d39a734d68dc0d92c3fa3eadeee8d5851ce7eee13501"} Jul 29 18:35:24.671791 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:24.671759 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-gls76" event={"ID":"4e3cd5b3-9c9f-4282-a3ee-9ebb947273a9","Type":"ContainerStarted","Data":"863638c04766541bcf012d25f6fbee567dbd7c767a0d221aa3824af3b297b560"} Jul 29 18:35:24.671908 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:24.671820 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-gls76" event={"ID":"4e3cd5b3-9c9f-4282-a3ee-9ebb947273a9","Type":"ContainerStarted","Data":"870ab59825168c39d94a30a32d82ce960c5b2d1570f64e3ef617868923d1f6a7"} Jul 29 18:35:24.671908 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:24.671878 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-gls76" Jul 29 18:35:24.673274 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:24.673256 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-nr2lw" event={"ID":"36f140ba-4691-4e5e-b58f-9305091a175e","Type":"ContainerStarted","Data":"bf4cb13617e4bdb2b6181e18c869d844455243b4099305b98a820984d88b8984"} Jul 29 18:35:24.674683 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:24.674665 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-f9stl" event={"ID":"615e2b70-a2a4-4564-ab30-eb2ca3869414","Type":"ContainerStarted","Data":"92410e5350163f1750286dc36014085374d234e8c99c536d54c8b22dd90b7c64"} Jul 29 18:35:24.706848 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:24.706811 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-f9stl" podStartSLOduration=1.035046294 podStartE2EDuration="6.706798585s" totalImagesPullingTime="5.671752291s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 18:35:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 18:35:24.705543053 +0000 UTC m=+59.931211925" watchObservedRunningTime="2026-07-29 18:35:24.706798585 +0000 UTC m=+59.932467457" Jul 29 18:35:24.707015 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:24.706997 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-gls76" podStartSLOduration=1.3532481299999999 podStartE2EDuration="6.706993054s" totalImagesPullingTime="5.353744924s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 18:35:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 18:35:24.690050164 +0000 UTC m=+59.915719035" watchObservedRunningTime="2026-07-29 18:35:24.706993054 +0000 UTC m=+59.932661920" Jul 29 18:35:24.983039 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:24.982959 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:35:25.036472 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:25.036422 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:35:25.124700 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:25.124672 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:35:25.182356 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:25.182319 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:35:26.600991 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:26.600967 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-v4n5x" Jul 29 18:35:26.685507 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:26.685433 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-cxj67" event={"ID":"64b54a01-00fb-446a-9def-aaa7127c3fef","Type":"ContainerStarted","Data":"989122a72fef2e432d8c2cf294367082c0eddb8c2bb577e6d3f416ff02b1eab0"} Jul 29 18:35:26.685507 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:26.685492 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-cxj67" event={"ID":"64b54a01-00fb-446a-9def-aaa7127c3fef","Type":"ContainerStarted","Data":"ee8049bef8b3c7885bc0e254bac383db119579c1b62c11bd426e79bccc53a10c"} Jul 29 18:35:26.698331 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:26.698299 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:35:26.705932 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:26.705888 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-84d69f794d-cxj67" podStartSLOduration=3.858889457 podStartE2EDuration="5.70586989s" totalImagesPullingTime="1.846980433s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 18:35:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 18:35:26.70476158 +0000 UTC m=+61.930430455" watchObservedRunningTime="2026-07-29 18:35:26.70586989 +0000 UTC m=+61.931538764" Jul 29 18:35:26.799407 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:26.799328 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:35:26.900831 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:26.900795 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:35:27.313260 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:27.313224 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:35:27.404309 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:27.404254 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:35:28.696063 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:28.695996 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:35:29.026870 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:29.026837 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:35:29.066167 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:29.066126 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-b8f49f5c6-s4rh6"] Jul 29 18:35:29.068779 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:29.068746 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-s4rh6" Jul 29 18:35:29.073098 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:29.073067 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-bnnrx\"" Jul 29 18:35:29.073506 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:29.073489 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Jul 29 18:35:29.073626 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:29.073509 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Jul 29 18:35:29.088940 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:29.088915 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-b8f49f5c6-s4rh6"] Jul 29 18:35:29.131906 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:29.131877 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-tb6xb"] Jul 29 18:35:29.134584 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:29.134563 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-tb6xb" Jul 29 18:35:29.156497 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:29.156467 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 29 18:35:29.156613 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:29.156593 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-sn2rv\"" Jul 29 18:35:29.156678 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:29.156640 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 29 18:35:29.156678 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:29.156641 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 29 18:35:29.168233 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:29.168203 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f74a7c59-0701-4b47-bdcd-a314fc2600de-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-b8f49f5c6-s4rh6\" (UID: \"f74a7c59-0701-4b47-bdcd-a314fc2600de\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-s4rh6" Jul 29 18:35:29.168367 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:29.168237 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f74a7c59-0701-4b47-bdcd-a314fc2600de-metrics-client-ca\") pod \"openshift-state-metrics-b8f49f5c6-s4rh6\" (UID: \"f74a7c59-0701-4b47-bdcd-a314fc2600de\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-s4rh6" Jul 29 18:35:29.168367 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:29.168317 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2gkch\" (UniqueName: \"kubernetes.io/projected/f74a7c59-0701-4b47-bdcd-a314fc2600de-kube-api-access-2gkch\") pod \"openshift-state-metrics-b8f49f5c6-s4rh6\" (UID: \"f74a7c59-0701-4b47-bdcd-a314fc2600de\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-s4rh6" Jul 29 18:35:29.168455 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:29.168442 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/f74a7c59-0701-4b47-bdcd-a314fc2600de-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-s4rh6\" (UID: \"f74a7c59-0701-4b47-bdcd-a314fc2600de\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-s4rh6" Jul 29 18:35:29.236168 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:29.236130 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:35:29.269506 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:29.269472 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lmdzk\" (UniqueName: \"kubernetes.io/projected/677a8b8b-b142-4cba-b81a-fefa36ee1f69-kube-api-access-lmdzk\") pod \"node-exporter-tb6xb\" (UID: \"677a8b8b-b142-4cba-b81a-fefa36ee1f69\") " pod="openshift-monitoring/node-exporter-tb6xb" Jul 29 18:35:29.269686 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:29.269532 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/f74a7c59-0701-4b47-bdcd-a314fc2600de-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-s4rh6\" (UID: \"f74a7c59-0701-4b47-bdcd-a314fc2600de\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-s4rh6" Jul 29 18:35:29.269752 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:29.269710 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/677a8b8b-b142-4cba-b81a-fefa36ee1f69-node-exporter-tls\") pod \"node-exporter-tb6xb\" (UID: \"677a8b8b-b142-4cba-b81a-fefa36ee1f69\") " pod="openshift-monitoring/node-exporter-tb6xb" Jul 29 18:35:29.269813 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:29.269752 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/677a8b8b-b142-4cba-b81a-fefa36ee1f69-node-exporter-wtmp\") pod \"node-exporter-tb6xb\" (UID: \"677a8b8b-b142-4cba-b81a-fefa36ee1f69\") " pod="openshift-monitoring/node-exporter-tb6xb" Jul 29 18:35:29.269813 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:29.269782 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/677a8b8b-b142-4cba-b81a-fefa36ee1f69-node-exporter-accelerators-collector-config\") pod \"node-exporter-tb6xb\" (UID: \"677a8b8b-b142-4cba-b81a-fefa36ee1f69\") " pod="openshift-monitoring/node-exporter-tb6xb" Jul 29 18:35:29.269906 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:29.269846 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/677a8b8b-b142-4cba-b81a-fefa36ee1f69-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-tb6xb\" (UID: \"677a8b8b-b142-4cba-b81a-fefa36ee1f69\") " pod="openshift-monitoring/node-exporter-tb6xb" Jul 29 18:35:29.269906 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:29.269880 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f74a7c59-0701-4b47-bdcd-a314fc2600de-metrics-client-ca\") pod \"openshift-state-metrics-b8f49f5c6-s4rh6\" (UID: \"f74a7c59-0701-4b47-bdcd-a314fc2600de\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-s4rh6" Jul 29 18:35:29.270003 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:29.269926 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/677a8b8b-b142-4cba-b81a-fefa36ee1f69-sys\") pod \"node-exporter-tb6xb\" (UID: \"677a8b8b-b142-4cba-b81a-fefa36ee1f69\") " pod="openshift-monitoring/node-exporter-tb6xb" Jul 29 18:35:29.270003 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:29.269948 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/677a8b8b-b142-4cba-b81a-fefa36ee1f69-root\") pod \"node-exporter-tb6xb\" (UID: \"677a8b8b-b142-4cba-b81a-fefa36ee1f69\") " pod="openshift-monitoring/node-exporter-tb6xb" Jul 29 18:35:29.270003 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:29.269974 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/677a8b8b-b142-4cba-b81a-fefa36ee1f69-metrics-client-ca\") pod \"node-exporter-tb6xb\" (UID: \"677a8b8b-b142-4cba-b81a-fefa36ee1f69\") " pod="openshift-monitoring/node-exporter-tb6xb" Jul 29 18:35:29.270003 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:29.269999 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/677a8b8b-b142-4cba-b81a-fefa36ee1f69-node-exporter-textfile\") pod \"node-exporter-tb6xb\" (UID: \"677a8b8b-b142-4cba-b81a-fefa36ee1f69\") " pod="openshift-monitoring/node-exporter-tb6xb" Jul 29 18:35:29.270184 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:29.270046 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2gkch\" (UniqueName: \"kubernetes.io/projected/f74a7c59-0701-4b47-bdcd-a314fc2600de-kube-api-access-2gkch\") pod \"openshift-state-metrics-b8f49f5c6-s4rh6\" (UID: \"f74a7c59-0701-4b47-bdcd-a314fc2600de\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-s4rh6" Jul 29 18:35:29.270234 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:29.270190 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f74a7c59-0701-4b47-bdcd-a314fc2600de-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-b8f49f5c6-s4rh6\" (UID: \"f74a7c59-0701-4b47-bdcd-a314fc2600de\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-s4rh6" Jul 29 18:35:29.270614 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:29.270581 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f74a7c59-0701-4b47-bdcd-a314fc2600de-metrics-client-ca\") pod \"openshift-state-metrics-b8f49f5c6-s4rh6\" (UID: \"f74a7c59-0701-4b47-bdcd-a314fc2600de\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-s4rh6" Jul 29 18:35:29.272512 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:29.272488 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/f74a7c59-0701-4b47-bdcd-a314fc2600de-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-s4rh6\" (UID: \"f74a7c59-0701-4b47-bdcd-a314fc2600de\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-s4rh6" Jul 29 18:35:29.272803 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:29.272777 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f74a7c59-0701-4b47-bdcd-a314fc2600de-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-b8f49f5c6-s4rh6\" (UID: \"f74a7c59-0701-4b47-bdcd-a314fc2600de\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-s4rh6" Jul 29 18:35:29.291776 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:29.291720 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2gkch\" (UniqueName: \"kubernetes.io/projected/f74a7c59-0701-4b47-bdcd-a314fc2600de-kube-api-access-2gkch\") pod \"openshift-state-metrics-b8f49f5c6-s4rh6\" (UID: \"f74a7c59-0701-4b47-bdcd-a314fc2600de\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-s4rh6" Jul 29 18:35:29.352288 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:29.352252 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:35:29.370584 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:29.370556 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/677a8b8b-b142-4cba-b81a-fefa36ee1f69-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-tb6xb\" (UID: \"677a8b8b-b142-4cba-b81a-fefa36ee1f69\") " pod="openshift-monitoring/node-exporter-tb6xb" Jul 29 18:35:29.370737 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:29.370603 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/677a8b8b-b142-4cba-b81a-fefa36ee1f69-sys\") pod \"node-exporter-tb6xb\" (UID: \"677a8b8b-b142-4cba-b81a-fefa36ee1f69\") " pod="openshift-monitoring/node-exporter-tb6xb" Jul 29 18:35:29.370737 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:29.370619 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/677a8b8b-b142-4cba-b81a-fefa36ee1f69-root\") pod \"node-exporter-tb6xb\" (UID: \"677a8b8b-b142-4cba-b81a-fefa36ee1f69\") " pod="openshift-monitoring/node-exporter-tb6xb" Jul 29 18:35:29.370737 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:29.370639 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/677a8b8b-b142-4cba-b81a-fefa36ee1f69-metrics-client-ca\") pod \"node-exporter-tb6xb\" (UID: \"677a8b8b-b142-4cba-b81a-fefa36ee1f69\") " pod="openshift-monitoring/node-exporter-tb6xb" Jul 29 18:35:29.370737 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:29.370664 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/677a8b8b-b142-4cba-b81a-fefa36ee1f69-node-exporter-textfile\") pod \"node-exporter-tb6xb\" (UID: \"677a8b8b-b142-4cba-b81a-fefa36ee1f69\") " pod="openshift-monitoring/node-exporter-tb6xb" Jul 29 18:35:29.370737 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:29.370690 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/677a8b8b-b142-4cba-b81a-fefa36ee1f69-sys\") pod \"node-exporter-tb6xb\" (UID: \"677a8b8b-b142-4cba-b81a-fefa36ee1f69\") " pod="openshift-monitoring/node-exporter-tb6xb" Jul 29 18:35:29.370737 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:29.370718 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lmdzk\" (UniqueName: \"kubernetes.io/projected/677a8b8b-b142-4cba-b81a-fefa36ee1f69-kube-api-access-lmdzk\") pod \"node-exporter-tb6xb\" (UID: \"677a8b8b-b142-4cba-b81a-fefa36ee1f69\") " pod="openshift-monitoring/node-exporter-tb6xb" Jul 29 18:35:29.371014 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:29.370773 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/677a8b8b-b142-4cba-b81a-fefa36ee1f69-node-exporter-tls\") pod \"node-exporter-tb6xb\" (UID: \"677a8b8b-b142-4cba-b81a-fefa36ee1f69\") " pod="openshift-monitoring/node-exporter-tb6xb" Jul 29 18:35:29.371014 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:29.370793 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/677a8b8b-b142-4cba-b81a-fefa36ee1f69-node-exporter-wtmp\") pod \"node-exporter-tb6xb\" (UID: \"677a8b8b-b142-4cba-b81a-fefa36ee1f69\") " pod="openshift-monitoring/node-exporter-tb6xb" Jul 29 18:35:29.371014 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:29.370692 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/677a8b8b-b142-4cba-b81a-fefa36ee1f69-root\") pod \"node-exporter-tb6xb\" (UID: \"677a8b8b-b142-4cba-b81a-fefa36ee1f69\") " pod="openshift-monitoring/node-exporter-tb6xb" Jul 29 18:35:29.371014 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:29.370817 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/677a8b8b-b142-4cba-b81a-fefa36ee1f69-node-exporter-accelerators-collector-config\") pod \"node-exporter-tb6xb\" (UID: \"677a8b8b-b142-4cba-b81a-fefa36ee1f69\") " pod="openshift-monitoring/node-exporter-tb6xb" Jul 29 18:35:29.371014 ip-10-0-140-205 kubenswrapper[2560]: E0729 18:35:29.370890 2560 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 29 18:35:29.371014 ip-10-0-140-205 kubenswrapper[2560]: E0729 18:35:29.370961 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/677a8b8b-b142-4cba-b81a-fefa36ee1f69-node-exporter-tls podName:677a8b8b-b142-4cba-b81a-fefa36ee1f69 nodeName:}" failed. No retries permitted until 2026-07-29 18:35:29.870942716 +0000 UTC m=+65.096611566 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/677a8b8b-b142-4cba-b81a-fefa36ee1f69-node-exporter-tls") pod "node-exporter-tb6xb" (UID: "677a8b8b-b142-4cba-b81a-fefa36ee1f69") : secret "node-exporter-tls" not found Jul 29 18:35:29.371304 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:29.371091 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/677a8b8b-b142-4cba-b81a-fefa36ee1f69-node-exporter-wtmp\") pod \"node-exporter-tb6xb\" (UID: \"677a8b8b-b142-4cba-b81a-fefa36ee1f69\") " pod="openshift-monitoring/node-exporter-tb6xb" Jul 29 18:35:29.371462 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:29.371433 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/677a8b8b-b142-4cba-b81a-fefa36ee1f69-node-exporter-textfile\") pod \"node-exporter-tb6xb\" (UID: \"677a8b8b-b142-4cba-b81a-fefa36ee1f69\") " pod="openshift-monitoring/node-exporter-tb6xb" Jul 29 18:35:29.371825 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:29.371803 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/677a8b8b-b142-4cba-b81a-fefa36ee1f69-node-exporter-accelerators-collector-config\") pod \"node-exporter-tb6xb\" (UID: \"677a8b8b-b142-4cba-b81a-fefa36ee1f69\") " pod="openshift-monitoring/node-exporter-tb6xb" Jul 29 18:35:29.371894 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:29.371803 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/677a8b8b-b142-4cba-b81a-fefa36ee1f69-metrics-client-ca\") pod \"node-exporter-tb6xb\" (UID: \"677a8b8b-b142-4cba-b81a-fefa36ee1f69\") " pod="openshift-monitoring/node-exporter-tb6xb" Jul 29 18:35:29.373695 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:29.373667 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/677a8b8b-b142-4cba-b81a-fefa36ee1f69-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-tb6xb\" (UID: \"677a8b8b-b142-4cba-b81a-fefa36ee1f69\") " pod="openshift-monitoring/node-exporter-tb6xb" Jul 29 18:35:29.379236 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:29.379211 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-s4rh6" Jul 29 18:35:29.382846 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:29.382824 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmdzk\" (UniqueName: \"kubernetes.io/projected/677a8b8b-b142-4cba-b81a-fefa36ee1f69-kube-api-access-lmdzk\") pod \"node-exporter-tb6xb\" (UID: \"677a8b8b-b142-4cba-b81a-fefa36ee1f69\") " pod="openshift-monitoring/node-exporter-tb6xb" Jul 29 18:35:29.444366 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:29.444313 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:35:29.545146 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:29.545081 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:35:29.650406 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:29.650372 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:35:29.759261 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:29.759229 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:35:29.875600 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:29.875557 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/677a8b8b-b142-4cba-b81a-fefa36ee1f69-node-exporter-tls\") pod \"node-exporter-tb6xb\" (UID: \"677a8b8b-b142-4cba-b81a-fefa36ee1f69\") " pod="openshift-monitoring/node-exporter-tb6xb" Jul 29 18:35:29.878404 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:29.878336 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/677a8b8b-b142-4cba-b81a-fefa36ee1f69-node-exporter-tls\") pod \"node-exporter-tb6xb\" (UID: \"677a8b8b-b142-4cba-b81a-fefa36ee1f69\") " pod="openshift-monitoring/node-exporter-tb6xb" Jul 29 18:35:30.010281 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:30.010158 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-b8f49f5c6-s4rh6"] Jul 29 18:35:30.014060 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:35:30.014037 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf74a7c59_0701_4b47_bdcd_a314fc2600de.slice/crio-96fdafa33da1b09262d83062771147c3eeebca195113ae52ab78f6fa0d8cbf00 WatchSource:0}: Error finding container 96fdafa33da1b09262d83062771147c3eeebca195113ae52ab78f6fa0d8cbf00: Status 404 returned error can't find the container with id 96fdafa33da1b09262d83062771147c3eeebca195113ae52ab78f6fa0d8cbf00 Jul 29 18:35:30.044023 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:30.043991 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-tb6xb" Jul 29 18:35:30.626590 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:30.626558 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:35:30.703321 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:30.703279 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-nr2lw" event={"ID":"36f140ba-4691-4e5e-b58f-9305091a175e","Type":"ContainerStarted","Data":"dcddc304e8b6cb8c2b46b03d4f30dc3d4f698b34fc8d46efe8ac767c9ace9e67"} Jul 29 18:35:30.705441 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:30.705410 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-s4rh6" event={"ID":"f74a7c59-0701-4b47-bdcd-a314fc2600de","Type":"ContainerStarted","Data":"ac24133757bd0f0ca1c4aa4f34d0f28bff03a0d25cf4c52aeec6e353114e9feb"} Jul 29 18:35:30.705576 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:30.705465 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-s4rh6" event={"ID":"f74a7c59-0701-4b47-bdcd-a314fc2600de","Type":"ContainerStarted","Data":"9cf006ee302a25c864eea71a7ff31437c5d4e23545498c2ee3c11915fdfc40f4"} Jul 29 18:35:30.705576 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:30.705497 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-s4rh6" event={"ID":"f74a7c59-0701-4b47-bdcd-a314fc2600de","Type":"ContainerStarted","Data":"96fdafa33da1b09262d83062771147c3eeebca195113ae52ab78f6fa0d8cbf00"} Jul 29 18:35:30.706752 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:30.706711 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-tb6xb" event={"ID":"677a8b8b-b142-4cba-b81a-fefa36ee1f69","Type":"ContainerStarted","Data":"c2687dcd54080247715177a2c216b0032f53b154fc06867d1a447e14bb8ec124"} Jul 29 18:35:30.708566 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:30.708544 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-765c45df68-wbklv" event={"ID":"dcaf4f9c-b5cb-4f1d-83ed-72639860bee6","Type":"ContainerStarted","Data":"254a77d41a419fe58d5929eb9eadf850e52d22bb5f6f62f3eca560d835f9b1a2"} Jul 29 18:35:30.729651 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:30.729617 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:35:30.729996 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:30.729954 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-765c45df68-wbklv" podStartSLOduration=5.393425109 podStartE2EDuration="10.729939672s" totalImagesPullingTime="5.336514563s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 18:35:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 18:35:30.728430875 +0000 UTC m=+65.954099752" watchObservedRunningTime="2026-07-29 18:35:30.729939672 +0000 UTC m=+65.955608541" Jul 29 18:35:30.942251 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:30.942160 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:35:30.988475 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:30.988441 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70307ff7-a0ca-4a2b-95be-94a5dde5d96e-metrics-certs\") pod \"network-metrics-daemon-mrshg\" (UID: \"70307ff7-a0ca-4a2b-95be-94a5dde5d96e\") " pod="openshift-multus/network-metrics-daemon-mrshg" Jul 29 18:35:30.988662 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:30.988494 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/13fd7656-9a1f-4221-a5ed-6ebecf3664c9-original-pull-secret\") pod \"global-pull-secret-syncer-zbwwp\" (UID: \"13fd7656-9a1f-4221-a5ed-6ebecf3664c9\") " pod="kube-system/global-pull-secret-syncer-zbwwp" Jul 29 18:35:30.991484 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:30.991234 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 29 18:35:30.991484 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:30.991234 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 29 18:35:31.002111 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:31.002054 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/13fd7656-9a1f-4221-a5ed-6ebecf3664c9-original-pull-secret\") pod \"global-pull-secret-syncer-zbwwp\" (UID: \"13fd7656-9a1f-4221-a5ed-6ebecf3664c9\") " pod="kube-system/global-pull-secret-syncer-zbwwp" Jul 29 18:35:31.003563 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:31.003537 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70307ff7-a0ca-4a2b-95be-94a5dde5d96e-metrics-certs\") pod \"network-metrics-daemon-mrshg\" (UID: \"70307ff7-a0ca-4a2b-95be-94a5dde5d96e\") " pod="openshift-multus/network-metrics-daemon-mrshg" Jul 29 18:35:31.076788 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:31.076753 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-mfhf2\"" Jul 29 18:35:31.085116 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:31.084870 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mrshg" Jul 29 18:35:31.089721 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:31.089699 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phzf8\" (UniqueName: \"kubernetes.io/projected/88336825-e84e-4595-9f68-f4d6b4e6edda-kube-api-access-phzf8\") pod \"network-check-target-8qdhk\" (UID: \"88336825-e84e-4595-9f68-f4d6b4e6edda\") " pod="openshift-network-diagnostics/network-check-target-8qdhk" Jul 29 18:35:31.092648 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:31.092628 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 29 18:35:31.103290 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:31.103265 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 29 18:35:31.114807 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:31.114760 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-phzf8\" (UniqueName: \"kubernetes.io/projected/88336825-e84e-4595-9f68-f4d6b4e6edda-kube-api-access-phzf8\") pod \"network-check-target-8qdhk\" (UID: \"88336825-e84e-4595-9f68-f4d6b4e6edda\") " pod="openshift-network-diagnostics/network-check-target-8qdhk" Jul 29 18:35:31.152127 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:31.152094 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:35:31.225147 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:31.225060 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:35:31.227335 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:31.227300 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-765c45df68-wbklv" Jul 29 18:35:31.227566 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:31.227521 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-765c45df68-wbklv" Jul 29 18:35:31.234163 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:31.234140 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-765c45df68-wbklv" Jul 29 18:35:31.256529 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:31.256297 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zbwwp" Jul 29 18:35:31.367561 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:31.367536 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-hfcvr\"" Jul 29 18:35:31.375749 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:31.375714 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8qdhk" Jul 29 18:35:31.717709 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:31.717623 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-s4rh6" event={"ID":"f74a7c59-0701-4b47-bdcd-a314fc2600de","Type":"ContainerStarted","Data":"d6b7e95cf2a158d181fb2de9d5308c238ef72cbe7245513a3569dd0ea434dcd2"} Jul 29 18:35:31.724501 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:31.724424 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-765c45df68-wbklv" Jul 29 18:35:31.739259 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:31.739088 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-s4rh6" podStartSLOduration=1.2733789070000001 podStartE2EDuration="2.73906867s" totalImagesPullingTime="1.465689763s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 18:35:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 18:35:31.738242814 +0000 UTC m=+66.963911713" watchObservedRunningTime="2026-07-29 18:35:31.73906867 +0000 UTC m=+66.964737542" Jul 29 18:35:31.769454 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:31.769385 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-mrshg"] Jul 29 18:35:31.771029 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:35:31.771006 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70307ff7_a0ca_4a2b_95be_94a5dde5d96e.slice/crio-2844dc432cf189e8b7b4a95e7b7bc71f9c93bb1f85d8a38ef185a32eaa81cb70 WatchSource:0}: Error finding container 2844dc432cf189e8b7b4a95e7b7bc71f9c93bb1f85d8a38ef185a32eaa81cb70: Status 404 returned error can't find the container with id 2844dc432cf189e8b7b4a95e7b7bc71f9c93bb1f85d8a38ef185a32eaa81cb70 Jul 29 18:35:31.812825 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:31.812773 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:35:31.885042 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:31.884971 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:35:31.943148 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:31.943113 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:35:31.994220 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:31.994167 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-8qdhk"] Jul 29 18:35:31.995799 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:31.995211 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-zbwwp"] Jul 29 18:35:32.013495 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:32.013473 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-76dc89c749-sn6bb"] Jul 29 18:35:32.017218 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:32.017199 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-76dc89c749-sn6bb" Jul 29 18:35:32.019965 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:32.019939 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Jul 29 18:35:32.020337 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:32.020315 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Jul 29 18:35:32.020473 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:32.020340 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Jul 29 18:35:32.020473 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:32.020389 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-f82ws\"" Jul 29 18:35:32.020473 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:32.020414 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Jul 29 18:35:32.020473 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:32.020340 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Jul 29 18:35:32.020649 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:32.020515 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-eodq2v4f0b53u\"" Jul 29 18:35:32.033224 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:32.033205 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-76dc89c749-sn6bb"] Jul 29 18:35:32.060552 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:32.060523 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:35:32.099739 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:32.099706 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3eb9d5f4-642e-4158-83d6-0d4bc40efd2f-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-76dc89c749-sn6bb\" (UID: \"3eb9d5f4-642e-4158-83d6-0d4bc40efd2f\") " pod="openshift-monitoring/thanos-querier-76dc89c749-sn6bb" Jul 29 18:35:32.099896 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:32.099750 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3eb9d5f4-642e-4158-83d6-0d4bc40efd2f-metrics-client-ca\") pod \"thanos-querier-76dc89c749-sn6bb\" (UID: \"3eb9d5f4-642e-4158-83d6-0d4bc40efd2f\") " pod="openshift-monitoring/thanos-querier-76dc89c749-sn6bb" Jul 29 18:35:32.099896 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:32.099771 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/3eb9d5f4-642e-4158-83d6-0d4bc40efd2f-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-76dc89c749-sn6bb\" (UID: \"3eb9d5f4-642e-4158-83d6-0d4bc40efd2f\") " pod="openshift-monitoring/thanos-querier-76dc89c749-sn6bb" Jul 29 18:35:32.099896 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:32.099799 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxccx\" (UniqueName: \"kubernetes.io/projected/3eb9d5f4-642e-4158-83d6-0d4bc40efd2f-kube-api-access-pxccx\") pod \"thanos-querier-76dc89c749-sn6bb\" (UID: \"3eb9d5f4-642e-4158-83d6-0d4bc40efd2f\") " pod="openshift-monitoring/thanos-querier-76dc89c749-sn6bb" Jul 29 18:35:32.099896 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:32.099882 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3eb9d5f4-642e-4158-83d6-0d4bc40efd2f-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-76dc89c749-sn6bb\" (UID: \"3eb9d5f4-642e-4158-83d6-0d4bc40efd2f\") " pod="openshift-monitoring/thanos-querier-76dc89c749-sn6bb" Jul 29 18:35:32.100092 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:32.099925 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/3eb9d5f4-642e-4158-83d6-0d4bc40efd2f-secret-thanos-querier-tls\") pod \"thanos-querier-76dc89c749-sn6bb\" (UID: \"3eb9d5f4-642e-4158-83d6-0d4bc40efd2f\") " pod="openshift-monitoring/thanos-querier-76dc89c749-sn6bb" Jul 29 18:35:32.100092 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:32.100000 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/3eb9d5f4-642e-4158-83d6-0d4bc40efd2f-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-76dc89c749-sn6bb\" (UID: \"3eb9d5f4-642e-4158-83d6-0d4bc40efd2f\") " pod="openshift-monitoring/thanos-querier-76dc89c749-sn6bb" Jul 29 18:35:32.100092 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:32.100040 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3eb9d5f4-642e-4158-83d6-0d4bc40efd2f-secret-grpc-tls\") pod \"thanos-querier-76dc89c749-sn6bb\" (UID: \"3eb9d5f4-642e-4158-83d6-0d4bc40efd2f\") " pod="openshift-monitoring/thanos-querier-76dc89c749-sn6bb" Jul 29 18:35:32.200896 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:32.200801 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3eb9d5f4-642e-4158-83d6-0d4bc40efd2f-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-76dc89c749-sn6bb\" (UID: \"3eb9d5f4-642e-4158-83d6-0d4bc40efd2f\") " pod="openshift-monitoring/thanos-querier-76dc89c749-sn6bb" Jul 29 18:35:32.200896 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:32.200858 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3eb9d5f4-642e-4158-83d6-0d4bc40efd2f-metrics-client-ca\") pod \"thanos-querier-76dc89c749-sn6bb\" (UID: \"3eb9d5f4-642e-4158-83d6-0d4bc40efd2f\") " pod="openshift-monitoring/thanos-querier-76dc89c749-sn6bb" Jul 29 18:35:32.200896 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:32.200888 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/3eb9d5f4-642e-4158-83d6-0d4bc40efd2f-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-76dc89c749-sn6bb\" (UID: \"3eb9d5f4-642e-4158-83d6-0d4bc40efd2f\") " pod="openshift-monitoring/thanos-querier-76dc89c749-sn6bb" Jul 29 18:35:32.201176 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:32.200912 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxccx\" (UniqueName: \"kubernetes.io/projected/3eb9d5f4-642e-4158-83d6-0d4bc40efd2f-kube-api-access-pxccx\") pod \"thanos-querier-76dc89c749-sn6bb\" (UID: \"3eb9d5f4-642e-4158-83d6-0d4bc40efd2f\") " pod="openshift-monitoring/thanos-querier-76dc89c749-sn6bb" Jul 29 18:35:32.201176 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:32.200944 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3eb9d5f4-642e-4158-83d6-0d4bc40efd2f-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-76dc89c749-sn6bb\" (UID: \"3eb9d5f4-642e-4158-83d6-0d4bc40efd2f\") " pod="openshift-monitoring/thanos-querier-76dc89c749-sn6bb" Jul 29 18:35:32.201176 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:32.200969 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/3eb9d5f4-642e-4158-83d6-0d4bc40efd2f-secret-thanos-querier-tls\") pod \"thanos-querier-76dc89c749-sn6bb\" (UID: \"3eb9d5f4-642e-4158-83d6-0d4bc40efd2f\") " pod="openshift-monitoring/thanos-querier-76dc89c749-sn6bb" Jul 29 18:35:32.201176 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:32.201015 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/3eb9d5f4-642e-4158-83d6-0d4bc40efd2f-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-76dc89c749-sn6bb\" (UID: \"3eb9d5f4-642e-4158-83d6-0d4bc40efd2f\") " pod="openshift-monitoring/thanos-querier-76dc89c749-sn6bb" Jul 29 18:35:32.201176 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:32.201063 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3eb9d5f4-642e-4158-83d6-0d4bc40efd2f-secret-grpc-tls\") pod \"thanos-querier-76dc89c749-sn6bb\" (UID: \"3eb9d5f4-642e-4158-83d6-0d4bc40efd2f\") " pod="openshift-monitoring/thanos-querier-76dc89c749-sn6bb" Jul 29 18:35:32.201764 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:32.201707 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3eb9d5f4-642e-4158-83d6-0d4bc40efd2f-metrics-client-ca\") pod \"thanos-querier-76dc89c749-sn6bb\" (UID: \"3eb9d5f4-642e-4158-83d6-0d4bc40efd2f\") " pod="openshift-monitoring/thanos-querier-76dc89c749-sn6bb" Jul 29 18:35:32.204085 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:32.204029 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/3eb9d5f4-642e-4158-83d6-0d4bc40efd2f-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-76dc89c749-sn6bb\" (UID: \"3eb9d5f4-642e-4158-83d6-0d4bc40efd2f\") " pod="openshift-monitoring/thanos-querier-76dc89c749-sn6bb" Jul 29 18:35:32.204237 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:32.204181 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3eb9d5f4-642e-4158-83d6-0d4bc40efd2f-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-76dc89c749-sn6bb\" (UID: \"3eb9d5f4-642e-4158-83d6-0d4bc40efd2f\") " pod="openshift-monitoring/thanos-querier-76dc89c749-sn6bb" Jul 29 18:35:32.204237 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:32.204223 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/3eb9d5f4-642e-4158-83d6-0d4bc40efd2f-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-76dc89c749-sn6bb\" (UID: \"3eb9d5f4-642e-4158-83d6-0d4bc40efd2f\") " pod="openshift-monitoring/thanos-querier-76dc89c749-sn6bb" Jul 29 18:35:32.204574 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:32.204553 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3eb9d5f4-642e-4158-83d6-0d4bc40efd2f-secret-grpc-tls\") pod \"thanos-querier-76dc89c749-sn6bb\" (UID: \"3eb9d5f4-642e-4158-83d6-0d4bc40efd2f\") " pod="openshift-monitoring/thanos-querier-76dc89c749-sn6bb" Jul 29 18:35:32.204719 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:32.204696 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/3eb9d5f4-642e-4158-83d6-0d4bc40efd2f-secret-thanos-querier-tls\") pod \"thanos-querier-76dc89c749-sn6bb\" (UID: \"3eb9d5f4-642e-4158-83d6-0d4bc40efd2f\") " pod="openshift-monitoring/thanos-querier-76dc89c749-sn6bb" Jul 29 18:35:32.204806 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:32.204729 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3eb9d5f4-642e-4158-83d6-0d4bc40efd2f-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-76dc89c749-sn6bb\" (UID: \"3eb9d5f4-642e-4158-83d6-0d4bc40efd2f\") " pod="openshift-monitoring/thanos-querier-76dc89c749-sn6bb" Jul 29 18:35:32.209340 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:32.209311 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxccx\" (UniqueName: \"kubernetes.io/projected/3eb9d5f4-642e-4158-83d6-0d4bc40efd2f-kube-api-access-pxccx\") pod \"thanos-querier-76dc89c749-sn6bb\" (UID: \"3eb9d5f4-642e-4158-83d6-0d4bc40efd2f\") " pod="openshift-monitoring/thanos-querier-76dc89c749-sn6bb" Jul 29 18:35:32.222611 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:32.222576 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:35:32.323050 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:32.323015 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:35:32.329080 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:32.329054 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-76dc89c749-sn6bb" Jul 29 18:35:32.382164 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:32.382137 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:35:32.721881 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:32.721846 2560 generic.go:356] "Generic (PLEG): container finished" podID="677a8b8b-b142-4cba-b81a-fefa36ee1f69" containerID="2634afb5498f1f9e206ee80a1792c504574b4ea1cc071f177b8775a5efbfa656" exitCode=0 Jul 29 18:35:32.722060 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:32.721929 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-tb6xb" event={"ID":"677a8b8b-b142-4cba-b81a-fefa36ee1f69","Type":"ContainerDied","Data":"2634afb5498f1f9e206ee80a1792c504574b4ea1cc071f177b8775a5efbfa656"} Jul 29 18:35:32.723564 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:32.723521 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-mrshg" event={"ID":"70307ff7-a0ca-4a2b-95be-94a5dde5d96e","Type":"ContainerStarted","Data":"2844dc432cf189e8b7b4a95e7b7bc71f9c93bb1f85d8a38ef185a32eaa81cb70"} Jul 29 18:35:33.188753 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:33.188711 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:35:33.296538 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:33.296495 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:35:33.556015 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:33.555978 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-784b5c7fdf-9xt45"] Jul 29 18:35:33.583165 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:33.583134 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-784b5c7fdf-9xt45"] Jul 29 18:35:33.583325 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:33.583258 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-784b5c7fdf-9xt45" Jul 29 18:35:33.586501 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:33.586482 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Jul 29 18:35:33.586708 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:33.586693 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Jul 29 18:35:33.586788 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:33.586765 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Jul 29 18:35:33.586838 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:33.586794 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 29 18:35:33.586961 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:33.586947 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-b994r\"" Jul 29 18:35:33.587093 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:33.587075 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-4kl6v5ah94kkk\"" Jul 29 18:35:33.650810 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:35:33.649811 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88336825_e84e_4595_9f68_f4d6b4e6edda.slice/crio-3989c7dbeceb69c096b7323ae15761139dd69fedc616379c981dba624bad70ca WatchSource:0}: Error finding container 3989c7dbeceb69c096b7323ae15761139dd69fedc616379c981dba624bad70ca: Status 404 returned error can't find the container with id 3989c7dbeceb69c096b7323ae15761139dd69fedc616379c981dba624bad70ca Jul 29 18:35:33.651317 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:35:33.651017 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13fd7656_9a1f_4221_a5ed_6ebecf3664c9.slice/crio-bac7ef475878e7c77c98d4f790e9b4023a8ca3d2fa80ea1163dfd22e8c102cbf WatchSource:0}: Error finding container bac7ef475878e7c77c98d4f790e9b4023a8ca3d2fa80ea1163dfd22e8c102cbf: Status 404 returned error can't find the container with id bac7ef475878e7c77c98d4f790e9b4023a8ca3d2fa80ea1163dfd22e8c102cbf Jul 29 18:35:33.686797 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:33.685734 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-68b7fc9574-xpcxx"] Jul 29 18:35:33.689252 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:33.689228 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-xpcxx" Jul 29 18:35:33.689749 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:33.689723 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:35:33.691932 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:33.691721 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Jul 29 18:35:33.691932 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:33.691722 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-8zjck\"" Jul 29 18:35:33.696979 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:33.696933 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-68b7fc9574-xpcxx"] Jul 29 18:35:33.712338 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:33.712313 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/5603d54b-3265-4734-8526-72296a9c600e-audit-log\") pod \"metrics-server-784b5c7fdf-9xt45\" (UID: \"5603d54b-3265-4734-8526-72296a9c600e\") " pod="openshift-monitoring/metrics-server-784b5c7fdf-9xt45" Jul 29 18:35:33.712522 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:33.712406 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5603d54b-3265-4734-8526-72296a9c600e-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-784b5c7fdf-9xt45\" (UID: \"5603d54b-3265-4734-8526-72296a9c600e\") " pod="openshift-monitoring/metrics-server-784b5c7fdf-9xt45" Jul 29 18:35:33.712522 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:33.712456 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/5603d54b-3265-4734-8526-72296a9c600e-secret-metrics-server-tls\") pod \"metrics-server-784b5c7fdf-9xt45\" (UID: \"5603d54b-3265-4734-8526-72296a9c600e\") " pod="openshift-monitoring/metrics-server-784b5c7fdf-9xt45" Jul 29 18:35:33.712522 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:33.712478 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/5603d54b-3265-4734-8526-72296a9c600e-client-ca-bundle\") pod \"metrics-server-784b5c7fdf-9xt45\" (UID: \"5603d54b-3265-4734-8526-72296a9c600e\") " pod="openshift-monitoring/metrics-server-784b5c7fdf-9xt45" Jul 29 18:35:33.712522 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:33.712510 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/5603d54b-3265-4734-8526-72296a9c600e-metrics-server-audit-profiles\") pod \"metrics-server-784b5c7fdf-9xt45\" (UID: \"5603d54b-3265-4734-8526-72296a9c600e\") " pod="openshift-monitoring/metrics-server-784b5c7fdf-9xt45" Jul 29 18:35:33.712721 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:33.712536 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v67zv\" (UniqueName: \"kubernetes.io/projected/5603d54b-3265-4734-8526-72296a9c600e-kube-api-access-v67zv\") pod \"metrics-server-784b5c7fdf-9xt45\" (UID: \"5603d54b-3265-4734-8526-72296a9c600e\") " pod="openshift-monitoring/metrics-server-784b5c7fdf-9xt45" Jul 29 18:35:33.712721 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:33.712607 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/5603d54b-3265-4734-8526-72296a9c600e-secret-metrics-server-client-certs\") pod \"metrics-server-784b5c7fdf-9xt45\" (UID: \"5603d54b-3265-4734-8526-72296a9c600e\") " pod="openshift-monitoring/metrics-server-784b5c7fdf-9xt45" Jul 29 18:35:33.729262 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:33.729237 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-8qdhk" event={"ID":"88336825-e84e-4595-9f68-f4d6b4e6edda","Type":"ContainerStarted","Data":"3989c7dbeceb69c096b7323ae15761139dd69fedc616379c981dba624bad70ca"} Jul 29 18:35:33.731152 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:33.730984 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-zbwwp" event={"ID":"13fd7656-9a1f-4221-a5ed-6ebecf3664c9","Type":"ContainerStarted","Data":"bac7ef475878e7c77c98d4f790e9b4023a8ca3d2fa80ea1163dfd22e8c102cbf"} Jul 29 18:35:33.801941 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:33.801913 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:35:33.815068 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:33.813779 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5603d54b-3265-4734-8526-72296a9c600e-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-784b5c7fdf-9xt45\" (UID: \"5603d54b-3265-4734-8526-72296a9c600e\") " pod="openshift-monitoring/metrics-server-784b5c7fdf-9xt45" Jul 29 18:35:33.815068 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:33.814404 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lzd8q\" (UniqueName: \"kubernetes.io/projected/233d2ddf-e1b6-4ee9-8ed3-4706f4fd5260-kube-api-access-lzd8q\") pod \"monitoring-plugin-68b7fc9574-xpcxx\" (UID: \"233d2ddf-e1b6-4ee9-8ed3-4706f4fd5260\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-xpcxx" Jul 29 18:35:33.815068 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:33.814488 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5603d54b-3265-4734-8526-72296a9c600e-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-784b5c7fdf-9xt45\" (UID: \"5603d54b-3265-4734-8526-72296a9c600e\") " pod="openshift-monitoring/metrics-server-784b5c7fdf-9xt45" Jul 29 18:35:33.815068 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:33.814771 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/5603d54b-3265-4734-8526-72296a9c600e-secret-metrics-server-tls\") pod \"metrics-server-784b5c7fdf-9xt45\" (UID: \"5603d54b-3265-4734-8526-72296a9c600e\") " pod="openshift-monitoring/metrics-server-784b5c7fdf-9xt45" Jul 29 18:35:33.815068 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:33.814972 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/5603d54b-3265-4734-8526-72296a9c600e-client-ca-bundle\") pod \"metrics-server-784b5c7fdf-9xt45\" (UID: \"5603d54b-3265-4734-8526-72296a9c600e\") " pod="openshift-monitoring/metrics-server-784b5c7fdf-9xt45" Jul 29 18:35:33.815068 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:33.815016 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/5603d54b-3265-4734-8526-72296a9c600e-metrics-server-audit-profiles\") pod \"metrics-server-784b5c7fdf-9xt45\" (UID: \"5603d54b-3265-4734-8526-72296a9c600e\") " pod="openshift-monitoring/metrics-server-784b5c7fdf-9xt45" Jul 29 18:35:33.815068 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:33.815045 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v67zv\" (UniqueName: \"kubernetes.io/projected/5603d54b-3265-4734-8526-72296a9c600e-kube-api-access-v67zv\") pod \"metrics-server-784b5c7fdf-9xt45\" (UID: \"5603d54b-3265-4734-8526-72296a9c600e\") " pod="openshift-monitoring/metrics-server-784b5c7fdf-9xt45" Jul 29 18:35:33.815483 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:33.815125 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/233d2ddf-e1b6-4ee9-8ed3-4706f4fd5260-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-xpcxx\" (UID: \"233d2ddf-e1b6-4ee9-8ed3-4706f4fd5260\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-xpcxx" Jul 29 18:35:33.815483 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:33.815162 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/5603d54b-3265-4734-8526-72296a9c600e-secret-metrics-server-client-certs\") pod \"metrics-server-784b5c7fdf-9xt45\" (UID: \"5603d54b-3265-4734-8526-72296a9c600e\") " pod="openshift-monitoring/metrics-server-784b5c7fdf-9xt45" Jul 29 18:35:33.815483 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:33.815215 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/5603d54b-3265-4734-8526-72296a9c600e-audit-log\") pod \"metrics-server-784b5c7fdf-9xt45\" (UID: \"5603d54b-3265-4734-8526-72296a9c600e\") " pod="openshift-monitoring/metrics-server-784b5c7fdf-9xt45" Jul 29 18:35:33.816964 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:33.815828 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/5603d54b-3265-4734-8526-72296a9c600e-audit-log\") pod \"metrics-server-784b5c7fdf-9xt45\" (UID: \"5603d54b-3265-4734-8526-72296a9c600e\") " pod="openshift-monitoring/metrics-server-784b5c7fdf-9xt45" Jul 29 18:35:33.816964 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:33.816679 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/5603d54b-3265-4734-8526-72296a9c600e-metrics-server-audit-profiles\") pod \"metrics-server-784b5c7fdf-9xt45\" (UID: \"5603d54b-3265-4734-8526-72296a9c600e\") " pod="openshift-monitoring/metrics-server-784b5c7fdf-9xt45" Jul 29 18:35:33.820196 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:33.819784 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-76dc89c749-sn6bb"] Jul 29 18:35:33.820582 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:33.820532 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/5603d54b-3265-4734-8526-72296a9c600e-secret-metrics-server-client-certs\") pod \"metrics-server-784b5c7fdf-9xt45\" (UID: \"5603d54b-3265-4734-8526-72296a9c600e\") " pod="openshift-monitoring/metrics-server-784b5c7fdf-9xt45" Jul 29 18:35:33.821017 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:33.820992 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/5603d54b-3265-4734-8526-72296a9c600e-client-ca-bundle\") pod \"metrics-server-784b5c7fdf-9xt45\" (UID: \"5603d54b-3265-4734-8526-72296a9c600e\") " pod="openshift-monitoring/metrics-server-784b5c7fdf-9xt45" Jul 29 18:35:33.821092 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:33.820995 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/5603d54b-3265-4734-8526-72296a9c600e-secret-metrics-server-tls\") pod \"metrics-server-784b5c7fdf-9xt45\" (UID: \"5603d54b-3265-4734-8526-72296a9c600e\") " pod="openshift-monitoring/metrics-server-784b5c7fdf-9xt45" Jul 29 18:35:33.822539 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:35:33.822156 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3eb9d5f4_642e_4158_83d6_0d4bc40efd2f.slice/crio-f0cdc96fe0a105bdb875b73904df90886f001420a698e60b26c3a0ed999ff8a1 WatchSource:0}: Error finding container f0cdc96fe0a105bdb875b73904df90886f001420a698e60b26c3a0ed999ff8a1: Status 404 returned error can't find the container with id f0cdc96fe0a105bdb875b73904df90886f001420a698e60b26c3a0ed999ff8a1 Jul 29 18:35:33.824639 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:33.824621 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v67zv\" (UniqueName: \"kubernetes.io/projected/5603d54b-3265-4734-8526-72296a9c600e-kube-api-access-v67zv\") pod \"metrics-server-784b5c7fdf-9xt45\" (UID: \"5603d54b-3265-4734-8526-72296a9c600e\") " pod="openshift-monitoring/metrics-server-784b5c7fdf-9xt45" Jul 29 18:35:33.893514 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:33.893481 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-784b5c7fdf-9xt45" Jul 29 18:35:33.894751 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:33.894719 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:35:33.916557 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:33.916530 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/233d2ddf-e1b6-4ee9-8ed3-4706f4fd5260-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-xpcxx\" (UID: \"233d2ddf-e1b6-4ee9-8ed3-4706f4fd5260\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-xpcxx" Jul 29 18:35:33.916678 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:33.916597 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzd8q\" (UniqueName: \"kubernetes.io/projected/233d2ddf-e1b6-4ee9-8ed3-4706f4fd5260-kube-api-access-lzd8q\") pod \"monitoring-plugin-68b7fc9574-xpcxx\" (UID: \"233d2ddf-e1b6-4ee9-8ed3-4706f4fd5260\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-xpcxx" Jul 29 18:35:33.919300 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:33.919280 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/233d2ddf-e1b6-4ee9-8ed3-4706f4fd5260-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-xpcxx\" (UID: \"233d2ddf-e1b6-4ee9-8ed3-4706f4fd5260\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-xpcxx" Jul 29 18:35:33.926584 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:33.926558 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzd8q\" (UniqueName: \"kubernetes.io/projected/233d2ddf-e1b6-4ee9-8ed3-4706f4fd5260-kube-api-access-lzd8q\") pod \"monitoring-plugin-68b7fc9574-xpcxx\" (UID: \"233d2ddf-e1b6-4ee9-8ed3-4706f4fd5260\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-xpcxx" Jul 29 18:35:34.015713 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:34.015673 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-xpcxx" Jul 29 18:35:34.050880 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:34.050852 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-784b5c7fdf-9xt45"] Jul 29 18:35:34.053499 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:35:34.053476 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5603d54b_3265_4734_8526_72296a9c600e.slice/crio-d1097c4122f357e8e286b463919c55bdd00179473bbaf87ba88712b8f12eaca7 WatchSource:0}: Error finding container d1097c4122f357e8e286b463919c55bdd00179473bbaf87ba88712b8f12eaca7: Status 404 returned error can't find the container with id d1097c4122f357e8e286b463919c55bdd00179473bbaf87ba88712b8f12eaca7 Jul 29 18:35:34.168664 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:34.168629 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:35:34.179013 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:34.178978 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-68b7fc9574-xpcxx"] Jul 29 18:35:34.235596 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:34.235545 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:35:34.436199 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:35:34.436105 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod233d2ddf_e1b6_4ee9_8ed3_4706f4fd5260.slice/crio-e3599cf9f8097dc7087d523b328c2301d58974d7c1fb83bd95c390201c06610b WatchSource:0}: Error finding container e3599cf9f8097dc7087d523b328c2301d58974d7c1fb83bd95c390201c06610b: Status 404 returned error can't find the container with id e3599cf9f8097dc7087d523b328c2301d58974d7c1fb83bd95c390201c06610b Jul 29 18:35:34.680175 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:34.680148 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-gls76" Jul 29 18:35:34.745273 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:34.745208 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-xpcxx" event={"ID":"233d2ddf-e1b6-4ee9-8ed3-4706f4fd5260","Type":"ContainerStarted","Data":"e3599cf9f8097dc7087d523b328c2301d58974d7c1fb83bd95c390201c06610b"} Jul 29 18:35:34.749203 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:34.749175 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-nr2lw" event={"ID":"36f140ba-4691-4e5e-b58f-9305091a175e","Type":"ContainerStarted","Data":"98adb06454e2f532994306450f94cc4a55f32ccb984e9ddc0a056058bcfae66f"} Jul 29 18:35:34.751488 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:34.751461 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-76dc89c749-sn6bb" event={"ID":"3eb9d5f4-642e-4158-83d6-0d4bc40efd2f","Type":"ContainerStarted","Data":"f0cdc96fe0a105bdb875b73904df90886f001420a698e60b26c3a0ed999ff8a1"} Jul 29 18:35:34.754411 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:34.754307 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-tb6xb" event={"ID":"677a8b8b-b142-4cba-b81a-fefa36ee1f69","Type":"ContainerStarted","Data":"329a834ea59564758e24b359a6a0c819a3fd702382c630df463fdbdf4ed9967c"} Jul 29 18:35:34.754411 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:34.754372 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-tb6xb" event={"ID":"677a8b8b-b142-4cba-b81a-fefa36ee1f69","Type":"ContainerStarted","Data":"7ae76cf46e8c87b53d16a21c32ba1b343ebe886ad4fcc23ce4f87cf16ef71c36"} Jul 29 18:35:34.758953 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:34.758913 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-784b5c7fdf-9xt45" event={"ID":"5603d54b-3265-4734-8526-72296a9c600e","Type":"ContainerStarted","Data":"d1097c4122f357e8e286b463919c55bdd00179473bbaf87ba88712b8f12eaca7"} Jul 29 18:35:34.772612 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:34.772568 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-nr2lw" podStartSLOduration=1.880688912 podStartE2EDuration="16.772552182s" totalImagesPullingTime="14.89186327s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 18:35:18 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 18:35:34.771992031 +0000 UTC m=+69.997660923" watchObservedRunningTime="2026-07-29 18:35:34.772552182 +0000 UTC m=+69.998221056" Jul 29 18:35:34.828766 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:34.828684 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:35:34.934497 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:34.934165 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:35:35.246045 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:35.245193 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:35:35.405717 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:35.405637 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:35:35.505843 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:35.505725 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:35:35.771565 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:35.771408 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-mrshg" event={"ID":"70307ff7-a0ca-4a2b-95be-94a5dde5d96e","Type":"ContainerStarted","Data":"ef485d87c80e88984011dfba434f4e1a520154421c35d03a6a0a9cbdb120f239"} Jul 29 18:35:35.771565 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:35.771508 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-mrshg" event={"ID":"70307ff7-a0ca-4a2b-95be-94a5dde5d96e","Type":"ContainerStarted","Data":"7ec5d029eb3852205f57467c062c9a70339071ffb451104d3dc869d4dbae3127"} Jul 29 18:35:35.792161 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:35.792097 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-tb6xb" podStartSLOduration=5.157608839 podStartE2EDuration="6.792077364s" totalImagesPullingTime="1.539388269s" totalInitContainerRuntime="95.080256ms" isStatefulPod=true podCreationTimestamp="2026-07-29 18:35:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 18:35:34.79261083 +0000 UTC m=+70.018279702" watchObservedRunningTime="2026-07-29 18:35:35.792077364 +0000 UTC m=+71.017746238" Jul 29 18:35:35.792412 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:35.792380 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-mrshg" podStartSLOduration=67.806176823 podStartE2EDuration="1m10.79236159s" totalImagesPullingTime="2.986184767s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 18:34:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 18:35:35.790248062 +0000 UTC m=+71.015916934" watchObservedRunningTime="2026-07-29 18:35:35.79236159 +0000 UTC m=+71.018030476" Jul 29 18:35:35.880066 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:35.879980 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:35:35.947452 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:35.947414 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:35:36.056245 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:36.056139 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:35:36.181230 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:36.181195 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:35:36.280298 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:36.280082 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:35:36.379140 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:36.379053 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:35:40.795669 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:40.795635 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-8qdhk" event={"ID":"88336825-e84e-4595-9f68-f4d6b4e6edda","Type":"ContainerStarted","Data":"0627347518f2bc1dac1b511ad88f24a1c7a8c028fe05b48cab91530c4d924c99"} Jul 29 18:35:40.796077 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:40.795766 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-8qdhk" Jul 29 18:35:40.797549 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:40.797522 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-xpcxx" event={"ID":"233d2ddf-e1b6-4ee9-8ed3-4706f4fd5260","Type":"ContainerStarted","Data":"e99cdc7545a83de8c1068e4915b8d7739ca8342535285f7ccc6a0bc5a7f0f064"} Jul 29 18:35:40.797741 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:40.797722 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-xpcxx" Jul 29 18:35:40.799153 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:40.799128 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-zbwwp" event={"ID":"13fd7656-9a1f-4221-a5ed-6ebecf3664c9","Type":"ContainerStarted","Data":"4c72e8d91733f9c4462c4f8d8b4ff57c094c3fcc790af73bd76d504848d5e931"} Jul 29 18:35:40.801550 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:40.801529 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-76dc89c749-sn6bb" event={"ID":"3eb9d5f4-642e-4158-83d6-0d4bc40efd2f","Type":"ContainerStarted","Data":"d161a73be35b3c8f411d7db944d193765b7eebc94c5592496777bba565f89638"} Jul 29 18:35:40.801647 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:40.801597 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-76dc89c749-sn6bb" event={"ID":"3eb9d5f4-642e-4158-83d6-0d4bc40efd2f","Type":"ContainerStarted","Data":"feb9facfcc2c7d394278452a12eb8aa998fe52c44b4d43ff679083f9505ad3d7"} Jul 29 18:35:40.801701 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:40.801646 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-76dc89c749-sn6bb" event={"ID":"3eb9d5f4-642e-4158-83d6-0d4bc40efd2f","Type":"ContainerStarted","Data":"bb5b6c186692b0cc38010677b1c932ecca1c6aa1f55ff679842c1dfe97999bc1"} Jul 29 18:35:40.803092 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:40.803067 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-784b5c7fdf-9xt45" event={"ID":"5603d54b-3265-4734-8526-72296a9c600e","Type":"ContainerStarted","Data":"4abc3a32c80b186da7624e52f9f1b6e08e166cc241eaac77fa743062c712b9d4"} Jul 29 18:35:40.803928 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:40.803909 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-xpcxx" Jul 29 18:35:40.813753 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:40.813709 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-8qdhk" podStartSLOduration=69.56813021 podStartE2EDuration="1m15.813694974s" totalImagesPullingTime="6.245564764s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 18:34:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 18:35:40.812962339 +0000 UTC m=+76.038631210" watchObservedRunningTime="2026-07-29 18:35:40.813694974 +0000 UTC m=+76.039363847" Jul 29 18:35:40.831039 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:40.830990 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-zbwwp" podStartSLOduration=69.568791587 podStartE2EDuration="1m15.830973472s" totalImagesPullingTime="6.262181885s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 18:34:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 18:35:40.829545902 +0000 UTC m=+76.055214772" watchObservedRunningTime="2026-07-29 18:35:40.830973472 +0000 UTC m=+76.056642347" Jul 29 18:35:40.848325 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:40.848290 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-xpcxx" podStartSLOduration=2.405899635 podStartE2EDuration="7.848278799s" totalImagesPullingTime="5.442379164s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 18:35:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 18:35:40.847754805 +0000 UTC m=+76.073423680" watchObservedRunningTime="2026-07-29 18:35:40.848278799 +0000 UTC m=+76.073947703" Jul 29 18:35:40.867628 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:40.867582 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-784b5c7fdf-9xt45" podStartSLOduration=2.0449222479999998 podStartE2EDuration="7.867563586s" totalImagesPullingTime="5.822641338s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 18:35:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 18:35:40.866981264 +0000 UTC m=+76.092650137" watchObservedRunningTime="2026-07-29 18:35:40.867563586 +0000 UTC m=+76.093232459" Jul 29 18:35:40.954323 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:40.954300 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:35:41.059924 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:41.059827 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:35:41.810042 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:41.809999 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-76dc89c749-sn6bb" event={"ID":"3eb9d5f4-642e-4158-83d6-0d4bc40efd2f","Type":"ContainerStarted","Data":"daf957c2a5f5abd4014f2c5abc3e0922d33bc11757104aee83e83c6304a3e1d5"} Jul 29 18:35:41.810508 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:41.810080 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-76dc89c749-sn6bb" event={"ID":"3eb9d5f4-642e-4158-83d6-0d4bc40efd2f","Type":"ContainerStarted","Data":"7d13e2e89e027ff8e804f67e81b187da8cc2421079b1091af6f50068b664fb91"} Jul 29 18:35:41.810508 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:41.810119 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-76dc89c749-sn6bb" event={"ID":"3eb9d5f4-642e-4158-83d6-0d4bc40efd2f","Type":"ContainerStarted","Data":"9ea1f7ce4f60a624c5d7bd311163403750c9297cb9c93a4909a4bfdaee34ec0d"} Jul 29 18:35:41.837730 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:41.837669 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-76dc89c749-sn6bb" podStartSLOduration=4.118121087 podStartE2EDuration="10.83765042s" totalImagesPullingTime="6.719529333s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 18:35:31 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 18:35:41.836406973 +0000 UTC m=+77.062075845" watchObservedRunningTime="2026-07-29 18:35:41.83765042 +0000 UTC m=+77.063319295" Jul 29 18:35:41.978194 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:41.978146 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:35:42.079457 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:42.079384 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:35:42.813695 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:42.813668 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-76dc89c749-sn6bb" Jul 29 18:35:44.542756 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:44.542721 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:35:44.600301 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:44.600275 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:35:45.223667 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:45.223483 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-765c45df68-wbklv"] Jul 29 18:35:45.332455 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:45.332413 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:35:45.644228 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:45.644194 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:35:45.745227 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:45.745195 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:35:48.823368 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:48.823319 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-76dc89c749-sn6bb" Jul 29 18:35:49.995005 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:49.994969 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:35:53.894640 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:53.894603 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-784b5c7fdf-9xt45" Jul 29 18:35:53.895110 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:53.894654 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-784b5c7fdf-9xt45" Jul 29 18:35:56.993449 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:56.993415 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:35:57.057720 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:35:57.057682 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:36:10.243657 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:36:10.243594 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-765c45df68-wbklv" podUID="dcaf4f9c-b5cb-4f1d-83ed-72639860bee6" containerName="console" containerID="cri-o://254a77d41a419fe58d5929eb9eadf850e52d22bb5f6f62f3eca560d835f9b1a2" gracePeriod=15 Jul 29 18:36:10.924693 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:36:10.924560 2560 generic.go:356] "Generic (PLEG): container finished" podID="dcaf4f9c-b5cb-4f1d-83ed-72639860bee6" containerID="254a77d41a419fe58d5929eb9eadf850e52d22bb5f6f62f3eca560d835f9b1a2" exitCode=0 Jul 29 18:36:10.924693 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:36:10.924622 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-765c45df68-wbklv" event={"ID":"dcaf4f9c-b5cb-4f1d-83ed-72639860bee6","Type":"ContainerDied","Data":"254a77d41a419fe58d5929eb9eadf850e52d22bb5f6f62f3eca560d835f9b1a2"} Jul 29 18:36:11.033095 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:36:11.033068 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-765c45df68-wbklv" Jul 29 18:36:11.194404 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:36:11.194313 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dcaf4f9c-b5cb-4f1d-83ed-72639860bee6-trusted-ca-bundle\") pod \"dcaf4f9c-b5cb-4f1d-83ed-72639860bee6\" (UID: \"dcaf4f9c-b5cb-4f1d-83ed-72639860bee6\") " Jul 29 18:36:11.194404 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:36:11.194375 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/dcaf4f9c-b5cb-4f1d-83ed-72639860bee6-oauth-serving-cert\") pod \"dcaf4f9c-b5cb-4f1d-83ed-72639860bee6\" (UID: \"dcaf4f9c-b5cb-4f1d-83ed-72639860bee6\") " Jul 29 18:36:11.194404 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:36:11.194396 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/dcaf4f9c-b5cb-4f1d-83ed-72639860bee6-console-oauth-config\") pod \"dcaf4f9c-b5cb-4f1d-83ed-72639860bee6\" (UID: \"dcaf4f9c-b5cb-4f1d-83ed-72639860bee6\") " Jul 29 18:36:11.194625 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:36:11.194564 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/dcaf4f9c-b5cb-4f1d-83ed-72639860bee6-console-serving-cert\") pod \"dcaf4f9c-b5cb-4f1d-83ed-72639860bee6\" (UID: \"dcaf4f9c-b5cb-4f1d-83ed-72639860bee6\") " Jul 29 18:36:11.194658 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:36:11.194619 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/dcaf4f9c-b5cb-4f1d-83ed-72639860bee6-service-ca\") pod \"dcaf4f9c-b5cb-4f1d-83ed-72639860bee6\" (UID: \"dcaf4f9c-b5cb-4f1d-83ed-72639860bee6\") " Jul 29 18:36:11.194720 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:36:11.194707 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f6gfr\" (UniqueName: \"kubernetes.io/projected/dcaf4f9c-b5cb-4f1d-83ed-72639860bee6-kube-api-access-f6gfr\") pod \"dcaf4f9c-b5cb-4f1d-83ed-72639860bee6\" (UID: \"dcaf4f9c-b5cb-4f1d-83ed-72639860bee6\") " Jul 29 18:36:11.194776 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:36:11.194707 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dcaf4f9c-b5cb-4f1d-83ed-72639860bee6-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "dcaf4f9c-b5cb-4f1d-83ed-72639860bee6" (UID: "dcaf4f9c-b5cb-4f1d-83ed-72639860bee6"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 18:36:11.194776 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:36:11.194733 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dcaf4f9c-b5cb-4f1d-83ed-72639860bee6-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "dcaf4f9c-b5cb-4f1d-83ed-72639860bee6" (UID: "dcaf4f9c-b5cb-4f1d-83ed-72639860bee6"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 18:36:11.194776 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:36:11.194746 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/dcaf4f9c-b5cb-4f1d-83ed-72639860bee6-console-config\") pod \"dcaf4f9c-b5cb-4f1d-83ed-72639860bee6\" (UID: \"dcaf4f9c-b5cb-4f1d-83ed-72639860bee6\") " Jul 29 18:36:11.194923 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:36:11.194896 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dcaf4f9c-b5cb-4f1d-83ed-72639860bee6-service-ca" (OuterVolumeSpecName: "service-ca") pod "dcaf4f9c-b5cb-4f1d-83ed-72639860bee6" (UID: "dcaf4f9c-b5cb-4f1d-83ed-72639860bee6"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 18:36:11.195027 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:36:11.194994 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dcaf4f9c-b5cb-4f1d-83ed-72639860bee6-console-config" (OuterVolumeSpecName: "console-config") pod "dcaf4f9c-b5cb-4f1d-83ed-72639860bee6" (UID: "dcaf4f9c-b5cb-4f1d-83ed-72639860bee6"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 18:36:11.195027 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:36:11.195012 2560 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dcaf4f9c-b5cb-4f1d-83ed-72639860bee6-trusted-ca-bundle\") on node \"ip-10-0-140-205.ec2.internal\" DevicePath \"\"" Jul 29 18:36:11.195027 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:36:11.195030 2560 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/dcaf4f9c-b5cb-4f1d-83ed-72639860bee6-oauth-serving-cert\") on node \"ip-10-0-140-205.ec2.internal\" DevicePath \"\"" Jul 29 18:36:11.195220 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:36:11.195044 2560 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/dcaf4f9c-b5cb-4f1d-83ed-72639860bee6-service-ca\") on node \"ip-10-0-140-205.ec2.internal\" DevicePath \"\"" Jul 29 18:36:11.197299 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:36:11.197273 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dcaf4f9c-b5cb-4f1d-83ed-72639860bee6-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "dcaf4f9c-b5cb-4f1d-83ed-72639860bee6" (UID: "dcaf4f9c-b5cb-4f1d-83ed-72639860bee6"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 18:36:11.197402 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:36:11.197335 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dcaf4f9c-b5cb-4f1d-83ed-72639860bee6-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "dcaf4f9c-b5cb-4f1d-83ed-72639860bee6" (UID: "dcaf4f9c-b5cb-4f1d-83ed-72639860bee6"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 18:36:11.197644 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:36:11.197621 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dcaf4f9c-b5cb-4f1d-83ed-72639860bee6-kube-api-access-f6gfr" (OuterVolumeSpecName: "kube-api-access-f6gfr") pod "dcaf4f9c-b5cb-4f1d-83ed-72639860bee6" (UID: "dcaf4f9c-b5cb-4f1d-83ed-72639860bee6"). InnerVolumeSpecName "kube-api-access-f6gfr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 18:36:11.296327 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:36:11.296295 2560 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/dcaf4f9c-b5cb-4f1d-83ed-72639860bee6-console-oauth-config\") on node \"ip-10-0-140-205.ec2.internal\" DevicePath \"\"" Jul 29 18:36:11.296327 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:36:11.296323 2560 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/dcaf4f9c-b5cb-4f1d-83ed-72639860bee6-console-serving-cert\") on node \"ip-10-0-140-205.ec2.internal\" DevicePath \"\"" Jul 29 18:36:11.296327 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:36:11.296335 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f6gfr\" (UniqueName: \"kubernetes.io/projected/dcaf4f9c-b5cb-4f1d-83ed-72639860bee6-kube-api-access-f6gfr\") on node \"ip-10-0-140-205.ec2.internal\" DevicePath \"\"" Jul 29 18:36:11.296885 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:36:11.296358 2560 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/dcaf4f9c-b5cb-4f1d-83ed-72639860bee6-console-config\") on node \"ip-10-0-140-205.ec2.internal\" DevicePath \"\"" Jul 29 18:36:11.813361 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:36:11.813315 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-8qdhk" Jul 29 18:36:11.932615 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:36:11.932503 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-765c45df68-wbklv" Jul 29 18:36:11.932615 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:36:11.932502 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-765c45df68-wbklv" event={"ID":"dcaf4f9c-b5cb-4f1d-83ed-72639860bee6","Type":"ContainerDied","Data":"a6d5b18e207a379466085b78728311e51b8bd7a32cab4ce82dd7eb4b4098f809"} Jul 29 18:36:11.932615 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:36:11.932650 2560 scope.go:117] "RemoveContainer" containerID="254a77d41a419fe58d5929eb9eadf850e52d22bb5f6f62f3eca560d835f9b1a2" Jul 29 18:36:11.948717 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:36:11.948687 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-765c45df68-wbklv"] Jul 29 18:36:11.952936 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:36:11.952911 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-765c45df68-wbklv"] Jul 29 18:36:13.352441 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:36:13.352412 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dcaf4f9c-b5cb-4f1d-83ed-72639860bee6" path="/var/lib/kubelet/pods/dcaf4f9c-b5cb-4f1d-83ed-72639860bee6/volumes" Jul 29 18:36:13.899418 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:36:13.899381 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-784b5c7fdf-9xt45" Jul 29 18:36:13.903243 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:36:13.903223 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-784b5c7fdf-9xt45" Jul 29 18:36:48.176101 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:36:48.176066 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:36:48.478644 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:36:48.478558 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:36:51.486700 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:36:51.486664 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 29 18:36:51.487088 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:36:51.486970 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dcaf4f9c-b5cb-4f1d-83ed-72639860bee6" containerName="console" Jul 29 18:36:51.487088 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:36:51.486983 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="dcaf4f9c-b5cb-4f1d-83ed-72639860bee6" containerName="console" Jul 29 18:36:51.487088 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:36:51.487036 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="dcaf4f9c-b5cb-4f1d-83ed-72639860bee6" containerName="console" Jul 29 18:36:51.489413 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:36:51.489395 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 29 18:36:51.492306 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:36:51.492274 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 29 18:36:51.492514 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:36:51.492491 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 29 18:36:51.492630 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:36:51.492589 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 29 18:36:51.492630 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:36:51.492604 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 29 18:36:51.492630 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:36:51.492520 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 29 18:36:51.492774 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:36:51.492607 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 29 18:36:51.492774 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:36:51.492568 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 29 18:36:51.492774 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:36:51.492565 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 29 18:36:51.492941 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:36:51.492926 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-2c4tl\"" Jul 29 18:36:51.498235 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:36:51.498218 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 29 18:36:51.505816 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:36:51.505794 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 29 18:36:51.571845 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:36:51.571820 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e38c79a4-a26e-4ac4-9e5b-e7060aca254e-tls-assets\") pod \"alertmanager-main-0\" (UID: \"e38c79a4-a26e-4ac4-9e5b-e7060aca254e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 18:36:51.571977 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:36:51.571848 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/e38c79a4-a26e-4ac4-9e5b-e7060aca254e-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"e38c79a4-a26e-4ac4-9e5b-e7060aca254e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 18:36:51.571977 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:36:51.571866 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e38c79a4-a26e-4ac4-9e5b-e7060aca254e-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"e38c79a4-a26e-4ac4-9e5b-e7060aca254e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 18:36:51.571977 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:36:51.571880 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e38c79a4-a26e-4ac4-9e5b-e7060aca254e-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"e38c79a4-a26e-4ac4-9e5b-e7060aca254e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 18:36:51.571977 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:36:51.571943 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e38c79a4-a26e-4ac4-9e5b-e7060aca254e-web-config\") pod \"alertmanager-main-0\" (UID: \"e38c79a4-a26e-4ac4-9e5b-e7060aca254e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 18:36:51.572191 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:36:51.571992 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/e38c79a4-a26e-4ac4-9e5b-e7060aca254e-config-volume\") pod \"alertmanager-main-0\" (UID: \"e38c79a4-a26e-4ac4-9e5b-e7060aca254e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 18:36:51.572191 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:36:51.572029 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e38c79a4-a26e-4ac4-9e5b-e7060aca254e-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"e38c79a4-a26e-4ac4-9e5b-e7060aca254e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 18:36:51.572191 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:36:51.572056 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brjh8\" (UniqueName: \"kubernetes.io/projected/e38c79a4-a26e-4ac4-9e5b-e7060aca254e-kube-api-access-brjh8\") pod \"alertmanager-main-0\" (UID: \"e38c79a4-a26e-4ac4-9e5b-e7060aca254e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 18:36:51.572191 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:36:51.572099 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/e38c79a4-a26e-4ac4-9e5b-e7060aca254e-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"e38c79a4-a26e-4ac4-9e5b-e7060aca254e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 18:36:51.572191 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:36:51.572138 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/e38c79a4-a26e-4ac4-9e5b-e7060aca254e-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"e38c79a4-a26e-4ac4-9e5b-e7060aca254e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 18:36:51.572191 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:36:51.572178 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/e38c79a4-a26e-4ac4-9e5b-e7060aca254e-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"e38c79a4-a26e-4ac4-9e5b-e7060aca254e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 18:36:51.572455 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:36:51.572201 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e38c79a4-a26e-4ac4-9e5b-e7060aca254e-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"e38c79a4-a26e-4ac4-9e5b-e7060aca254e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 18:36:51.572455 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:36:51.572246 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e38c79a4-a26e-4ac4-9e5b-e7060aca254e-config-out\") pod \"alertmanager-main-0\" (UID: \"e38c79a4-a26e-4ac4-9e5b-e7060aca254e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 18:36:51.673433 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:36:51.673394 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/e38c79a4-a26e-4ac4-9e5b-e7060aca254e-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"e38c79a4-a26e-4ac4-9e5b-e7060aca254e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 18:36:51.673632 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:36:51.673450 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e38c79a4-a26e-4ac4-9e5b-e7060aca254e-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"e38c79a4-a26e-4ac4-9e5b-e7060aca254e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 18:36:51.673692 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:36:51.673629 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e38c79a4-a26e-4ac4-9e5b-e7060aca254e-config-out\") pod \"alertmanager-main-0\" (UID: \"e38c79a4-a26e-4ac4-9e5b-e7060aca254e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 18:36:51.673692 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:36:51.673676 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e38c79a4-a26e-4ac4-9e5b-e7060aca254e-tls-assets\") pod \"alertmanager-main-0\" (UID: \"e38c79a4-a26e-4ac4-9e5b-e7060aca254e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 18:36:51.673800 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:36:51.673704 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/e38c79a4-a26e-4ac4-9e5b-e7060aca254e-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"e38c79a4-a26e-4ac4-9e5b-e7060aca254e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 18:36:51.673800 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:36:51.673732 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e38c79a4-a26e-4ac4-9e5b-e7060aca254e-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"e38c79a4-a26e-4ac4-9e5b-e7060aca254e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 18:36:51.673800 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:36:51.673766 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e38c79a4-a26e-4ac4-9e5b-e7060aca254e-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"e38c79a4-a26e-4ac4-9e5b-e7060aca254e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 18:36:51.673960 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:36:51.673801 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e38c79a4-a26e-4ac4-9e5b-e7060aca254e-web-config\") pod \"alertmanager-main-0\" (UID: \"e38c79a4-a26e-4ac4-9e5b-e7060aca254e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 18:36:51.673960 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:36:51.673826 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/e38c79a4-a26e-4ac4-9e5b-e7060aca254e-config-volume\") pod \"alertmanager-main-0\" (UID: \"e38c79a4-a26e-4ac4-9e5b-e7060aca254e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 18:36:51.673960 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:36:51.673862 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e38c79a4-a26e-4ac4-9e5b-e7060aca254e-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"e38c79a4-a26e-4ac4-9e5b-e7060aca254e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 18:36:51.673960 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:36:51.673889 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brjh8\" (UniqueName: \"kubernetes.io/projected/e38c79a4-a26e-4ac4-9e5b-e7060aca254e-kube-api-access-brjh8\") pod \"alertmanager-main-0\" (UID: \"e38c79a4-a26e-4ac4-9e5b-e7060aca254e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 18:36:51.673960 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:36:51.673925 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/e38c79a4-a26e-4ac4-9e5b-e7060aca254e-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"e38c79a4-a26e-4ac4-9e5b-e7060aca254e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 18:36:51.674390 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:36:51.673965 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/e38c79a4-a26e-4ac4-9e5b-e7060aca254e-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"e38c79a4-a26e-4ac4-9e5b-e7060aca254e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 18:36:51.676854 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:36:51.676783 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e38c79a4-a26e-4ac4-9e5b-e7060aca254e-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"e38c79a4-a26e-4ac4-9e5b-e7060aca254e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 18:36:51.676970 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:36:51.676934 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/e38c79a4-a26e-4ac4-9e5b-e7060aca254e-config-volume\") pod \"alertmanager-main-0\" (UID: \"e38c79a4-a26e-4ac4-9e5b-e7060aca254e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 18:36:51.676970 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:36:51.676960 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e38c79a4-a26e-4ac4-9e5b-e7060aca254e-tls-assets\") pod \"alertmanager-main-0\" (UID: \"e38c79a4-a26e-4ac4-9e5b-e7060aca254e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 18:36:51.677092 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:36:51.676935 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/e38c79a4-a26e-4ac4-9e5b-e7060aca254e-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"e38c79a4-a26e-4ac4-9e5b-e7060aca254e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 18:36:51.677092 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:36:51.677062 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e38c79a4-a26e-4ac4-9e5b-e7060aca254e-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"e38c79a4-a26e-4ac4-9e5b-e7060aca254e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 18:36:51.677212 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:36:51.677121 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/e38c79a4-a26e-4ac4-9e5b-e7060aca254e-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"e38c79a4-a26e-4ac4-9e5b-e7060aca254e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 18:36:51.677280 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:36:51.677260 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/e38c79a4-a26e-4ac4-9e5b-e7060aca254e-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"e38c79a4-a26e-4ac4-9e5b-e7060aca254e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 18:36:51.677332 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:36:51.677279 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e38c79a4-a26e-4ac4-9e5b-e7060aca254e-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"e38c79a4-a26e-4ac4-9e5b-e7060aca254e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 18:36:51.677413 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:36:51.677393 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e38c79a4-a26e-4ac4-9e5b-e7060aca254e-config-out\") pod \"alertmanager-main-0\" (UID: \"e38c79a4-a26e-4ac4-9e5b-e7060aca254e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 18:36:51.677584 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:36:51.677569 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/e38c79a4-a26e-4ac4-9e5b-e7060aca254e-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"e38c79a4-a26e-4ac4-9e5b-e7060aca254e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 18:36:51.678330 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:36:51.678310 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e38c79a4-a26e-4ac4-9e5b-e7060aca254e-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"e38c79a4-a26e-4ac4-9e5b-e7060aca254e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 18:36:51.678886 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:36:51.678866 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e38c79a4-a26e-4ac4-9e5b-e7060aca254e-web-config\") pod \"alertmanager-main-0\" (UID: \"e38c79a4-a26e-4ac4-9e5b-e7060aca254e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 18:36:51.683956 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:36:51.683934 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-brjh8\" (UniqueName: \"kubernetes.io/projected/e38c79a4-a26e-4ac4-9e5b-e7060aca254e-kube-api-access-brjh8\") pod \"alertmanager-main-0\" (UID: \"e38c79a4-a26e-4ac4-9e5b-e7060aca254e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 18:36:51.802210 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:36:51.802175 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 29 18:36:51.937048 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:36:51.937025 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 29 18:36:51.939421 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:36:51.939390 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode38c79a4_a26e_4ac4_9e5b_e7060aca254e.slice/crio-31128884b886fec247736c7c8d44b8746d9d188b311ffeaf2dab15594e9bb0d5 WatchSource:0}: Error finding container 31128884b886fec247736c7c8d44b8746d9d188b311ffeaf2dab15594e9bb0d5: Status 404 returned error can't find the container with id 31128884b886fec247736c7c8d44b8746d9d188b311ffeaf2dab15594e9bb0d5 Jul 29 18:36:52.075749 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:36:52.075665 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e38c79a4-a26e-4ac4-9e5b-e7060aca254e","Type":"ContainerStarted","Data":"31128884b886fec247736c7c8d44b8746d9d188b311ffeaf2dab15594e9bb0d5"} Jul 29 18:36:53.247131 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:36:53.247096 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-6c8f44d988-8hrdm"] Jul 29 18:36:53.249604 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:36:53.249579 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-6c8f44d988-8hrdm" Jul 29 18:36:53.255423 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:36:53.255399 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-8hzf7\"" Jul 29 18:36:53.255546 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:36:53.255482 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Jul 29 18:36:53.264199 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:36:53.264177 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Jul 29 18:36:53.264396 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:36:53.264376 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Jul 29 18:36:53.264740 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:36:53.264715 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Jul 29 18:36:53.264843 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:36:53.264749 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Jul 29 18:36:53.272701 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:36:53.272680 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Jul 29 18:36:53.281411 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:36:53.281391 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-6c8f44d988-8hrdm"] Jul 29 18:36:53.284324 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:36:53.284127 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cz5qn\" (UniqueName: \"kubernetes.io/projected/bc69e3d8-7af1-4366-bfe1-531aecbcf766-kube-api-access-cz5qn\") pod \"telemeter-client-6c8f44d988-8hrdm\" (UID: \"bc69e3d8-7af1-4366-bfe1-531aecbcf766\") " pod="openshift-monitoring/telemeter-client-6c8f44d988-8hrdm" Jul 29 18:36:53.284324 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:36:53.284172 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/bc69e3d8-7af1-4366-bfe1-531aecbcf766-telemeter-client-tls\") pod \"telemeter-client-6c8f44d988-8hrdm\" (UID: \"bc69e3d8-7af1-4366-bfe1-531aecbcf766\") " pod="openshift-monitoring/telemeter-client-6c8f44d988-8hrdm" Jul 29 18:36:53.284324 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:36:53.284203 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bc69e3d8-7af1-4366-bfe1-531aecbcf766-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6c8f44d988-8hrdm\" (UID: \"bc69e3d8-7af1-4366-bfe1-531aecbcf766\") " pod="openshift-monitoring/telemeter-client-6c8f44d988-8hrdm" Jul 29 18:36:53.284324 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:36:53.284231 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/bc69e3d8-7af1-4366-bfe1-531aecbcf766-federate-client-tls\") pod \"telemeter-client-6c8f44d988-8hrdm\" (UID: \"bc69e3d8-7af1-4366-bfe1-531aecbcf766\") " pod="openshift-monitoring/telemeter-client-6c8f44d988-8hrdm" Jul 29 18:36:53.284324 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:36:53.284272 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/bc69e3d8-7af1-4366-bfe1-531aecbcf766-secret-telemeter-client\") pod \"telemeter-client-6c8f44d988-8hrdm\" (UID: \"bc69e3d8-7af1-4366-bfe1-531aecbcf766\") " pod="openshift-monitoring/telemeter-client-6c8f44d988-8hrdm" Jul 29 18:36:53.284635 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:36:53.284325 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bc69e3d8-7af1-4366-bfe1-531aecbcf766-serving-certs-ca-bundle\") pod \"telemeter-client-6c8f44d988-8hrdm\" (UID: \"bc69e3d8-7af1-4366-bfe1-531aecbcf766\") " pod="openshift-monitoring/telemeter-client-6c8f44d988-8hrdm" Jul 29 18:36:53.284635 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:36:53.284432 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bc69e3d8-7af1-4366-bfe1-531aecbcf766-metrics-client-ca\") pod \"telemeter-client-6c8f44d988-8hrdm\" (UID: \"bc69e3d8-7af1-4366-bfe1-531aecbcf766\") " pod="openshift-monitoring/telemeter-client-6c8f44d988-8hrdm" Jul 29 18:36:53.284635 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:36:53.284486 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bc69e3d8-7af1-4366-bfe1-531aecbcf766-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6c8f44d988-8hrdm\" (UID: \"bc69e3d8-7af1-4366-bfe1-531aecbcf766\") " pod="openshift-monitoring/telemeter-client-6c8f44d988-8hrdm" Jul 29 18:36:53.386002 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:36:53.385957 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/bc69e3d8-7af1-4366-bfe1-531aecbcf766-secret-telemeter-client\") pod \"telemeter-client-6c8f44d988-8hrdm\" (UID: \"bc69e3d8-7af1-4366-bfe1-531aecbcf766\") " pod="openshift-monitoring/telemeter-client-6c8f44d988-8hrdm" Jul 29 18:36:53.386313 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:36:53.386018 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bc69e3d8-7af1-4366-bfe1-531aecbcf766-serving-certs-ca-bundle\") pod \"telemeter-client-6c8f44d988-8hrdm\" (UID: \"bc69e3d8-7af1-4366-bfe1-531aecbcf766\") " pod="openshift-monitoring/telemeter-client-6c8f44d988-8hrdm" Jul 29 18:36:53.386313 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:36:53.386090 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bc69e3d8-7af1-4366-bfe1-531aecbcf766-metrics-client-ca\") pod \"telemeter-client-6c8f44d988-8hrdm\" (UID: \"bc69e3d8-7af1-4366-bfe1-531aecbcf766\") " pod="openshift-monitoring/telemeter-client-6c8f44d988-8hrdm" Jul 29 18:36:53.386313 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:36:53.386177 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bc69e3d8-7af1-4366-bfe1-531aecbcf766-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6c8f44d988-8hrdm\" (UID: \"bc69e3d8-7af1-4366-bfe1-531aecbcf766\") " pod="openshift-monitoring/telemeter-client-6c8f44d988-8hrdm" Jul 29 18:36:53.386313 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:36:53.386212 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cz5qn\" (UniqueName: \"kubernetes.io/projected/bc69e3d8-7af1-4366-bfe1-531aecbcf766-kube-api-access-cz5qn\") pod \"telemeter-client-6c8f44d988-8hrdm\" (UID: \"bc69e3d8-7af1-4366-bfe1-531aecbcf766\") " pod="openshift-monitoring/telemeter-client-6c8f44d988-8hrdm" Jul 29 18:36:53.386313 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:36:53.386246 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/bc69e3d8-7af1-4366-bfe1-531aecbcf766-telemeter-client-tls\") pod \"telemeter-client-6c8f44d988-8hrdm\" (UID: \"bc69e3d8-7af1-4366-bfe1-531aecbcf766\") " pod="openshift-monitoring/telemeter-client-6c8f44d988-8hrdm" Jul 29 18:36:53.386313 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:36:53.386279 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bc69e3d8-7af1-4366-bfe1-531aecbcf766-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6c8f44d988-8hrdm\" (UID: \"bc69e3d8-7af1-4366-bfe1-531aecbcf766\") " pod="openshift-monitoring/telemeter-client-6c8f44d988-8hrdm" Jul 29 18:36:53.386313 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:36:53.386307 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/bc69e3d8-7af1-4366-bfe1-531aecbcf766-federate-client-tls\") pod \"telemeter-client-6c8f44d988-8hrdm\" (UID: \"bc69e3d8-7af1-4366-bfe1-531aecbcf766\") " pod="openshift-monitoring/telemeter-client-6c8f44d988-8hrdm" Jul 29 18:36:53.390778 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:36:53.390722 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/bc69e3d8-7af1-4366-bfe1-531aecbcf766-federate-client-tls\") pod \"telemeter-client-6c8f44d988-8hrdm\" (UID: \"bc69e3d8-7af1-4366-bfe1-531aecbcf766\") " pod="openshift-monitoring/telemeter-client-6c8f44d988-8hrdm" Jul 29 18:36:53.390778 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:36:53.390737 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bc69e3d8-7af1-4366-bfe1-531aecbcf766-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6c8f44d988-8hrdm\" (UID: \"bc69e3d8-7af1-4366-bfe1-531aecbcf766\") " pod="openshift-monitoring/telemeter-client-6c8f44d988-8hrdm" Jul 29 18:36:53.390969 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:36:53.390886 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/bc69e3d8-7af1-4366-bfe1-531aecbcf766-secret-telemeter-client\") pod \"telemeter-client-6c8f44d988-8hrdm\" (UID: \"bc69e3d8-7af1-4366-bfe1-531aecbcf766\") " pod="openshift-monitoring/telemeter-client-6c8f44d988-8hrdm" Jul 29 18:36:53.390969 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:36:53.390913 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bc69e3d8-7af1-4366-bfe1-531aecbcf766-serving-certs-ca-bundle\") pod \"telemeter-client-6c8f44d988-8hrdm\" (UID: \"bc69e3d8-7af1-4366-bfe1-531aecbcf766\") " pod="openshift-monitoring/telemeter-client-6c8f44d988-8hrdm" Jul 29 18:36:53.390969 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:36:53.390924 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bc69e3d8-7af1-4366-bfe1-531aecbcf766-metrics-client-ca\") pod \"telemeter-client-6c8f44d988-8hrdm\" (UID: \"bc69e3d8-7af1-4366-bfe1-531aecbcf766\") " pod="openshift-monitoring/telemeter-client-6c8f44d988-8hrdm" Jul 29 18:36:53.392139 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:36:53.392113 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/bc69e3d8-7af1-4366-bfe1-531aecbcf766-telemeter-client-tls\") pod \"telemeter-client-6c8f44d988-8hrdm\" (UID: \"bc69e3d8-7af1-4366-bfe1-531aecbcf766\") " pod="openshift-monitoring/telemeter-client-6c8f44d988-8hrdm" Jul 29 18:36:53.393080 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:36:53.393051 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bc69e3d8-7af1-4366-bfe1-531aecbcf766-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6c8f44d988-8hrdm\" (UID: \"bc69e3d8-7af1-4366-bfe1-531aecbcf766\") " pod="openshift-monitoring/telemeter-client-6c8f44d988-8hrdm" Jul 29 18:36:53.401687 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:36:53.401658 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cz5qn\" (UniqueName: \"kubernetes.io/projected/bc69e3d8-7af1-4366-bfe1-531aecbcf766-kube-api-access-cz5qn\") pod \"telemeter-client-6c8f44d988-8hrdm\" (UID: \"bc69e3d8-7af1-4366-bfe1-531aecbcf766\") " pod="openshift-monitoring/telemeter-client-6c8f44d988-8hrdm" Jul 29 18:36:53.562713 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:36:53.562690 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-6c8f44d988-8hrdm" Jul 29 18:36:53.722393 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:36:53.722326 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-6c8f44d988-8hrdm"] Jul 29 18:36:53.724498 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:36:53.724472 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc69e3d8_7af1_4366_bfe1_531aecbcf766.slice/crio-ab3ccd00bcd5bd88918f6b9079ecb743ff3f46b9e0351db7659db59d14495950 WatchSource:0}: Error finding container ab3ccd00bcd5bd88918f6b9079ecb743ff3f46b9e0351db7659db59d14495950: Status 404 returned error can't find the container with id ab3ccd00bcd5bd88918f6b9079ecb743ff3f46b9e0351db7659db59d14495950 Jul 29 18:36:54.083811 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:36:54.083771 2560 generic.go:356] "Generic (PLEG): container finished" podID="e38c79a4-a26e-4ac4-9e5b-e7060aca254e" containerID="3763a8a547efb77055e8b1c3c9521186c5a3034bf50f0858903605b07309c5a2" exitCode=0 Jul 29 18:36:54.083991 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:36:54.083867 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e38c79a4-a26e-4ac4-9e5b-e7060aca254e","Type":"ContainerDied","Data":"3763a8a547efb77055e8b1c3c9521186c5a3034bf50f0858903605b07309c5a2"} Jul 29 18:36:54.085065 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:36:54.085036 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6c8f44d988-8hrdm" event={"ID":"bc69e3d8-7af1-4366-bfe1-531aecbcf766","Type":"ContainerStarted","Data":"ab3ccd00bcd5bd88918f6b9079ecb743ff3f46b9e0351db7659db59d14495950"} Jul 29 18:36:57.101448 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:36:57.101413 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e38c79a4-a26e-4ac4-9e5b-e7060aca254e","Type":"ContainerStarted","Data":"59ea21b828569bd5c486582a4c19da9e183765764e4fbbf172094b8058091e97"} Jul 29 18:36:57.101880 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:36:57.101488 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e38c79a4-a26e-4ac4-9e5b-e7060aca254e","Type":"ContainerStarted","Data":"edc8b89683dc1cd52c2365e431133f1885f8254e9bd4e1bac237027b38398c82"} Jul 29 18:36:57.101880 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:36:57.101517 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e38c79a4-a26e-4ac4-9e5b-e7060aca254e","Type":"ContainerStarted","Data":"9617ebc140d60540c4d556c920fb3a5bcbce963b9181df733571378eacb9941e"} Jul 29 18:36:57.101880 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:36:57.101559 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e38c79a4-a26e-4ac4-9e5b-e7060aca254e","Type":"ContainerStarted","Data":"9b32f5281015c221cd2455b0515133cb69291642f4986364e50185578b6184a3"} Jul 29 18:36:57.101880 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:36:57.101587 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e38c79a4-a26e-4ac4-9e5b-e7060aca254e","Type":"ContainerStarted","Data":"9f6e9af8f8eab6cfd5bd7562911848790a5b803cda30ffbc6ebcd1eaeb49c952"} Jul 29 18:36:57.101880 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:36:57.101609 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e38c79a4-a26e-4ac4-9e5b-e7060aca254e","Type":"ContainerStarted","Data":"3df9aa32fa2e58418b1049d0ccf9d4165752307cbd2e6be152c3b07b9cfd5602"} Jul 29 18:36:57.103507 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:36:57.103487 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6c8f44d988-8hrdm" event={"ID":"bc69e3d8-7af1-4366-bfe1-531aecbcf766","Type":"ContainerStarted","Data":"d92d145c170709891efa18729906ea38e429eea695c1b0718dde78c5f7520509"} Jul 29 18:36:57.103608 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:36:57.103527 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6c8f44d988-8hrdm" event={"ID":"bc69e3d8-7af1-4366-bfe1-531aecbcf766","Type":"ContainerStarted","Data":"2f331ea67aaa8fa868d88a4edb8ed92ba3da79acdf6a30503868a3ebed60dd06"} Jul 29 18:36:57.103608 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:36:57.103544 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6c8f44d988-8hrdm" event={"ID":"bc69e3d8-7af1-4366-bfe1-531aecbcf766","Type":"ContainerStarted","Data":"357a9bb0b4956437b477d4f47baf160c59ba40ecbb98441acdaa42ddbd7676d4"} Jul 29 18:36:57.128303 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:36:57.128252 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.514272854 podStartE2EDuration="6.128235091s" totalImagesPullingTime="3.594580662s" totalInitContainerRuntime="19.381575ms" isStatefulPod=false podCreationTimestamp="2026-07-29 18:36:51 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 18:36:57.127416113 +0000 UTC m=+152.353084985" watchObservedRunningTime="2026-07-29 18:36:57.128235091 +0000 UTC m=+152.353903965" Jul 29 18:36:57.148526 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:36:57.148472 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-6c8f44d988-8hrdm" podStartSLOduration=1.785245787 podStartE2EDuration="4.148455259s" totalImagesPullingTime="2.363209472s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 18:36:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 18:36:57.147278159 +0000 UTC m=+152.372947033" watchObservedRunningTime="2026-07-29 18:36:57.148455259 +0000 UTC m=+152.374124132" Jul 29 18:36:57.955706 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:36:57.955672 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-84f48789c9-jhm89"] Jul 29 18:36:57.958088 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:36:57.958072 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-84f48789c9-jhm89" Jul 29 18:36:57.960676 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:36:57.960654 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 29 18:36:57.960788 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:36:57.960653 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 29 18:36:57.960788 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:36:57.960686 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 29 18:36:57.961516 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:36:57.961493 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 29 18:36:57.961613 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:36:57.961528 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-9hd9l\"" Jul 29 18:36:57.961613 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:36:57.961564 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 29 18:36:57.961855 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:36:57.961839 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 29 18:36:57.961906 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:36:57.961853 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 29 18:36:57.965890 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:36:57.965872 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 29 18:36:57.969501 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:36:57.969480 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-84f48789c9-jhm89"] Jul 29 18:36:58.002803 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:36:58.002780 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:36:58.029117 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:36:58.029096 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2167ef34-3541-4220-b346-8d0410d865ed-console-serving-cert\") pod \"console-84f48789c9-jhm89\" (UID: \"2167ef34-3541-4220-b346-8d0410d865ed\") " pod="openshift-console/console-84f48789c9-jhm89" Jul 29 18:36:58.029180 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:36:58.029130 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2167ef34-3541-4220-b346-8d0410d865ed-trusted-ca-bundle\") pod \"console-84f48789c9-jhm89\" (UID: \"2167ef34-3541-4220-b346-8d0410d865ed\") " pod="openshift-console/console-84f48789c9-jhm89" Jul 29 18:36:58.029180 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:36:58.029167 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2167ef34-3541-4220-b346-8d0410d865ed-console-config\") pod \"console-84f48789c9-jhm89\" (UID: \"2167ef34-3541-4220-b346-8d0410d865ed\") " pod="openshift-console/console-84f48789c9-jhm89" Jul 29 18:36:58.029257 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:36:58.029241 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2167ef34-3541-4220-b346-8d0410d865ed-service-ca\") pod \"console-84f48789c9-jhm89\" (UID: \"2167ef34-3541-4220-b346-8d0410d865ed\") " pod="openshift-console/console-84f48789c9-jhm89" Jul 29 18:36:58.029296 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:36:58.029286 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ff84b\" (UniqueName: \"kubernetes.io/projected/2167ef34-3541-4220-b346-8d0410d865ed-kube-api-access-ff84b\") pod \"console-84f48789c9-jhm89\" (UID: \"2167ef34-3541-4220-b346-8d0410d865ed\") " pod="openshift-console/console-84f48789c9-jhm89" Jul 29 18:36:58.029328 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:36:58.029311 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2167ef34-3541-4220-b346-8d0410d865ed-console-oauth-config\") pod \"console-84f48789c9-jhm89\" (UID: \"2167ef34-3541-4220-b346-8d0410d865ed\") " pod="openshift-console/console-84f48789c9-jhm89" Jul 29 18:36:58.029380 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:36:58.029331 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2167ef34-3541-4220-b346-8d0410d865ed-oauth-serving-cert\") pod \"console-84f48789c9-jhm89\" (UID: \"2167ef34-3541-4220-b346-8d0410d865ed\") " pod="openshift-console/console-84f48789c9-jhm89" Jul 29 18:36:58.130473 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:36:58.130448 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2167ef34-3541-4220-b346-8d0410d865ed-console-config\") pod \"console-84f48789c9-jhm89\" (UID: \"2167ef34-3541-4220-b346-8d0410d865ed\") " pod="openshift-console/console-84f48789c9-jhm89" Jul 29 18:36:58.130823 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:36:58.130506 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2167ef34-3541-4220-b346-8d0410d865ed-service-ca\") pod \"console-84f48789c9-jhm89\" (UID: \"2167ef34-3541-4220-b346-8d0410d865ed\") " pod="openshift-console/console-84f48789c9-jhm89" Jul 29 18:36:58.130823 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:36:58.130538 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ff84b\" (UniqueName: \"kubernetes.io/projected/2167ef34-3541-4220-b346-8d0410d865ed-kube-api-access-ff84b\") pod \"console-84f48789c9-jhm89\" (UID: \"2167ef34-3541-4220-b346-8d0410d865ed\") " pod="openshift-console/console-84f48789c9-jhm89" Jul 29 18:36:58.130823 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:36:58.130644 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2167ef34-3541-4220-b346-8d0410d865ed-console-oauth-config\") pod \"console-84f48789c9-jhm89\" (UID: \"2167ef34-3541-4220-b346-8d0410d865ed\") " pod="openshift-console/console-84f48789c9-jhm89" Jul 29 18:36:58.130823 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:36:58.130684 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2167ef34-3541-4220-b346-8d0410d865ed-oauth-serving-cert\") pod \"console-84f48789c9-jhm89\" (UID: \"2167ef34-3541-4220-b346-8d0410d865ed\") " pod="openshift-console/console-84f48789c9-jhm89" Jul 29 18:36:58.130823 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:36:58.130728 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2167ef34-3541-4220-b346-8d0410d865ed-console-serving-cert\") pod \"console-84f48789c9-jhm89\" (UID: \"2167ef34-3541-4220-b346-8d0410d865ed\") " pod="openshift-console/console-84f48789c9-jhm89" Jul 29 18:36:58.130823 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:36:58.130752 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2167ef34-3541-4220-b346-8d0410d865ed-trusted-ca-bundle\") pod \"console-84f48789c9-jhm89\" (UID: \"2167ef34-3541-4220-b346-8d0410d865ed\") " pod="openshift-console/console-84f48789c9-jhm89" Jul 29 18:36:58.131133 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:36:58.131085 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2167ef34-3541-4220-b346-8d0410d865ed-console-config\") pod \"console-84f48789c9-jhm89\" (UID: \"2167ef34-3541-4220-b346-8d0410d865ed\") " pod="openshift-console/console-84f48789c9-jhm89" Jul 29 18:36:58.131281 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:36:58.131255 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2167ef34-3541-4220-b346-8d0410d865ed-service-ca\") pod \"console-84f48789c9-jhm89\" (UID: \"2167ef34-3541-4220-b346-8d0410d865ed\") " pod="openshift-console/console-84f48789c9-jhm89" Jul 29 18:36:58.131416 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:36:58.131297 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2167ef34-3541-4220-b346-8d0410d865ed-oauth-serving-cert\") pod \"console-84f48789c9-jhm89\" (UID: \"2167ef34-3541-4220-b346-8d0410d865ed\") " pod="openshift-console/console-84f48789c9-jhm89" Jul 29 18:36:58.131640 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:36:58.131618 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2167ef34-3541-4220-b346-8d0410d865ed-trusted-ca-bundle\") pod \"console-84f48789c9-jhm89\" (UID: \"2167ef34-3541-4220-b346-8d0410d865ed\") " pod="openshift-console/console-84f48789c9-jhm89" Jul 29 18:36:58.133561 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:36:58.133542 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2167ef34-3541-4220-b346-8d0410d865ed-console-oauth-config\") pod \"console-84f48789c9-jhm89\" (UID: \"2167ef34-3541-4220-b346-8d0410d865ed\") " pod="openshift-console/console-84f48789c9-jhm89" Jul 29 18:36:58.133687 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:36:58.133670 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2167ef34-3541-4220-b346-8d0410d865ed-console-serving-cert\") pod \"console-84f48789c9-jhm89\" (UID: \"2167ef34-3541-4220-b346-8d0410d865ed\") " pod="openshift-console/console-84f48789c9-jhm89" Jul 29 18:36:58.139514 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:36:58.139494 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ff84b\" (UniqueName: \"kubernetes.io/projected/2167ef34-3541-4220-b346-8d0410d865ed-kube-api-access-ff84b\") pod \"console-84f48789c9-jhm89\" (UID: \"2167ef34-3541-4220-b346-8d0410d865ed\") " pod="openshift-console/console-84f48789c9-jhm89" Jul 29 18:36:58.268535 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:36:58.268506 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-84f48789c9-jhm89" Jul 29 18:36:58.417759 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:36:58.417718 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-84f48789c9-jhm89"] Jul 29 18:36:58.420817 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:36:58.420791 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2167ef34_3541_4220_b346_8d0410d865ed.slice/crio-01a3daa32e2a62f38bb3c49f88f23ea267fc2a973eb40c8b875309a09547e557 WatchSource:0}: Error finding container 01a3daa32e2a62f38bb3c49f88f23ea267fc2a973eb40c8b875309a09547e557: Status 404 returned error can't find the container with id 01a3daa32e2a62f38bb3c49f88f23ea267fc2a973eb40c8b875309a09547e557 Jul 29 18:36:59.112692 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:36:59.112657 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-84f48789c9-jhm89" event={"ID":"2167ef34-3541-4220-b346-8d0410d865ed","Type":"ContainerStarted","Data":"675a8cb64140c43291f33da84c68d969da747a3c8b78505ec2b25c0f81f6cc7a"} Jul 29 18:36:59.112860 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:36:59.112714 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-84f48789c9-jhm89" event={"ID":"2167ef34-3541-4220-b346-8d0410d865ed","Type":"ContainerStarted","Data":"01a3daa32e2a62f38bb3c49f88f23ea267fc2a973eb40c8b875309a09547e557"} Jul 29 18:36:59.130219 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:36:59.130175 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-84f48789c9-jhm89" podStartSLOduration=2.130162352 podStartE2EDuration="2.130162352s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 18:36:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 18:36:59.129014127 +0000 UTC m=+154.354682999" watchObservedRunningTime="2026-07-29 18:36:59.130162352 +0000 UTC m=+154.355831224" Jul 29 18:37:08.269163 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:37:08.269110 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-84f48789c9-jhm89" Jul 29 18:37:08.269163 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:37:08.269177 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-84f48789c9-jhm89" Jul 29 18:37:08.274015 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:37:08.273995 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-84f48789c9-jhm89" Jul 29 18:37:09.158748 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:37:09.158719 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-84f48789c9-jhm89" Jul 29 18:37:09.325633 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:37:09.325602 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:37:09.424977 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:37:09.424899 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:37:09.485120 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:37:09.485095 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:37:28.863958 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:37:28.863928 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:38:03.423333 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:38:03.423278 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:38:03.479953 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:38:03.479882 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:38:04.797246 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:38:04.797211 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6f54987646-w92xj"] Jul 29 18:38:04.799699 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:38:04.799681 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6f54987646-w92xj" Jul 29 18:38:04.813134 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:38:04.813110 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6f54987646-w92xj"] Jul 29 18:38:04.851430 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:38:04.851396 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/96478f22-2f3c-44c6-b499-6906f50cc643-console-oauth-config\") pod \"console-6f54987646-w92xj\" (UID: \"96478f22-2f3c-44c6-b499-6906f50cc643\") " pod="openshift-console/console-6f54987646-w92xj" Jul 29 18:38:04.851553 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:38:04.851439 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/96478f22-2f3c-44c6-b499-6906f50cc643-service-ca\") pod \"console-6f54987646-w92xj\" (UID: \"96478f22-2f3c-44c6-b499-6906f50cc643\") " pod="openshift-console/console-6f54987646-w92xj" Jul 29 18:38:04.851553 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:38:04.851455 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/96478f22-2f3c-44c6-b499-6906f50cc643-trusted-ca-bundle\") pod \"console-6f54987646-w92xj\" (UID: \"96478f22-2f3c-44c6-b499-6906f50cc643\") " pod="openshift-console/console-6f54987646-w92xj" Jul 29 18:38:04.851553 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:38:04.851533 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/96478f22-2f3c-44c6-b499-6906f50cc643-console-serving-cert\") pod \"console-6f54987646-w92xj\" (UID: \"96478f22-2f3c-44c6-b499-6906f50cc643\") " pod="openshift-console/console-6f54987646-w92xj" Jul 29 18:38:04.851668 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:38:04.851580 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/96478f22-2f3c-44c6-b499-6906f50cc643-oauth-serving-cert\") pod \"console-6f54987646-w92xj\" (UID: \"96478f22-2f3c-44c6-b499-6906f50cc643\") " pod="openshift-console/console-6f54987646-w92xj" Jul 29 18:38:04.851668 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:38:04.851631 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/96478f22-2f3c-44c6-b499-6906f50cc643-console-config\") pod \"console-6f54987646-w92xj\" (UID: \"96478f22-2f3c-44c6-b499-6906f50cc643\") " pod="openshift-console/console-6f54987646-w92xj" Jul 29 18:38:04.851668 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:38:04.851648 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-llfj6\" (UniqueName: \"kubernetes.io/projected/96478f22-2f3c-44c6-b499-6906f50cc643-kube-api-access-llfj6\") pod \"console-6f54987646-w92xj\" (UID: \"96478f22-2f3c-44c6-b499-6906f50cc643\") " pod="openshift-console/console-6f54987646-w92xj" Jul 29 18:38:04.856569 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:38:04.856543 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:38:04.911741 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:38:04.911716 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:38:04.953019 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:38:04.952987 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/96478f22-2f3c-44c6-b499-6906f50cc643-console-config\") pod \"console-6f54987646-w92xj\" (UID: \"96478f22-2f3c-44c6-b499-6906f50cc643\") " pod="openshift-console/console-6f54987646-w92xj" Jul 29 18:38:04.953154 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:38:04.953028 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-llfj6\" (UniqueName: \"kubernetes.io/projected/96478f22-2f3c-44c6-b499-6906f50cc643-kube-api-access-llfj6\") pod \"console-6f54987646-w92xj\" (UID: \"96478f22-2f3c-44c6-b499-6906f50cc643\") " pod="openshift-console/console-6f54987646-w92xj" Jul 29 18:38:04.953154 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:38:04.953052 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/96478f22-2f3c-44c6-b499-6906f50cc643-console-oauth-config\") pod \"console-6f54987646-w92xj\" (UID: \"96478f22-2f3c-44c6-b499-6906f50cc643\") " pod="openshift-console/console-6f54987646-w92xj" Jul 29 18:38:04.953154 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:38:04.953075 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/96478f22-2f3c-44c6-b499-6906f50cc643-service-ca\") pod \"console-6f54987646-w92xj\" (UID: \"96478f22-2f3c-44c6-b499-6906f50cc643\") " pod="openshift-console/console-6f54987646-w92xj" Jul 29 18:38:04.953154 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:38:04.953097 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/96478f22-2f3c-44c6-b499-6906f50cc643-trusted-ca-bundle\") pod \"console-6f54987646-w92xj\" (UID: \"96478f22-2f3c-44c6-b499-6906f50cc643\") " pod="openshift-console/console-6f54987646-w92xj" Jul 29 18:38:04.953154 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:38:04.953135 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/96478f22-2f3c-44c6-b499-6906f50cc643-console-serving-cert\") pod \"console-6f54987646-w92xj\" (UID: \"96478f22-2f3c-44c6-b499-6906f50cc643\") " pod="openshift-console/console-6f54987646-w92xj" Jul 29 18:38:04.953382 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:38:04.953184 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/96478f22-2f3c-44c6-b499-6906f50cc643-oauth-serving-cert\") pod \"console-6f54987646-w92xj\" (UID: \"96478f22-2f3c-44c6-b499-6906f50cc643\") " pod="openshift-console/console-6f54987646-w92xj" Jul 29 18:38:04.953900 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:38:04.953880 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/96478f22-2f3c-44c6-b499-6906f50cc643-console-config\") pod \"console-6f54987646-w92xj\" (UID: \"96478f22-2f3c-44c6-b499-6906f50cc643\") " pod="openshift-console/console-6f54987646-w92xj" Jul 29 18:38:04.954003 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:38:04.953904 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/96478f22-2f3c-44c6-b499-6906f50cc643-service-ca\") pod \"console-6f54987646-w92xj\" (UID: \"96478f22-2f3c-44c6-b499-6906f50cc643\") " pod="openshift-console/console-6f54987646-w92xj" Jul 29 18:38:04.954003 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:38:04.953938 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/96478f22-2f3c-44c6-b499-6906f50cc643-oauth-serving-cert\") pod \"console-6f54987646-w92xj\" (UID: \"96478f22-2f3c-44c6-b499-6906f50cc643\") " pod="openshift-console/console-6f54987646-w92xj" Jul 29 18:38:04.954383 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:38:04.954331 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/96478f22-2f3c-44c6-b499-6906f50cc643-trusted-ca-bundle\") pod \"console-6f54987646-w92xj\" (UID: \"96478f22-2f3c-44c6-b499-6906f50cc643\") " pod="openshift-console/console-6f54987646-w92xj" Jul 29 18:38:04.955734 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:38:04.955706 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/96478f22-2f3c-44c6-b499-6906f50cc643-console-serving-cert\") pod \"console-6f54987646-w92xj\" (UID: \"96478f22-2f3c-44c6-b499-6906f50cc643\") " pod="openshift-console/console-6f54987646-w92xj" Jul 29 18:38:04.955815 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:38:04.955760 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/96478f22-2f3c-44c6-b499-6906f50cc643-console-oauth-config\") pod \"console-6f54987646-w92xj\" (UID: \"96478f22-2f3c-44c6-b499-6906f50cc643\") " pod="openshift-console/console-6f54987646-w92xj" Jul 29 18:38:04.964121 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:38:04.964103 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-llfj6\" (UniqueName: \"kubernetes.io/projected/96478f22-2f3c-44c6-b499-6906f50cc643-kube-api-access-llfj6\") pod \"console-6f54987646-w92xj\" (UID: \"96478f22-2f3c-44c6-b499-6906f50cc643\") " pod="openshift-console/console-6f54987646-w92xj" Jul 29 18:38:04.996621 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:38:04.996588 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:38:05.050258 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:38:05.050185 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:38:05.108134 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:38:05.108107 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6f54987646-w92xj" Jul 29 18:38:05.233471 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:38:05.233446 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6f54987646-w92xj"] Jul 29 18:38:05.235907 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:38:05.235874 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96478f22_2f3c_44c6_b499_6906f50cc643.slice/crio-4ddec03e9ddb0a312c4775841a188866d1f42cc1db38388da496cbc121364ed0 WatchSource:0}: Error finding container 4ddec03e9ddb0a312c4775841a188866d1f42cc1db38388da496cbc121364ed0: Status 404 returned error can't find the container with id 4ddec03e9ddb0a312c4775841a188866d1f42cc1db38388da496cbc121364ed0 Jul 29 18:38:05.358474 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:38:05.358409 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6f54987646-w92xj" event={"ID":"96478f22-2f3c-44c6-b499-6906f50cc643","Type":"ContainerStarted","Data":"e532ca10a1e45b3532d6cd1ff76bb9600456a14713a14250378d56a4de2e90be"} Jul 29 18:38:05.358474 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:38:05.358450 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6f54987646-w92xj" event={"ID":"96478f22-2f3c-44c6-b499-6906f50cc643","Type":"ContainerStarted","Data":"4ddec03e9ddb0a312c4775841a188866d1f42cc1db38388da496cbc121364ed0"} Jul 29 18:38:05.374703 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:38:05.374659 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6f54987646-w92xj" podStartSLOduration=1.374644 podStartE2EDuration="1.374644s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 18:38:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 18:38:05.37424991 +0000 UTC m=+220.599918782" watchObservedRunningTime="2026-07-29 18:38:05.374644 +0000 UTC m=+220.600312882" Jul 29 18:38:05.608997 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:38:05.608894 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:38:05.728416 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:38:05.728376 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:38:15.109193 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:38:15.109158 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6f54987646-w92xj" Jul 29 18:38:15.109774 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:38:15.109337 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6f54987646-w92xj" Jul 29 18:38:15.113902 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:38:15.113879 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6f54987646-w92xj" Jul 29 18:38:15.279729 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:38:15.279698 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:38:15.328285 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:38:15.328258 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:38:15.396235 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:38:15.396157 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6f54987646-w92xj" Jul 29 18:38:15.443877 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:38:15.443844 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-84f48789c9-jhm89"] Jul 29 18:38:15.865738 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:38:15.865708 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:38:15.920951 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:38:15.920922 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:38:19.538245 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:38:19.538207 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:38:26.996022 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:38:26.995984 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:38:27.099241 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:38:27.099212 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:38:36.169591 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:38:36.169558 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:38:36.230133 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:38:36.230108 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:38:40.463637 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:38:40.463581 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-84f48789c9-jhm89" podUID="2167ef34-3541-4220-b346-8d0410d865ed" containerName="console" containerID="cri-o://675a8cb64140c43291f33da84c68d969da747a3c8b78505ec2b25c0f81f6cc7a" gracePeriod=15 Jul 29 18:38:40.707024 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:38:40.707003 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-84f48789c9-jhm89" Jul 29 18:38:40.743455 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:38:40.743401 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2167ef34-3541-4220-b346-8d0410d865ed-console-oauth-config\") pod \"2167ef34-3541-4220-b346-8d0410d865ed\" (UID: \"2167ef34-3541-4220-b346-8d0410d865ed\") " Jul 29 18:38:40.743455 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:38:40.743438 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2167ef34-3541-4220-b346-8d0410d865ed-service-ca\") pod \"2167ef34-3541-4220-b346-8d0410d865ed\" (UID: \"2167ef34-3541-4220-b346-8d0410d865ed\") " Jul 29 18:38:40.743627 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:38:40.743458 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2167ef34-3541-4220-b346-8d0410d865ed-console-serving-cert\") pod \"2167ef34-3541-4220-b346-8d0410d865ed\" (UID: \"2167ef34-3541-4220-b346-8d0410d865ed\") " Jul 29 18:38:40.743627 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:38:40.743473 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2167ef34-3541-4220-b346-8d0410d865ed-console-config\") pod \"2167ef34-3541-4220-b346-8d0410d865ed\" (UID: \"2167ef34-3541-4220-b346-8d0410d865ed\") " Jul 29 18:38:40.743627 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:38:40.743509 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2167ef34-3541-4220-b346-8d0410d865ed-trusted-ca-bundle\") pod \"2167ef34-3541-4220-b346-8d0410d865ed\" (UID: \"2167ef34-3541-4220-b346-8d0410d865ed\") " Jul 29 18:38:40.743627 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:38:40.743555 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2167ef34-3541-4220-b346-8d0410d865ed-oauth-serving-cert\") pod \"2167ef34-3541-4220-b346-8d0410d865ed\" (UID: \"2167ef34-3541-4220-b346-8d0410d865ed\") " Jul 29 18:38:40.743627 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:38:40.743574 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ff84b\" (UniqueName: \"kubernetes.io/projected/2167ef34-3541-4220-b346-8d0410d865ed-kube-api-access-ff84b\") pod \"2167ef34-3541-4220-b346-8d0410d865ed\" (UID: \"2167ef34-3541-4220-b346-8d0410d865ed\") " Jul 29 18:38:40.743881 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:38:40.743842 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2167ef34-3541-4220-b346-8d0410d865ed-service-ca" (OuterVolumeSpecName: "service-ca") pod "2167ef34-3541-4220-b346-8d0410d865ed" (UID: "2167ef34-3541-4220-b346-8d0410d865ed"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 18:38:40.743881 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:38:40.743840 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2167ef34-3541-4220-b346-8d0410d865ed-console-config" (OuterVolumeSpecName: "console-config") pod "2167ef34-3541-4220-b346-8d0410d865ed" (UID: "2167ef34-3541-4220-b346-8d0410d865ed"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 18:38:40.743971 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:38:40.743910 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2167ef34-3541-4220-b346-8d0410d865ed-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "2167ef34-3541-4220-b346-8d0410d865ed" (UID: "2167ef34-3541-4220-b346-8d0410d865ed"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 18:38:40.743971 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:38:40.743940 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2167ef34-3541-4220-b346-8d0410d865ed-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "2167ef34-3541-4220-b346-8d0410d865ed" (UID: "2167ef34-3541-4220-b346-8d0410d865ed"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 18:38:40.745756 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:38:40.745733 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2167ef34-3541-4220-b346-8d0410d865ed-kube-api-access-ff84b" (OuterVolumeSpecName: "kube-api-access-ff84b") pod "2167ef34-3541-4220-b346-8d0410d865ed" (UID: "2167ef34-3541-4220-b346-8d0410d865ed"). InnerVolumeSpecName "kube-api-access-ff84b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 18:38:40.745843 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:38:40.745802 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2167ef34-3541-4220-b346-8d0410d865ed-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "2167ef34-3541-4220-b346-8d0410d865ed" (UID: "2167ef34-3541-4220-b346-8d0410d865ed"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 18:38:40.745913 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:38:40.745892 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2167ef34-3541-4220-b346-8d0410d865ed-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "2167ef34-3541-4220-b346-8d0410d865ed" (UID: "2167ef34-3541-4220-b346-8d0410d865ed"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 18:38:40.844547 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:38:40.844516 2560 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2167ef34-3541-4220-b346-8d0410d865ed-oauth-serving-cert\") on node \"ip-10-0-140-205.ec2.internal\" DevicePath \"\"" Jul 29 18:38:40.844547 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:38:40.844541 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ff84b\" (UniqueName: \"kubernetes.io/projected/2167ef34-3541-4220-b346-8d0410d865ed-kube-api-access-ff84b\") on node \"ip-10-0-140-205.ec2.internal\" DevicePath \"\"" Jul 29 18:38:40.844547 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:38:40.844552 2560 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2167ef34-3541-4220-b346-8d0410d865ed-console-oauth-config\") on node \"ip-10-0-140-205.ec2.internal\" DevicePath \"\"" Jul 29 18:38:40.844740 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:38:40.844562 2560 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2167ef34-3541-4220-b346-8d0410d865ed-service-ca\") on node \"ip-10-0-140-205.ec2.internal\" DevicePath \"\"" Jul 29 18:38:40.844740 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:38:40.844570 2560 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2167ef34-3541-4220-b346-8d0410d865ed-console-serving-cert\") on node \"ip-10-0-140-205.ec2.internal\" DevicePath \"\"" Jul 29 18:38:40.844740 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:38:40.844579 2560 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2167ef34-3541-4220-b346-8d0410d865ed-console-config\") on node \"ip-10-0-140-205.ec2.internal\" DevicePath \"\"" Jul 29 18:38:40.844740 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:38:40.844587 2560 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2167ef34-3541-4220-b346-8d0410d865ed-trusted-ca-bundle\") on node \"ip-10-0-140-205.ec2.internal\" DevicePath \"\"" Jul 29 18:38:41.487563 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:38:41.487526 2560 generic.go:356] "Generic (PLEG): container finished" podID="2167ef34-3541-4220-b346-8d0410d865ed" containerID="675a8cb64140c43291f33da84c68d969da747a3c8b78505ec2b25c0f81f6cc7a" exitCode=0 Jul 29 18:38:41.487945 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:38:41.487576 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-84f48789c9-jhm89" Jul 29 18:38:41.487945 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:38:41.487603 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-84f48789c9-jhm89" event={"ID":"2167ef34-3541-4220-b346-8d0410d865ed","Type":"ContainerDied","Data":"675a8cb64140c43291f33da84c68d969da747a3c8b78505ec2b25c0f81f6cc7a"} Jul 29 18:38:41.487945 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:38:41.487659 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-84f48789c9-jhm89" event={"ID":"2167ef34-3541-4220-b346-8d0410d865ed","Type":"ContainerDied","Data":"01a3daa32e2a62f38bb3c49f88f23ea267fc2a973eb40c8b875309a09547e557"} Jul 29 18:38:41.487945 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:38:41.487679 2560 scope.go:117] "RemoveContainer" containerID="675a8cb64140c43291f33da84c68d969da747a3c8b78505ec2b25c0f81f6cc7a" Jul 29 18:38:41.496171 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:38:41.496152 2560 scope.go:117] "RemoveContainer" containerID="675a8cb64140c43291f33da84c68d969da747a3c8b78505ec2b25c0f81f6cc7a" Jul 29 18:38:41.496453 ip-10-0-140-205 kubenswrapper[2560]: E0729 18:38:41.496431 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"675a8cb64140c43291f33da84c68d969da747a3c8b78505ec2b25c0f81f6cc7a\": container with ID starting with 675a8cb64140c43291f33da84c68d969da747a3c8b78505ec2b25c0f81f6cc7a not found: ID does not exist" containerID="675a8cb64140c43291f33da84c68d969da747a3c8b78505ec2b25c0f81f6cc7a" Jul 29 18:38:41.496521 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:38:41.496463 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"675a8cb64140c43291f33da84c68d969da747a3c8b78505ec2b25c0f81f6cc7a"} err="failed to get container status \"675a8cb64140c43291f33da84c68d969da747a3c8b78505ec2b25c0f81f6cc7a\": rpc error: code = NotFound desc = could not find container \"675a8cb64140c43291f33da84c68d969da747a3c8b78505ec2b25c0f81f6cc7a\": container with ID starting with 675a8cb64140c43291f33da84c68d969da747a3c8b78505ec2b25c0f81f6cc7a not found: ID does not exist" Jul 29 18:38:41.504290 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:38:41.504256 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-84f48789c9-jhm89"] Jul 29 18:38:41.509464 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:38:41.509441 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-84f48789c9-jhm89"] Jul 29 18:38:43.352548 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:38:43.352516 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2167ef34-3541-4220-b346-8d0410d865ed" path="/var/lib/kubelet/pods/2167ef34-3541-4220-b346-8d0410d865ed/volumes" Jul 29 18:39:17.824461 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:39:17.824388 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:39:18.461186 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:39:18.461152 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mkb5d"] Jul 29 18:39:18.461623 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:39:18.461605 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2167ef34-3541-4220-b346-8d0410d865ed" containerName="console" Jul 29 18:39:18.461678 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:39:18.461628 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="2167ef34-3541-4220-b346-8d0410d865ed" containerName="console" Jul 29 18:39:18.461745 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:39:18.461733 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="2167ef34-3541-4220-b346-8d0410d865ed" containerName="console" Jul 29 18:39:18.464851 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:39:18.464830 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mkb5d" Jul 29 18:39:18.467397 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:39:18.467378 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 29 18:39:18.467397 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:39:18.467394 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 29 18:39:18.467560 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:39:18.467435 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-dl8pd\"" Jul 29 18:39:18.474135 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:39:18.474114 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mkb5d"] Jul 29 18:39:18.636003 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:39:18.635968 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/40b0762f-1f72-4570-a907-e5deab74a627-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mkb5d\" (UID: \"40b0762f-1f72-4570-a907-e5deab74a627\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mkb5d" Jul 29 18:39:18.636183 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:39:18.636034 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h52dm\" (UniqueName: \"kubernetes.io/projected/40b0762f-1f72-4570-a907-e5deab74a627-kube-api-access-h52dm\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mkb5d\" (UID: \"40b0762f-1f72-4570-a907-e5deab74a627\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mkb5d" Jul 29 18:39:18.636183 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:39:18.636060 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/40b0762f-1f72-4570-a907-e5deab74a627-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mkb5d\" (UID: \"40b0762f-1f72-4570-a907-e5deab74a627\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mkb5d" Jul 29 18:39:18.736984 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:39:18.736895 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h52dm\" (UniqueName: \"kubernetes.io/projected/40b0762f-1f72-4570-a907-e5deab74a627-kube-api-access-h52dm\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mkb5d\" (UID: \"40b0762f-1f72-4570-a907-e5deab74a627\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mkb5d" Jul 29 18:39:18.736984 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:39:18.736933 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/40b0762f-1f72-4570-a907-e5deab74a627-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mkb5d\" (UID: \"40b0762f-1f72-4570-a907-e5deab74a627\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mkb5d" Jul 29 18:39:18.737191 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:39:18.737013 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/40b0762f-1f72-4570-a907-e5deab74a627-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mkb5d\" (UID: \"40b0762f-1f72-4570-a907-e5deab74a627\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mkb5d" Jul 29 18:39:18.737323 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:39:18.737302 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/40b0762f-1f72-4570-a907-e5deab74a627-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mkb5d\" (UID: \"40b0762f-1f72-4570-a907-e5deab74a627\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mkb5d" Jul 29 18:39:18.737458 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:39:18.737439 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/40b0762f-1f72-4570-a907-e5deab74a627-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mkb5d\" (UID: \"40b0762f-1f72-4570-a907-e5deab74a627\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mkb5d" Jul 29 18:39:18.745867 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:39:18.745844 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h52dm\" (UniqueName: \"kubernetes.io/projected/40b0762f-1f72-4570-a907-e5deab74a627-kube-api-access-h52dm\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mkb5d\" (UID: \"40b0762f-1f72-4570-a907-e5deab74a627\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mkb5d" Jul 29 18:39:18.775824 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:39:18.775803 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mkb5d" Jul 29 18:39:18.900953 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:39:18.900915 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mkb5d"] Jul 29 18:39:18.903694 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:39:18.903663 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40b0762f_1f72_4570_a907_e5deab74a627.slice/crio-45e3953b6e51ec32f7a5a8ff61b361b13dd613eb5a4dc25410da5ffbf7dca000 WatchSource:0}: Error finding container 45e3953b6e51ec32f7a5a8ff61b361b13dd613eb5a4dc25410da5ffbf7dca000: Status 404 returned error can't find the container with id 45e3953b6e51ec32f7a5a8ff61b361b13dd613eb5a4dc25410da5ffbf7dca000 Jul 29 18:39:19.436871 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:39:19.436837 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:39:19.643229 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:39:19.643190 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mkb5d" event={"ID":"40b0762f-1f72-4570-a907-e5deab74a627","Type":"ContainerStarted","Data":"45e3953b6e51ec32f7a5a8ff61b361b13dd613eb5a4dc25410da5ffbf7dca000"} Jul 29 18:39:25.306654 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:39:25.306628 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-v4n5x_1727fb40-fbed-4984-91d4-a5890d0f4103/ovn-acl-logging/0.log" Jul 29 18:39:25.307539 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:39:25.307522 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-v4n5x_1727fb40-fbed-4984-91d4-a5890d0f4103/ovn-acl-logging/0.log" Jul 29 18:39:25.312332 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:39:25.312237 2560 kubelet.go:1648] "Image garbage collection succeeded" Jul 29 18:39:25.670635 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:39:25.670597 2560 generic.go:356] "Generic (PLEG): container finished" podID="40b0762f-1f72-4570-a907-e5deab74a627" containerID="ec5f9987f0fbeb563e8568b1d81011177dd7cd81911884cf555a765f985fd36f" exitCode=0 Jul 29 18:39:25.670779 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:39:25.670682 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mkb5d" event={"ID":"40b0762f-1f72-4570-a907-e5deab74a627","Type":"ContainerDied","Data":"ec5f9987f0fbeb563e8568b1d81011177dd7cd81911884cf555a765f985fd36f"} Jul 29 18:39:27.682325 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:39:27.682293 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mkb5d" event={"ID":"40b0762f-1f72-4570-a907-e5deab74a627","Type":"ContainerStarted","Data":"f78771a05a120d76c84e5509aef7ce1d9630312930d923c12aca9e5195783a7f"} Jul 29 18:39:28.687870 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:39:28.687833 2560 generic.go:356] "Generic (PLEG): container finished" podID="40b0762f-1f72-4570-a907-e5deab74a627" containerID="f78771a05a120d76c84e5509aef7ce1d9630312930d923c12aca9e5195783a7f" exitCode=0 Jul 29 18:39:28.688276 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:39:28.687909 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mkb5d" event={"ID":"40b0762f-1f72-4570-a907-e5deab74a627","Type":"ContainerDied","Data":"f78771a05a120d76c84e5509aef7ce1d9630312930d923c12aca9e5195783a7f"} Jul 29 18:39:28.688915 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:39:28.688901 2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 29 18:39:35.718754 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:39:35.718716 2560 generic.go:356] "Generic (PLEG): container finished" podID="40b0762f-1f72-4570-a907-e5deab74a627" containerID="ffe3887927ea4f5c0cb7673f181e656672088b9ea2ae3c355e6b8d6eac6a019d" exitCode=0 Jul 29 18:39:35.719107 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:39:35.718794 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mkb5d" event={"ID":"40b0762f-1f72-4570-a907-e5deab74a627","Type":"ContainerDied","Data":"ffe3887927ea4f5c0cb7673f181e656672088b9ea2ae3c355e6b8d6eac6a019d"} Jul 29 18:39:36.845563 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:39:36.845539 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mkb5d" Jul 29 18:39:36.899367 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:39:36.899324 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/40b0762f-1f72-4570-a907-e5deab74a627-bundle\") pod \"40b0762f-1f72-4570-a907-e5deab74a627\" (UID: \"40b0762f-1f72-4570-a907-e5deab74a627\") " Jul 29 18:39:36.899526 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:39:36.899431 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h52dm\" (UniqueName: \"kubernetes.io/projected/40b0762f-1f72-4570-a907-e5deab74a627-kube-api-access-h52dm\") pod \"40b0762f-1f72-4570-a907-e5deab74a627\" (UID: \"40b0762f-1f72-4570-a907-e5deab74a627\") " Jul 29 18:39:36.899526 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:39:36.899455 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/40b0762f-1f72-4570-a907-e5deab74a627-util\") pod \"40b0762f-1f72-4570-a907-e5deab74a627\" (UID: \"40b0762f-1f72-4570-a907-e5deab74a627\") " Jul 29 18:39:36.900171 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:39:36.900099 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40b0762f-1f72-4570-a907-e5deab74a627-bundle" (OuterVolumeSpecName: "bundle") pod "40b0762f-1f72-4570-a907-e5deab74a627" (UID: "40b0762f-1f72-4570-a907-e5deab74a627"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 18:39:36.901803 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:39:36.901776 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40b0762f-1f72-4570-a907-e5deab74a627-kube-api-access-h52dm" (OuterVolumeSpecName: "kube-api-access-h52dm") pod "40b0762f-1f72-4570-a907-e5deab74a627" (UID: "40b0762f-1f72-4570-a907-e5deab74a627"). InnerVolumeSpecName "kube-api-access-h52dm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 18:39:36.904366 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:39:36.904326 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40b0762f-1f72-4570-a907-e5deab74a627-util" (OuterVolumeSpecName: "util") pod "40b0762f-1f72-4570-a907-e5deab74a627" (UID: "40b0762f-1f72-4570-a907-e5deab74a627"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 18:39:37.000834 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:39:37.000802 2560 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/40b0762f-1f72-4570-a907-e5deab74a627-util\") on node \"ip-10-0-140-205.ec2.internal\" DevicePath \"\"" Jul 29 18:39:37.000834 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:39:37.000832 2560 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/40b0762f-1f72-4570-a907-e5deab74a627-bundle\") on node \"ip-10-0-140-205.ec2.internal\" DevicePath \"\"" Jul 29 18:39:37.001009 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:39:37.000846 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h52dm\" (UniqueName: \"kubernetes.io/projected/40b0762f-1f72-4570-a907-e5deab74a627-kube-api-access-h52dm\") on node \"ip-10-0-140-205.ec2.internal\" DevicePath \"\"" Jul 29 18:39:37.729043 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:39:37.728961 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mkb5d" event={"ID":"40b0762f-1f72-4570-a907-e5deab74a627","Type":"ContainerDied","Data":"45e3953b6e51ec32f7a5a8ff61b361b13dd613eb5a4dc25410da5ffbf7dca000"} Jul 29 18:39:37.729043 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:39:37.729013 2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="45e3953b6e51ec32f7a5a8ff61b361b13dd613eb5a4dc25410da5ffbf7dca000" Jul 29 18:39:37.729043 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:39:37.728978 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mkb5d" Jul 29 18:39:39.440897 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:39:39.440864 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:39:40.421897 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:39:40.421864 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:39:41.397237 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:39:41.397203 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-74zj9"] Jul 29 18:39:41.397620 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:39:41.397506 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="40b0762f-1f72-4570-a907-e5deab74a627" containerName="util" Jul 29 18:39:41.397620 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:39:41.397517 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="40b0762f-1f72-4570-a907-e5deab74a627" containerName="util" Jul 29 18:39:41.397620 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:39:41.397525 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="40b0762f-1f72-4570-a907-e5deab74a627" containerName="pull" Jul 29 18:39:41.397620 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:39:41.397530 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="40b0762f-1f72-4570-a907-e5deab74a627" containerName="pull" Jul 29 18:39:41.397620 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:39:41.397537 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="40b0762f-1f72-4570-a907-e5deab74a627" containerName="extract" Jul 29 18:39:41.397620 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:39:41.397542 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="40b0762f-1f72-4570-a907-e5deab74a627" containerName="extract" Jul 29 18:39:41.397620 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:39:41.397590 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="40b0762f-1f72-4570-a907-e5deab74a627" containerName="extract" Jul 29 18:39:41.439656 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:39:41.439629 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-74zj9"] Jul 29 18:39:41.439783 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:39:41.439721 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-74zj9" Jul 29 18:39:41.442144 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:39:41.442122 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Jul 29 18:39:41.442267 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:39:41.442147 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Jul 29 18:39:41.442267 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:39:41.442159 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-g85rm\"" Jul 29 18:39:41.538818 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:39:41.538792 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/49b024a7-b9b4-470b-9dbd-4a1dfe31ef07-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-74zj9\" (UID: \"49b024a7-b9b4-470b-9dbd-4a1dfe31ef07\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-74zj9" Jul 29 18:39:41.538961 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:39:41.538840 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4v9f2\" (UniqueName: \"kubernetes.io/projected/49b024a7-b9b4-470b-9dbd-4a1dfe31ef07-kube-api-access-4v9f2\") pod \"cert-manager-operator-controller-manager-55c444bf86-74zj9\" (UID: \"49b024a7-b9b4-470b-9dbd-4a1dfe31ef07\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-74zj9" Jul 29 18:39:41.639402 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:39:41.639375 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/49b024a7-b9b4-470b-9dbd-4a1dfe31ef07-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-74zj9\" (UID: \"49b024a7-b9b4-470b-9dbd-4a1dfe31ef07\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-74zj9" Jul 29 18:39:41.639525 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:39:41.639437 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4v9f2\" (UniqueName: \"kubernetes.io/projected/49b024a7-b9b4-470b-9dbd-4a1dfe31ef07-kube-api-access-4v9f2\") pod \"cert-manager-operator-controller-manager-55c444bf86-74zj9\" (UID: \"49b024a7-b9b4-470b-9dbd-4a1dfe31ef07\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-74zj9" Jul 29 18:39:41.639795 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:39:41.639778 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/49b024a7-b9b4-470b-9dbd-4a1dfe31ef07-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-74zj9\" (UID: \"49b024a7-b9b4-470b-9dbd-4a1dfe31ef07\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-74zj9" Jul 29 18:39:41.653441 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:39:41.653389 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4v9f2\" (UniqueName: \"kubernetes.io/projected/49b024a7-b9b4-470b-9dbd-4a1dfe31ef07-kube-api-access-4v9f2\") pod \"cert-manager-operator-controller-manager-55c444bf86-74zj9\" (UID: \"49b024a7-b9b4-470b-9dbd-4a1dfe31ef07\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-74zj9" Jul 29 18:39:41.748322 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:39:41.748302 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-74zj9" Jul 29 18:39:41.901631 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:39:41.901599 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-74zj9"] Jul 29 18:39:41.905521 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:39:41.905457 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49b024a7_b9b4_470b_9dbd_4a1dfe31ef07.slice/crio-84c12bc42a6526b9690629f46b37027dc82e9a9f59ccee1a0eb1757433ae740c WatchSource:0}: Error finding container 84c12bc42a6526b9690629f46b37027dc82e9a9f59ccee1a0eb1757433ae740c: Status 404 returned error can't find the container with id 84c12bc42a6526b9690629f46b37027dc82e9a9f59ccee1a0eb1757433ae740c Jul 29 18:39:42.749151 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:39:42.749116 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-74zj9" event={"ID":"49b024a7-b9b4-470b-9dbd-4a1dfe31ef07","Type":"ContainerStarted","Data":"84c12bc42a6526b9690629f46b37027dc82e9a9f59ccee1a0eb1757433ae740c"} Jul 29 18:39:43.065958 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:39:43.065870 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:39:43.161916 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:39:43.161877 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:39:44.759752 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:39:44.759719 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-74zj9" event={"ID":"49b024a7-b9b4-470b-9dbd-4a1dfe31ef07","Type":"ContainerStarted","Data":"039528d0ad190bcd9e36d298635241f1550e5f4409f52c894ef28178bb2b4b3f"} Jul 29 18:39:44.780407 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:39:44.780340 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-74zj9" podStartSLOduration=1.445353866 podStartE2EDuration="3.7803216s" totalImagesPullingTime="2.334967734s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 18:39:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 18:39:44.778030406 +0000 UTC m=+320.003699278" watchObservedRunningTime="2026-07-29 18:39:44.7803216 +0000 UTC m=+320.005990472" Jul 29 18:39:44.993377 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:39:44.993329 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:39:45.108206 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:39:45.108131 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:39:45.288521 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:39:45.288486 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:39:45.386307 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:39:45.386233 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:39:45.923308 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:39:45.923272 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcd8vx"] Jul 29 18:39:45.939736 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:39:45.939711 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcd8vx"] Jul 29 18:39:45.939889 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:39:45.939827 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcd8vx" Jul 29 18:39:45.942219 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:39:45.942194 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 29 18:39:45.943103 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:39:45.943076 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 29 18:39:45.943216 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:39:45.943199 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-dl8pd\"" Jul 29 18:39:46.077289 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:39:46.077257 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8krpf\" (UniqueName: \"kubernetes.io/projected/ad3eb59d-7201-4362-bb6f-4c05c5acc327-kube-api-access-8krpf\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcd8vx\" (UID: \"ad3eb59d-7201-4362-bb6f-4c05c5acc327\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcd8vx" Jul 29 18:39:46.077494 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:39:46.077309 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ad3eb59d-7201-4362-bb6f-4c05c5acc327-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcd8vx\" (UID: \"ad3eb59d-7201-4362-bb6f-4c05c5acc327\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcd8vx" Jul 29 18:39:46.077494 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:39:46.077327 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ad3eb59d-7201-4362-bb6f-4c05c5acc327-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcd8vx\" (UID: \"ad3eb59d-7201-4362-bb6f-4c05c5acc327\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcd8vx" Jul 29 18:39:46.177858 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:39:46.177767 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ad3eb59d-7201-4362-bb6f-4c05c5acc327-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcd8vx\" (UID: \"ad3eb59d-7201-4362-bb6f-4c05c5acc327\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcd8vx" Jul 29 18:39:46.178019 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:39:46.177869 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8krpf\" (UniqueName: \"kubernetes.io/projected/ad3eb59d-7201-4362-bb6f-4c05c5acc327-kube-api-access-8krpf\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcd8vx\" (UID: \"ad3eb59d-7201-4362-bb6f-4c05c5acc327\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcd8vx" Jul 29 18:39:46.178019 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:39:46.177934 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ad3eb59d-7201-4362-bb6f-4c05c5acc327-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcd8vx\" (UID: \"ad3eb59d-7201-4362-bb6f-4c05c5acc327\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcd8vx" Jul 29 18:39:46.178241 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:39:46.178216 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ad3eb59d-7201-4362-bb6f-4c05c5acc327-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcd8vx\" (UID: \"ad3eb59d-7201-4362-bb6f-4c05c5acc327\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcd8vx" Jul 29 18:39:46.178285 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:39:46.178231 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ad3eb59d-7201-4362-bb6f-4c05c5acc327-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcd8vx\" (UID: \"ad3eb59d-7201-4362-bb6f-4c05c5acc327\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcd8vx" Jul 29 18:39:46.187718 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:39:46.187692 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8krpf\" (UniqueName: \"kubernetes.io/projected/ad3eb59d-7201-4362-bb6f-4c05c5acc327-kube-api-access-8krpf\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcd8vx\" (UID: \"ad3eb59d-7201-4362-bb6f-4c05c5acc327\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcd8vx" Jul 29 18:39:46.249669 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:39:46.249645 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcd8vx" Jul 29 18:39:46.401951 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:39:46.401782 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcd8vx"] Jul 29 18:39:46.404504 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:39:46.404479 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad3eb59d_7201_4362_bb6f_4c05c5acc327.slice/crio-2050456f3032f65a47f895515941ba2b4743e671a4114f23de47ab405987ba07 WatchSource:0}: Error finding container 2050456f3032f65a47f895515941ba2b4743e671a4114f23de47ab405987ba07: Status 404 returned error can't find the container with id 2050456f3032f65a47f895515941ba2b4743e671a4114f23de47ab405987ba07 Jul 29 18:39:46.771591 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:39:46.771560 2560 generic.go:356] "Generic (PLEG): container finished" podID="ad3eb59d-7201-4362-bb6f-4c05c5acc327" containerID="fbd0842899a2b4d20e0d2b01a57636f61f475c2afe3ac84e9061a6a0071ecb2b" exitCode=0 Jul 29 18:39:46.771759 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:39:46.771618 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcd8vx" event={"ID":"ad3eb59d-7201-4362-bb6f-4c05c5acc327","Type":"ContainerDied","Data":"fbd0842899a2b4d20e0d2b01a57636f61f475c2afe3ac84e9061a6a0071ecb2b"} Jul 29 18:39:46.771759 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:39:46.771698 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcd8vx" event={"ID":"ad3eb59d-7201-4362-bb6f-4c05c5acc327","Type":"ContainerStarted","Data":"2050456f3032f65a47f895515941ba2b4743e671a4114f23de47ab405987ba07"} Jul 29 18:39:47.212761 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:39:47.212733 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:39:47.611913 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:39:47.611883 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:39:47.813474 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:39:47.813441 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-gd2tf"] Jul 29 18:39:47.816005 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:39:47.815985 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-gd2tf" Jul 29 18:39:47.818214 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:39:47.818191 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 29 18:39:47.818998 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:39:47.818979 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-z87kn\"" Jul 29 18:39:47.819093 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:39:47.819046 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 29 18:39:47.826145 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:39:47.826114 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-gd2tf"] Jul 29 18:39:47.892308 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:39:47.892229 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8b632efb-a94a-4a58-bea2-2caca8cd4c7b-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-gd2tf\" (UID: \"8b632efb-a94a-4a58-bea2-2caca8cd4c7b\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-gd2tf" Jul 29 18:39:47.892445 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:39:47.892375 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mrthg\" (UniqueName: \"kubernetes.io/projected/8b632efb-a94a-4a58-bea2-2caca8cd4c7b-kube-api-access-mrthg\") pod \"cert-manager-cainjector-559cdd8c87-gd2tf\" (UID: \"8b632efb-a94a-4a58-bea2-2caca8cd4c7b\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-gd2tf" Jul 29 18:39:47.993149 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:39:47.993115 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mrthg\" (UniqueName: \"kubernetes.io/projected/8b632efb-a94a-4a58-bea2-2caca8cd4c7b-kube-api-access-mrthg\") pod \"cert-manager-cainjector-559cdd8c87-gd2tf\" (UID: \"8b632efb-a94a-4a58-bea2-2caca8cd4c7b\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-gd2tf" Jul 29 18:39:47.993309 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:39:47.993178 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8b632efb-a94a-4a58-bea2-2caca8cd4c7b-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-gd2tf\" (UID: \"8b632efb-a94a-4a58-bea2-2caca8cd4c7b\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-gd2tf" Jul 29 18:39:48.001854 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:39:48.001824 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8b632efb-a94a-4a58-bea2-2caca8cd4c7b-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-gd2tf\" (UID: \"8b632efb-a94a-4a58-bea2-2caca8cd4c7b\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-gd2tf" Jul 29 18:39:48.002012 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:39:48.001991 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mrthg\" (UniqueName: \"kubernetes.io/projected/8b632efb-a94a-4a58-bea2-2caca8cd4c7b-kube-api-access-mrthg\") pod \"cert-manager-cainjector-559cdd8c87-gd2tf\" (UID: \"8b632efb-a94a-4a58-bea2-2caca8cd4c7b\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-gd2tf" Jul 29 18:39:48.133912 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:39:48.133878 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-gd2tf" Jul 29 18:39:48.286159 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:39:48.285997 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-gd2tf"] Jul 29 18:39:48.981466 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:39:48.981425 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8b632efb_a94a_4a58_bea2_2caca8cd4c7b.slice/crio-3f0a986b17ed559701fd8b0433c7a0dec10857c64173edf9fff4e417d617a6de WatchSource:0}: Error finding container 3f0a986b17ed559701fd8b0433c7a0dec10857c64173edf9fff4e417d617a6de: Status 404 returned error can't find the container with id 3f0a986b17ed559701fd8b0433c7a0dec10857c64173edf9fff4e417d617a6de Jul 29 18:39:49.788562 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:39:49.788518 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-gd2tf" event={"ID":"8b632efb-a94a-4a58-bea2-2caca8cd4c7b","Type":"ContainerStarted","Data":"3f0a986b17ed559701fd8b0433c7a0dec10857c64173edf9fff4e417d617a6de"} Jul 29 18:39:49.790695 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:39:49.790663 2560 generic.go:356] "Generic (PLEG): container finished" podID="ad3eb59d-7201-4362-bb6f-4c05c5acc327" containerID="a56a92b1011ecea26976835e12e639a6b62cddd0843095ff8ec12896a52b401b" exitCode=0 Jul 29 18:39:49.790839 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:39:49.790702 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcd8vx" event={"ID":"ad3eb59d-7201-4362-bb6f-4c05c5acc327","Type":"ContainerDied","Data":"a56a92b1011ecea26976835e12e639a6b62cddd0843095ff8ec12896a52b401b"} Jul 29 18:39:50.056629 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:39:50.056546 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:39:50.123184 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:39:50.123152 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:39:50.798795 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:39:50.798758 2560 generic.go:356] "Generic (PLEG): container finished" podID="ad3eb59d-7201-4362-bb6f-4c05c5acc327" containerID="61c7d54d685f46fcee685d2bb7a3ba712c959d404386abb48d36fb821a4cf9f9" exitCode=0 Jul 29 18:39:50.799290 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:39:50.798816 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcd8vx" event={"ID":"ad3eb59d-7201-4362-bb6f-4c05c5acc327","Type":"ContainerDied","Data":"61c7d54d685f46fcee685d2bb7a3ba712c959d404386abb48d36fb821a4cf9f9"} Jul 29 18:39:51.804795 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:39:51.804740 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-gd2tf" event={"ID":"8b632efb-a94a-4a58-bea2-2caca8cd4c7b","Type":"ContainerStarted","Data":"ef7b5c3b82a68b55e411844eefeea51b8964c380b181fe92800b053d2566d179"} Jul 29 18:39:51.823497 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:39:51.823436 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-gd2tf" podStartSLOduration=2.1347312880000002 podStartE2EDuration="4.82341754s" totalImagesPullingTime="2.688686252s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 18:39:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 18:39:51.821589145 +0000 UTC m=+327.047258033" watchObservedRunningTime="2026-07-29 18:39:51.82341754 +0000 UTC m=+327.049086413" Jul 29 18:39:51.941695 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:39:51.941672 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcd8vx" Jul 29 18:39:52.031525 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:39:52.031483 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ad3eb59d-7201-4362-bb6f-4c05c5acc327-util\") pod \"ad3eb59d-7201-4362-bb6f-4c05c5acc327\" (UID: \"ad3eb59d-7201-4362-bb6f-4c05c5acc327\") " Jul 29 18:39:52.031676 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:39:52.031573 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8krpf\" (UniqueName: \"kubernetes.io/projected/ad3eb59d-7201-4362-bb6f-4c05c5acc327-kube-api-access-8krpf\") pod \"ad3eb59d-7201-4362-bb6f-4c05c5acc327\" (UID: \"ad3eb59d-7201-4362-bb6f-4c05c5acc327\") " Jul 29 18:39:52.031676 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:39:52.031669 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ad3eb59d-7201-4362-bb6f-4c05c5acc327-bundle\") pod \"ad3eb59d-7201-4362-bb6f-4c05c5acc327\" (UID: \"ad3eb59d-7201-4362-bb6f-4c05c5acc327\") " Jul 29 18:39:52.032004 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:39:52.031981 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad3eb59d-7201-4362-bb6f-4c05c5acc327-bundle" (OuterVolumeSpecName: "bundle") pod "ad3eb59d-7201-4362-bb6f-4c05c5acc327" (UID: "ad3eb59d-7201-4362-bb6f-4c05c5acc327"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 18:39:52.033802 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:39:52.033776 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad3eb59d-7201-4362-bb6f-4c05c5acc327-kube-api-access-8krpf" (OuterVolumeSpecName: "kube-api-access-8krpf") pod "ad3eb59d-7201-4362-bb6f-4c05c5acc327" (UID: "ad3eb59d-7201-4362-bb6f-4c05c5acc327"). InnerVolumeSpecName "kube-api-access-8krpf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 18:39:52.094082 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:39:52.094047 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad3eb59d-7201-4362-bb6f-4c05c5acc327-util" (OuterVolumeSpecName: "util") pod "ad3eb59d-7201-4362-bb6f-4c05c5acc327" (UID: "ad3eb59d-7201-4362-bb6f-4c05c5acc327"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 18:39:52.132503 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:39:52.132469 2560 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ad3eb59d-7201-4362-bb6f-4c05c5acc327-bundle\") on node \"ip-10-0-140-205.ec2.internal\" DevicePath \"\"" Jul 29 18:39:52.132503 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:39:52.132506 2560 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ad3eb59d-7201-4362-bb6f-4c05c5acc327-util\") on node \"ip-10-0-140-205.ec2.internal\" DevicePath \"\"" Jul 29 18:39:52.132681 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:39:52.132521 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8krpf\" (UniqueName: \"kubernetes.io/projected/ad3eb59d-7201-4362-bb6f-4c05c5acc327-kube-api-access-8krpf\") on node \"ip-10-0-140-205.ec2.internal\" DevicePath \"\"" Jul 29 18:39:52.810454 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:39:52.810428 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcd8vx" Jul 29 18:39:52.810928 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:39:52.810452 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcd8vx" event={"ID":"ad3eb59d-7201-4362-bb6f-4c05c5acc327","Type":"ContainerDied","Data":"2050456f3032f65a47f895515941ba2b4743e671a4114f23de47ab405987ba07"} Jul 29 18:39:52.810928 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:39:52.810499 2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2050456f3032f65a47f895515941ba2b4743e671a4114f23de47ab405987ba07" Jul 29 18:39:53.374288 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:39:53.374255 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-nm8nb"] Jul 29 18:39:53.374565 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:39:53.374552 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad3eb59d-7201-4362-bb6f-4c05c5acc327" containerName="extract" Jul 29 18:39:53.374614 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:39:53.374566 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad3eb59d-7201-4362-bb6f-4c05c5acc327" containerName="extract" Jul 29 18:39:53.374614 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:39:53.374579 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad3eb59d-7201-4362-bb6f-4c05c5acc327" containerName="util" Jul 29 18:39:53.374614 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:39:53.374585 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad3eb59d-7201-4362-bb6f-4c05c5acc327" containerName="util" Jul 29 18:39:53.374614 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:39:53.374600 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad3eb59d-7201-4362-bb6f-4c05c5acc327" containerName="pull" Jul 29 18:39:53.374614 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:39:53.374605 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad3eb59d-7201-4362-bb6f-4c05c5acc327" containerName="pull" Jul 29 18:39:53.374754 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:39:53.374648 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="ad3eb59d-7201-4362-bb6f-4c05c5acc327" containerName="extract" Jul 29 18:39:53.376470 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:39:53.376455 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-nm8nb" Jul 29 18:39:53.378840 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:39:53.378819 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-5shbc\"" Jul 29 18:39:53.385391 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:39:53.385370 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-nm8nb"] Jul 29 18:39:53.441871 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:39:53.441841 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x7fg2\" (UniqueName: \"kubernetes.io/projected/805fc454-3b65-40fc-b1a5-d3cf804df29c-kube-api-access-x7fg2\") pod \"cert-manager-webhook-6486ffd997-nm8nb\" (UID: \"805fc454-3b65-40fc-b1a5-d3cf804df29c\") " pod="cert-manager/cert-manager-webhook-6486ffd997-nm8nb" Jul 29 18:39:53.441990 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:39:53.441901 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/805fc454-3b65-40fc-b1a5-d3cf804df29c-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-nm8nb\" (UID: \"805fc454-3b65-40fc-b1a5-d3cf804df29c\") " pod="cert-manager/cert-manager-webhook-6486ffd997-nm8nb" Jul 29 18:39:53.542744 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:39:53.542710 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/805fc454-3b65-40fc-b1a5-d3cf804df29c-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-nm8nb\" (UID: \"805fc454-3b65-40fc-b1a5-d3cf804df29c\") " pod="cert-manager/cert-manager-webhook-6486ffd997-nm8nb" Jul 29 18:39:53.542881 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:39:53.542792 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7fg2\" (UniqueName: \"kubernetes.io/projected/805fc454-3b65-40fc-b1a5-d3cf804df29c-kube-api-access-x7fg2\") pod \"cert-manager-webhook-6486ffd997-nm8nb\" (UID: \"805fc454-3b65-40fc-b1a5-d3cf804df29c\") " pod="cert-manager/cert-manager-webhook-6486ffd997-nm8nb" Jul 29 18:39:53.551097 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:39:53.551075 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/805fc454-3b65-40fc-b1a5-d3cf804df29c-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-nm8nb\" (UID: \"805fc454-3b65-40fc-b1a5-d3cf804df29c\") " pod="cert-manager/cert-manager-webhook-6486ffd997-nm8nb" Jul 29 18:39:53.551190 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:39:53.551154 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7fg2\" (UniqueName: \"kubernetes.io/projected/805fc454-3b65-40fc-b1a5-d3cf804df29c-kube-api-access-x7fg2\") pod \"cert-manager-webhook-6486ffd997-nm8nb\" (UID: \"805fc454-3b65-40fc-b1a5-d3cf804df29c\") " pod="cert-manager/cert-manager-webhook-6486ffd997-nm8nb" Jul 29 18:39:53.685602 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:39:53.685525 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-nm8nb" Jul 29 18:39:53.811814 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:39:53.811778 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-nm8nb"] Jul 29 18:39:53.817172 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:39:53.817146 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod805fc454_3b65_40fc_b1a5_d3cf804df29c.slice/crio-680d5a3cee1b544155817fe1b877514690428231e4e8fb310491611e886c7cd7 WatchSource:0}: Error finding container 680d5a3cee1b544155817fe1b877514690428231e4e8fb310491611e886c7cd7: Status 404 returned error can't find the container with id 680d5a3cee1b544155817fe1b877514690428231e4e8fb310491611e886c7cd7 Jul 29 18:39:54.067307 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:39:54.067278 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:39:54.126717 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:39:54.126681 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:39:54.824777 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:39:54.824738 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-nm8nb" event={"ID":"805fc454-3b65-40fc-b1a5-d3cf804df29c","Type":"ContainerStarted","Data":"21fb20415db9e7f4006613cde7c099ab0df5d4a2a567549c2d372bebeb3f6e72"} Jul 29 18:39:54.824777 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:39:54.824785 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-nm8nb" event={"ID":"805fc454-3b65-40fc-b1a5-d3cf804df29c","Type":"ContainerStarted","Data":"680d5a3cee1b544155817fe1b877514690428231e4e8fb310491611e886c7cd7"} Jul 29 18:39:54.825184 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:39:54.824865 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-nm8nb" Jul 29 18:39:54.840733 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:39:54.840687 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-nm8nb" podStartSLOduration=1.8406719919999999 podStartE2EDuration="1.840671992s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 18:39:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 18:39:54.839553709 +0000 UTC m=+330.065222581" watchObservedRunningTime="2026-07-29 18:39:54.840671992 +0000 UTC m=+330.066340863" Jul 29 18:39:55.010380 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:39:55.010323 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:39:56.452624 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:39:56.452581 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-ng5tr"] Jul 29 18:39:56.455073 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:39:56.455051 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-ng5tr" Jul 29 18:39:56.457539 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:39:56.457516 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-csl5t\"" Jul 29 18:39:56.468466 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:39:56.468437 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-ng5tr"] Jul 29 18:39:56.472938 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:39:56.472915 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:39:56.571335 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:39:56.571304 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/54bc09fa-7680-470c-83b9-fbe70f4a9db5-bound-sa-token\") pod \"cert-manager-55b4d954d5-ng5tr\" (UID: \"54bc09fa-7680-470c-83b9-fbe70f4a9db5\") " pod="cert-manager/cert-manager-55b4d954d5-ng5tr" Jul 29 18:39:56.571335 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:39:56.571321 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:39:56.571549 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:39:56.571408 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lcvqj\" (UniqueName: \"kubernetes.io/projected/54bc09fa-7680-470c-83b9-fbe70f4a9db5-kube-api-access-lcvqj\") pod \"cert-manager-55b4d954d5-ng5tr\" (UID: \"54bc09fa-7680-470c-83b9-fbe70f4a9db5\") " pod="cert-manager/cert-manager-55b4d954d5-ng5tr" Jul 29 18:39:56.672105 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:39:56.672071 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lcvqj\" (UniqueName: \"kubernetes.io/projected/54bc09fa-7680-470c-83b9-fbe70f4a9db5-kube-api-access-lcvqj\") pod \"cert-manager-55b4d954d5-ng5tr\" (UID: \"54bc09fa-7680-470c-83b9-fbe70f4a9db5\") " pod="cert-manager/cert-manager-55b4d954d5-ng5tr" Jul 29 18:39:56.672282 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:39:56.672118 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/54bc09fa-7680-470c-83b9-fbe70f4a9db5-bound-sa-token\") pod \"cert-manager-55b4d954d5-ng5tr\" (UID: \"54bc09fa-7680-470c-83b9-fbe70f4a9db5\") " pod="cert-manager/cert-manager-55b4d954d5-ng5tr" Jul 29 18:39:56.680484 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:39:56.680453 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/54bc09fa-7680-470c-83b9-fbe70f4a9db5-bound-sa-token\") pod \"cert-manager-55b4d954d5-ng5tr\" (UID: \"54bc09fa-7680-470c-83b9-fbe70f4a9db5\") " pod="cert-manager/cert-manager-55b4d954d5-ng5tr" Jul 29 18:39:56.680620 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:39:56.680593 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lcvqj\" (UniqueName: \"kubernetes.io/projected/54bc09fa-7680-470c-83b9-fbe70f4a9db5-kube-api-access-lcvqj\") pod \"cert-manager-55b4d954d5-ng5tr\" (UID: \"54bc09fa-7680-470c-83b9-fbe70f4a9db5\") " pod="cert-manager/cert-manager-55b4d954d5-ng5tr" Jul 29 18:39:56.764825 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:39:56.764806 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-ng5tr" Jul 29 18:39:56.892414 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:39:56.892383 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-ng5tr"] Jul 29 18:39:56.895616 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:39:56.895586 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod54bc09fa_7680_470c_83b9_fbe70f4a9db5.slice/crio-1e59da7ea0ce4ac509b47d9c5dfeaadc790e3560abc3cb341c8efa80ecb1786a WatchSource:0}: Error finding container 1e59da7ea0ce4ac509b47d9c5dfeaadc790e3560abc3cb341c8efa80ecb1786a: Status 404 returned error can't find the container with id 1e59da7ea0ce4ac509b47d9c5dfeaadc790e3560abc3cb341c8efa80ecb1786a Jul 29 18:39:57.838031 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:39:57.837998 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-ng5tr" event={"ID":"54bc09fa-7680-470c-83b9-fbe70f4a9db5","Type":"ContainerStarted","Data":"3e180473204b86868f3a412f748e2edc321e4198ad58f6e22b5eb99bd8855e03"} Jul 29 18:39:57.838031 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:39:57.838032 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-ng5tr" event={"ID":"54bc09fa-7680-470c-83b9-fbe70f4a9db5","Type":"ContainerStarted","Data":"1e59da7ea0ce4ac509b47d9c5dfeaadc790e3560abc3cb341c8efa80ecb1786a"} Jul 29 18:39:57.856853 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:39:57.856803 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-ng5tr" podStartSLOduration=1.85678225 podStartE2EDuration="1.85678225s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 18:39:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 18:39:57.855543836 +0000 UTC m=+333.081212723" watchObservedRunningTime="2026-07-29 18:39:57.85678225 +0000 UTC m=+333.082451125" Jul 29 18:40:00.831318 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:40:00.831286 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-nm8nb" Jul 29 18:40:03.004368 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:40:03.004313 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:40:03.109492 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:40:03.109452 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:40:05.041702 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:40:05.041663 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af2p44l"] Jul 29 18:40:05.044599 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:40:05.044577 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af2p44l" Jul 29 18:40:05.047020 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:40:05.046997 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 29 18:40:05.047130 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:40:05.047119 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 29 18:40:05.047959 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:40:05.047930 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-dl8pd\"" Jul 29 18:40:05.054430 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:40:05.054408 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af2p44l"] Jul 29 18:40:05.140471 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:40:05.140435 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b7cf3976-06af-451a-8bc4-95b1b50d04d3-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af2p44l\" (UID: \"b7cf3976-06af-451a-8bc4-95b1b50d04d3\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af2p44l" Jul 29 18:40:05.140633 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:40:05.140488 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b7cf3976-06af-451a-8bc4-95b1b50d04d3-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af2p44l\" (UID: \"b7cf3976-06af-451a-8bc4-95b1b50d04d3\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af2p44l" Jul 29 18:40:05.140633 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:40:05.140514 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-knhx5\" (UniqueName: \"kubernetes.io/projected/b7cf3976-06af-451a-8bc4-95b1b50d04d3-kube-api-access-knhx5\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af2p44l\" (UID: \"b7cf3976-06af-451a-8bc4-95b1b50d04d3\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af2p44l" Jul 29 18:40:05.241080 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:40:05.241049 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b7cf3976-06af-451a-8bc4-95b1b50d04d3-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af2p44l\" (UID: \"b7cf3976-06af-451a-8bc4-95b1b50d04d3\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af2p44l" Jul 29 18:40:05.241227 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:40:05.241091 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b7cf3976-06af-451a-8bc4-95b1b50d04d3-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af2p44l\" (UID: \"b7cf3976-06af-451a-8bc4-95b1b50d04d3\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af2p44l" Jul 29 18:40:05.241227 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:40:05.241110 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-knhx5\" (UniqueName: \"kubernetes.io/projected/b7cf3976-06af-451a-8bc4-95b1b50d04d3-kube-api-access-knhx5\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af2p44l\" (UID: \"b7cf3976-06af-451a-8bc4-95b1b50d04d3\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af2p44l" Jul 29 18:40:05.241467 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:40:05.241450 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b7cf3976-06af-451a-8bc4-95b1b50d04d3-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af2p44l\" (UID: \"b7cf3976-06af-451a-8bc4-95b1b50d04d3\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af2p44l" Jul 29 18:40:05.241522 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:40:05.241506 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b7cf3976-06af-451a-8bc4-95b1b50d04d3-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af2p44l\" (UID: \"b7cf3976-06af-451a-8bc4-95b1b50d04d3\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af2p44l" Jul 29 18:40:05.253915 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:40:05.253889 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-knhx5\" (UniqueName: \"kubernetes.io/projected/b7cf3976-06af-451a-8bc4-95b1b50d04d3-kube-api-access-knhx5\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af2p44l\" (UID: \"b7cf3976-06af-451a-8bc4-95b1b50d04d3\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af2p44l" Jul 29 18:40:05.354966 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:40:05.354910 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af2p44l" Jul 29 18:40:05.374633 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:40:05.374613 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:40:05.485748 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:40:05.485717 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af2p44l"] Jul 29 18:40:05.488087 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:40:05.488059 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7cf3976_06af_451a_8bc4_95b1b50d04d3.slice/crio-e1472a30934bc9d00bdbc9b154942a09442c81c14936319c399c69c7a3b60ac8 WatchSource:0}: Error finding container e1472a30934bc9d00bdbc9b154942a09442c81c14936319c399c69c7a3b60ac8: Status 404 returned error can't find the container with id e1472a30934bc9d00bdbc9b154942a09442c81c14936319c399c69c7a3b60ac8 Jul 29 18:40:05.873606 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:40:05.873567 2560 generic.go:356] "Generic (PLEG): container finished" podID="b7cf3976-06af-451a-8bc4-95b1b50d04d3" containerID="b7bd0f6d43536dfa32fac7d640df388a267512cadf087f9beb6d806477f13298" exitCode=0 Jul 29 18:40:05.873777 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:40:05.873664 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af2p44l" event={"ID":"b7cf3976-06af-451a-8bc4-95b1b50d04d3","Type":"ContainerDied","Data":"b7bd0f6d43536dfa32fac7d640df388a267512cadf087f9beb6d806477f13298"} Jul 29 18:40:05.873777 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:40:05.873706 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af2p44l" event={"ID":"b7cf3976-06af-451a-8bc4-95b1b50d04d3","Type":"ContainerStarted","Data":"e1472a30934bc9d00bdbc9b154942a09442c81c14936319c399c69c7a3b60ac8"} Jul 29 18:40:06.880026 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:40:06.879996 2560 generic.go:356] "Generic (PLEG): container finished" podID="b7cf3976-06af-451a-8bc4-95b1b50d04d3" containerID="43b62aa88f6e9cc4f7f0faf772bc1961635338ed6042c72fba28ee4a914fe13b" exitCode=0 Jul 29 18:40:06.880342 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:40:06.880091 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af2p44l" event={"ID":"b7cf3976-06af-451a-8bc4-95b1b50d04d3","Type":"ContainerDied","Data":"43b62aa88f6e9cc4f7f0faf772bc1961635338ed6042c72fba28ee4a914fe13b"} Jul 29 18:40:07.886663 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:40:07.886627 2560 generic.go:356] "Generic (PLEG): container finished" podID="b7cf3976-06af-451a-8bc4-95b1b50d04d3" containerID="922409b01ee3a3537abd087d5d80d15cf38d2e33197b25adcb20e1a1d6e3e07e" exitCode=0 Jul 29 18:40:07.887034 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:40:07.886712 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af2p44l" event={"ID":"b7cf3976-06af-451a-8bc4-95b1b50d04d3","Type":"ContainerDied","Data":"922409b01ee3a3537abd087d5d80d15cf38d2e33197b25adcb20e1a1d6e3e07e"} Jul 29 18:40:09.020499 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:40:09.020477 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af2p44l" Jul 29 18:40:09.172994 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:40:09.172917 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b7cf3976-06af-451a-8bc4-95b1b50d04d3-bundle\") pod \"b7cf3976-06af-451a-8bc4-95b1b50d04d3\" (UID: \"b7cf3976-06af-451a-8bc4-95b1b50d04d3\") " Jul 29 18:40:09.172994 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:40:09.172950 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b7cf3976-06af-451a-8bc4-95b1b50d04d3-util\") pod \"b7cf3976-06af-451a-8bc4-95b1b50d04d3\" (UID: \"b7cf3976-06af-451a-8bc4-95b1b50d04d3\") " Jul 29 18:40:09.173165 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:40:09.173016 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-knhx5\" (UniqueName: \"kubernetes.io/projected/b7cf3976-06af-451a-8bc4-95b1b50d04d3-kube-api-access-knhx5\") pod \"b7cf3976-06af-451a-8bc4-95b1b50d04d3\" (UID: \"b7cf3976-06af-451a-8bc4-95b1b50d04d3\") " Jul 29 18:40:09.173684 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:40:09.173647 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7cf3976-06af-451a-8bc4-95b1b50d04d3-bundle" (OuterVolumeSpecName: "bundle") pod "b7cf3976-06af-451a-8bc4-95b1b50d04d3" (UID: "b7cf3976-06af-451a-8bc4-95b1b50d04d3"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 18:40:09.175218 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:40:09.175183 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7cf3976-06af-451a-8bc4-95b1b50d04d3-kube-api-access-knhx5" (OuterVolumeSpecName: "kube-api-access-knhx5") pod "b7cf3976-06af-451a-8bc4-95b1b50d04d3" (UID: "b7cf3976-06af-451a-8bc4-95b1b50d04d3"). InnerVolumeSpecName "kube-api-access-knhx5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 18:40:09.178564 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:40:09.178543 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7cf3976-06af-451a-8bc4-95b1b50d04d3-util" (OuterVolumeSpecName: "util") pod "b7cf3976-06af-451a-8bc4-95b1b50d04d3" (UID: "b7cf3976-06af-451a-8bc4-95b1b50d04d3"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 18:40:09.274229 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:40:09.274197 2560 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b7cf3976-06af-451a-8bc4-95b1b50d04d3-bundle\") on node \"ip-10-0-140-205.ec2.internal\" DevicePath \"\"" Jul 29 18:40:09.274229 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:40:09.274224 2560 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b7cf3976-06af-451a-8bc4-95b1b50d04d3-util\") on node \"ip-10-0-140-205.ec2.internal\" DevicePath \"\"" Jul 29 18:40:09.274415 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:40:09.274236 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-knhx5\" (UniqueName: \"kubernetes.io/projected/b7cf3976-06af-451a-8bc4-95b1b50d04d3-kube-api-access-knhx5\") on node \"ip-10-0-140-205.ec2.internal\" DevicePath \"\"" Jul 29 18:40:09.897625 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:40:09.897595 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af2p44l" event={"ID":"b7cf3976-06af-451a-8bc4-95b1b50d04d3","Type":"ContainerDied","Data":"e1472a30934bc9d00bdbc9b154942a09442c81c14936319c399c69c7a3b60ac8"} Jul 29 18:40:09.897625 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:40:09.897621 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af2p44l" Jul 29 18:40:09.897952 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:40:09.897637 2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e1472a30934bc9d00bdbc9b154942a09442c81c14936319c399c69c7a3b60ac8" Jul 29 18:40:10.488963 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:40:10.488933 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:40:10.546478 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:40:10.546423 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:40:14.076506 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:40:14.076470 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:40:14.136111 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:40:14.136067 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:40:16.888400 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:40:16.888284 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:40:16.995112 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:40:16.995080 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:40:17.534760 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:40:17.534711 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:40:17.831636 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:40:17.831525 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:40:17.947879 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:40:17.947839 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:40:18.002097 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:40:18.002073 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-76c6f8f9b-9dbbf"] Jul 29 18:40:18.002432 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:40:18.002418 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b7cf3976-06af-451a-8bc4-95b1b50d04d3" containerName="util" Jul 29 18:40:18.002484 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:40:18.002434 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="b7cf3976-06af-451a-8bc4-95b1b50d04d3" containerName="util" Jul 29 18:40:18.002484 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:40:18.002446 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b7cf3976-06af-451a-8bc4-95b1b50d04d3" containerName="pull" Jul 29 18:40:18.002484 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:40:18.002451 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="b7cf3976-06af-451a-8bc4-95b1b50d04d3" containerName="pull" Jul 29 18:40:18.002484 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:40:18.002463 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b7cf3976-06af-451a-8bc4-95b1b50d04d3" containerName="extract" Jul 29 18:40:18.002484 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:40:18.002468 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="b7cf3976-06af-451a-8bc4-95b1b50d04d3" containerName="extract" Jul 29 18:40:18.002617 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:40:18.002527 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="b7cf3976-06af-451a-8bc4-95b1b50d04d3" containerName="extract" Jul 29 18:40:18.005635 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:40:18.005610 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-76c6f8f9b-9dbbf" Jul 29 18:40:18.009800 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:40:18.009780 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 29 18:40:18.009891 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:40:18.009812 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-b6vgb\"" Jul 29 18:40:18.010229 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:40:18.010209 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 29 18:40:18.010338 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:40:18.010324 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 29 18:40:18.010457 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:40:18.010439 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 29 18:40:18.022794 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:40:18.022773 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-76c6f8f9b-9dbbf"] Jul 29 18:40:18.041621 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:40:18.041590 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/a07c02c3-7bf1-45e5-b28b-24f29d48494a-manifests\") pod \"opendatahub-operator-controller-manager-76c6f8f9b-9dbbf\" (UID: \"a07c02c3-7bf1-45e5-b28b-24f29d48494a\") " pod="opendatahub/opendatahub-operator-controller-manager-76c6f8f9b-9dbbf" Jul 29 18:40:18.041714 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:40:18.041631 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a07c02c3-7bf1-45e5-b28b-24f29d48494a-webhook-cert\") pod \"opendatahub-operator-controller-manager-76c6f8f9b-9dbbf\" (UID: \"a07c02c3-7bf1-45e5-b28b-24f29d48494a\") " pod="opendatahub/opendatahub-operator-controller-manager-76c6f8f9b-9dbbf" Jul 29 18:40:18.041714 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:40:18.041682 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9lkmd\" (UniqueName: \"kubernetes.io/projected/a07c02c3-7bf1-45e5-b28b-24f29d48494a-kube-api-access-9lkmd\") pod \"opendatahub-operator-controller-manager-76c6f8f9b-9dbbf\" (UID: \"a07c02c3-7bf1-45e5-b28b-24f29d48494a\") " pod="opendatahub/opendatahub-operator-controller-manager-76c6f8f9b-9dbbf" Jul 29 18:40:18.041714 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:40:18.041709 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a07c02c3-7bf1-45e5-b28b-24f29d48494a-tmp\") pod \"opendatahub-operator-controller-manager-76c6f8f9b-9dbbf\" (UID: \"a07c02c3-7bf1-45e5-b28b-24f29d48494a\") " pod="opendatahub/opendatahub-operator-controller-manager-76c6f8f9b-9dbbf" Jul 29 18:40:18.041849 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:40:18.041730 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/a07c02c3-7bf1-45e5-b28b-24f29d48494a-apiservice-cert\") pod \"opendatahub-operator-controller-manager-76c6f8f9b-9dbbf\" (UID: \"a07c02c3-7bf1-45e5-b28b-24f29d48494a\") " pod="opendatahub/opendatahub-operator-controller-manager-76c6f8f9b-9dbbf" Jul 29 18:40:18.142451 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:40:18.142395 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/a07c02c3-7bf1-45e5-b28b-24f29d48494a-manifests\") pod \"opendatahub-operator-controller-manager-76c6f8f9b-9dbbf\" (UID: \"a07c02c3-7bf1-45e5-b28b-24f29d48494a\") " pod="opendatahub/opendatahub-operator-controller-manager-76c6f8f9b-9dbbf" Jul 29 18:40:18.142451 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:40:18.142440 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a07c02c3-7bf1-45e5-b28b-24f29d48494a-webhook-cert\") pod \"opendatahub-operator-controller-manager-76c6f8f9b-9dbbf\" (UID: \"a07c02c3-7bf1-45e5-b28b-24f29d48494a\") " pod="opendatahub/opendatahub-operator-controller-manager-76c6f8f9b-9dbbf" Jul 29 18:40:18.142611 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:40:18.142484 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9lkmd\" (UniqueName: \"kubernetes.io/projected/a07c02c3-7bf1-45e5-b28b-24f29d48494a-kube-api-access-9lkmd\") pod \"opendatahub-operator-controller-manager-76c6f8f9b-9dbbf\" (UID: \"a07c02c3-7bf1-45e5-b28b-24f29d48494a\") " pod="opendatahub/opendatahub-operator-controller-manager-76c6f8f9b-9dbbf" Jul 29 18:40:18.142611 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:40:18.142507 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a07c02c3-7bf1-45e5-b28b-24f29d48494a-tmp\") pod \"opendatahub-operator-controller-manager-76c6f8f9b-9dbbf\" (UID: \"a07c02c3-7bf1-45e5-b28b-24f29d48494a\") " pod="opendatahub/opendatahub-operator-controller-manager-76c6f8f9b-9dbbf" Jul 29 18:40:18.142709 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:40:18.142641 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/a07c02c3-7bf1-45e5-b28b-24f29d48494a-apiservice-cert\") pod \"opendatahub-operator-controller-manager-76c6f8f9b-9dbbf\" (UID: \"a07c02c3-7bf1-45e5-b28b-24f29d48494a\") " pod="opendatahub/opendatahub-operator-controller-manager-76c6f8f9b-9dbbf" Jul 29 18:40:18.142709 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:40:18.142685 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/a07c02c3-7bf1-45e5-b28b-24f29d48494a-manifests\") pod \"opendatahub-operator-controller-manager-76c6f8f9b-9dbbf\" (UID: \"a07c02c3-7bf1-45e5-b28b-24f29d48494a\") " pod="opendatahub/opendatahub-operator-controller-manager-76c6f8f9b-9dbbf" Jul 29 18:40:18.144937 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:40:18.144916 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a07c02c3-7bf1-45e5-b28b-24f29d48494a-tmp\") pod \"opendatahub-operator-controller-manager-76c6f8f9b-9dbbf\" (UID: \"a07c02c3-7bf1-45e5-b28b-24f29d48494a\") " pod="opendatahub/opendatahub-operator-controller-manager-76c6f8f9b-9dbbf" Jul 29 18:40:18.145122 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:40:18.145100 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/a07c02c3-7bf1-45e5-b28b-24f29d48494a-apiservice-cert\") pod \"opendatahub-operator-controller-manager-76c6f8f9b-9dbbf\" (UID: \"a07c02c3-7bf1-45e5-b28b-24f29d48494a\") " pod="opendatahub/opendatahub-operator-controller-manager-76c6f8f9b-9dbbf" Jul 29 18:40:18.145251 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:40:18.145236 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a07c02c3-7bf1-45e5-b28b-24f29d48494a-webhook-cert\") pod \"opendatahub-operator-controller-manager-76c6f8f9b-9dbbf\" (UID: \"a07c02c3-7bf1-45e5-b28b-24f29d48494a\") " pod="opendatahub/opendatahub-operator-controller-manager-76c6f8f9b-9dbbf" Jul 29 18:40:18.154989 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:40:18.154969 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9lkmd\" (UniqueName: \"kubernetes.io/projected/a07c02c3-7bf1-45e5-b28b-24f29d48494a-kube-api-access-9lkmd\") pod \"opendatahub-operator-controller-manager-76c6f8f9b-9dbbf\" (UID: \"a07c02c3-7bf1-45e5-b28b-24f29d48494a\") " pod="opendatahub/opendatahub-operator-controller-manager-76c6f8f9b-9dbbf" Jul 29 18:40:18.246635 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:40:18.246610 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:40:18.315402 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:40:18.315380 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-76c6f8f9b-9dbbf" Jul 29 18:40:18.347550 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:40:18.347515 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:40:18.483325 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:40:18.483303 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-76c6f8f9b-9dbbf"] Jul 29 18:40:18.485561 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:40:18.485535 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda07c02c3_7bf1_45e5_b28b_24f29d48494a.slice/crio-9c97c37e155837d22e7617ad714c6f52e651f1c7a34a7a9dd2dbdb62e59d2b2d WatchSource:0}: Error finding container 9c97c37e155837d22e7617ad714c6f52e651f1c7a34a7a9dd2dbdb62e59d2b2d: Status 404 returned error can't find the container with id 9c97c37e155837d22e7617ad714c6f52e651f1c7a34a7a9dd2dbdb62e59d2b2d Jul 29 18:40:18.506707 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:40:18.506682 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:40:18.623704 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:40:18.623678 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:40:18.749323 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:40:18.749301 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:40:18.843957 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:40:18.843931 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:40:18.938412 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:40:18.938378 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-76c6f8f9b-9dbbf" event={"ID":"a07c02c3-7bf1-45e5-b28b-24f29d48494a","Type":"ContainerStarted","Data":"9c97c37e155837d22e7617ad714c6f52e651f1c7a34a7a9dd2dbdb62e59d2b2d"} Jul 29 18:40:21.955772 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:40:21.955740 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-76c6f8f9b-9dbbf" event={"ID":"a07c02c3-7bf1-45e5-b28b-24f29d48494a","Type":"ContainerStarted","Data":"c863fbd719b22455926d8f99451e4e7a7982fd8db1962633acd34be3c751de58"} Jul 29 18:40:22.967513 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:40:22.967482 2560 generic.go:356] "Generic (PLEG): container finished" podID="a07c02c3-7bf1-45e5-b28b-24f29d48494a" containerID="c863fbd719b22455926d8f99451e4e7a7982fd8db1962633acd34be3c751de58" exitCode=0 Jul 29 18:40:22.967901 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:40:22.967574 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-76c6f8f9b-9dbbf" event={"ID":"a07c02c3-7bf1-45e5-b28b-24f29d48494a","Type":"ContainerDied","Data":"c863fbd719b22455926d8f99451e4e7a7982fd8db1962633acd34be3c751de58"} Jul 29 18:40:23.616754 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:40:23.616711 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:40:23.766568 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:40:23.766532 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:40:23.974509 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:40:23.974471 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-76c6f8f9b-9dbbf" event={"ID":"a07c02c3-7bf1-45e5-b28b-24f29d48494a","Type":"ContainerStarted","Data":"c5822faf4047683c5745709f25782278ecdbb78f2ec6275193ec2507beacfb9f"} Jul 29 18:40:23.974855 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:40:23.974571 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-76c6f8f9b-9dbbf" Jul 29 18:40:24.007390 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:40:24.007326 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-76c6f8f9b-9dbbf" podStartSLOduration=2.226177958 podStartE2EDuration="7.007312614s" totalImagesPullingTime="3.756994941s" totalInitContainerRuntime="1.024139715s" isStatefulPod=false podCreationTimestamp="2026-07-29 18:40:17 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 18:40:24.00483053 +0000 UTC m=+359.230499440" watchObservedRunningTime="2026-07-29 18:40:24.007312614 +0000 UTC m=+359.232981486" Jul 29 18:40:34.981752 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:40:34.981722 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-76c6f8f9b-9dbbf" Jul 29 18:40:37.082235 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:40:37.082204 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:40:37.210975 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:40:37.210938 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:40:37.567577 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:40:37.567541 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:40:37.588219 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:40:37.588186 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-54cd75bc79-ns79p"] Jul 29 18:40:37.597737 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:40:37.597720 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-54cd75bc79-ns79p" Jul 29 18:40:37.600554 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:40:37.600530 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 29 18:40:37.600554 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:40:37.600552 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 29 18:40:37.601037 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:40:37.601018 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 29 18:40:37.601130 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:40:37.601022 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 29 18:40:37.608586 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:40:37.608566 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-54cd75bc79-ns79p"] Jul 29 18:40:37.700893 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:40:37.700852 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tp5pd\" (UniqueName: \"kubernetes.io/projected/2d9798e3-f217-4a46-ab4d-9f2bd22e6516-kube-api-access-tp5pd\") pod \"kube-auth-proxy-54cd75bc79-ns79p\" (UID: \"2d9798e3-f217-4a46-ab4d-9f2bd22e6516\") " pod="openshift-ingress/kube-auth-proxy-54cd75bc79-ns79p" Jul 29 18:40:37.701040 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:40:37.700902 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d9798e3-f217-4a46-ab4d-9f2bd22e6516-tls-certs\") pod \"kube-auth-proxy-54cd75bc79-ns79p\" (UID: \"2d9798e3-f217-4a46-ab4d-9f2bd22e6516\") " pod="openshift-ingress/kube-auth-proxy-54cd75bc79-ns79p" Jul 29 18:40:37.701040 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:40:37.700997 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2d9798e3-f217-4a46-ab4d-9f2bd22e6516-tmp\") pod \"kube-auth-proxy-54cd75bc79-ns79p\" (UID: \"2d9798e3-f217-4a46-ab4d-9f2bd22e6516\") " pod="openshift-ingress/kube-auth-proxy-54cd75bc79-ns79p" Jul 29 18:40:37.801445 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:40:37.801414 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tp5pd\" (UniqueName: \"kubernetes.io/projected/2d9798e3-f217-4a46-ab4d-9f2bd22e6516-kube-api-access-tp5pd\") pod \"kube-auth-proxy-54cd75bc79-ns79p\" (UID: \"2d9798e3-f217-4a46-ab4d-9f2bd22e6516\") " pod="openshift-ingress/kube-auth-proxy-54cd75bc79-ns79p" Jul 29 18:40:37.801608 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:40:37.801454 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d9798e3-f217-4a46-ab4d-9f2bd22e6516-tls-certs\") pod \"kube-auth-proxy-54cd75bc79-ns79p\" (UID: \"2d9798e3-f217-4a46-ab4d-9f2bd22e6516\") " pod="openshift-ingress/kube-auth-proxy-54cd75bc79-ns79p" Jul 29 18:40:37.801608 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:40:37.801496 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2d9798e3-f217-4a46-ab4d-9f2bd22e6516-tmp\") pod \"kube-auth-proxy-54cd75bc79-ns79p\" (UID: \"2d9798e3-f217-4a46-ab4d-9f2bd22e6516\") " pod="openshift-ingress/kube-auth-proxy-54cd75bc79-ns79p" Jul 29 18:40:37.801903 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:40:37.801882 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:40:37.803921 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:40:37.803898 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2d9798e3-f217-4a46-ab4d-9f2bd22e6516-tmp\") pod \"kube-auth-proxy-54cd75bc79-ns79p\" (UID: \"2d9798e3-f217-4a46-ab4d-9f2bd22e6516\") " pod="openshift-ingress/kube-auth-proxy-54cd75bc79-ns79p" Jul 29 18:40:37.804126 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:40:37.804108 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d9798e3-f217-4a46-ab4d-9f2bd22e6516-tls-certs\") pod \"kube-auth-proxy-54cd75bc79-ns79p\" (UID: \"2d9798e3-f217-4a46-ab4d-9f2bd22e6516\") " pod="openshift-ingress/kube-auth-proxy-54cd75bc79-ns79p" Jul 29 18:40:37.818186 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:40:37.818119 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-cc9zh"] Jul 29 18:40:37.821536 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:40:37.821517 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-cc9zh" Jul 29 18:40:37.823941 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:40:37.823918 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tp5pd\" (UniqueName: \"kubernetes.io/projected/2d9798e3-f217-4a46-ab4d-9f2bd22e6516-kube-api-access-tp5pd\") pod \"kube-auth-proxy-54cd75bc79-ns79p\" (UID: \"2d9798e3-f217-4a46-ab4d-9f2bd22e6516\") " pod="openshift-ingress/kube-auth-proxy-54cd75bc79-ns79p" Jul 29 18:40:37.825009 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:40:37.824991 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-rdm8x\"" Jul 29 18:40:37.825102 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:40:37.825087 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Jul 29 18:40:37.836630 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:40:37.836611 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-cc9zh"] Jul 29 18:40:37.902146 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:40:37.902117 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c677dc98-e3ee-497f-ae69-7ce442751820-cert\") pod \"odh-model-controller-7998c67dc4-cc9zh\" (UID: \"c677dc98-e3ee-497f-ae69-7ce442751820\") " pod="opendatahub/odh-model-controller-7998c67dc4-cc9zh" Jul 29 18:40:37.902313 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:40:37.902176 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8rds\" (UniqueName: \"kubernetes.io/projected/c677dc98-e3ee-497f-ae69-7ce442751820-kube-api-access-g8rds\") pod \"odh-model-controller-7998c67dc4-cc9zh\" (UID: \"c677dc98-e3ee-497f-ae69-7ce442751820\") " pod="opendatahub/odh-model-controller-7998c67dc4-cc9zh" Jul 29 18:40:37.907035 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:40:37.907009 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-54cd75bc79-ns79p" Jul 29 18:40:37.927060 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:40:37.927034 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:40:38.002913 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:40:38.002879 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8rds\" (UniqueName: \"kubernetes.io/projected/c677dc98-e3ee-497f-ae69-7ce442751820-kube-api-access-g8rds\") pod \"odh-model-controller-7998c67dc4-cc9zh\" (UID: \"c677dc98-e3ee-497f-ae69-7ce442751820\") " pod="opendatahub/odh-model-controller-7998c67dc4-cc9zh" Jul 29 18:40:38.003080 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:40:38.002982 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c677dc98-e3ee-497f-ae69-7ce442751820-cert\") pod \"odh-model-controller-7998c67dc4-cc9zh\" (UID: \"c677dc98-e3ee-497f-ae69-7ce442751820\") " pod="opendatahub/odh-model-controller-7998c67dc4-cc9zh" Jul 29 18:40:38.003172 ip-10-0-140-205 kubenswrapper[2560]: E0729 18:40:38.003147 2560 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found Jul 29 18:40:38.003244 ip-10-0-140-205 kubenswrapper[2560]: E0729 18:40:38.003231 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c677dc98-e3ee-497f-ae69-7ce442751820-cert podName:c677dc98-e3ee-497f-ae69-7ce442751820 nodeName:}" failed. No retries permitted until 2026-07-29 18:40:38.503211018 +0000 UTC m=+373.728879871 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/c677dc98-e3ee-497f-ae69-7ce442751820-cert") pod "odh-model-controller-7998c67dc4-cc9zh" (UID: "c677dc98-e3ee-497f-ae69-7ce442751820") : secret "odh-model-controller-webhook-cert" not found Jul 29 18:40:38.013033 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:40:38.013008 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8rds\" (UniqueName: \"kubernetes.io/projected/c677dc98-e3ee-497f-ae69-7ce442751820-kube-api-access-g8rds\") pod \"odh-model-controller-7998c67dc4-cc9zh\" (UID: \"c677dc98-e3ee-497f-ae69-7ce442751820\") " pod="opendatahub/odh-model-controller-7998c67dc4-cc9zh" Jul 29 18:40:38.028821 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:40:38.028789 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:40:38.055196 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:40:38.055172 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-54cd75bc79-ns79p"] Jul 29 18:40:38.057288 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:40:38.057251 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d9798e3_f217_4a46_ab4d_9f2bd22e6516.slice/crio-9027fc4a6505272a0877b382843dd8e9138fd319f3baba9e00336e28c7da50cf WatchSource:0}: Error finding container 9027fc4a6505272a0877b382843dd8e9138fd319f3baba9e00336e28c7da50cf: Status 404 returned error can't find the container with id 9027fc4a6505272a0877b382843dd8e9138fd319f3baba9e00336e28c7da50cf Jul 29 18:40:38.121078 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:40:38.121020 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:40:38.138467 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:40:38.138436 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-5b7494b68d-72vwg"] Jul 29 18:40:38.143179 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:40:38.143164 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5b7494b68d-72vwg" Jul 29 18:40:38.146074 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:40:38.146053 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 29 18:40:38.146074 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:40:38.146061 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 29 18:40:38.146239 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:40:38.146053 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 29 18:40:38.147318 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:40:38.147047 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-5c2bs\"" Jul 29 18:40:38.147438 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:40:38.147313 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 29 18:40:38.147577 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:40:38.147536 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 29 18:40:38.156472 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:40:38.156449 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5b7494b68d-72vwg"] Jul 29 18:40:38.204704 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:40:38.204678 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/571420ac-fc46-4194-9a92-515eb6937cec-cert\") pod \"lws-controller-manager-5b7494b68d-72vwg\" (UID: \"571420ac-fc46-4194-9a92-515eb6937cec\") " pod="openshift-lws-operator/lws-controller-manager-5b7494b68d-72vwg" Jul 29 18:40:38.204809 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:40:38.204716 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8lhd\" (UniqueName: \"kubernetes.io/projected/571420ac-fc46-4194-9a92-515eb6937cec-kube-api-access-b8lhd\") pod \"lws-controller-manager-5b7494b68d-72vwg\" (UID: \"571420ac-fc46-4194-9a92-515eb6937cec\") " pod="openshift-lws-operator/lws-controller-manager-5b7494b68d-72vwg" Jul 29 18:40:38.204809 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:40:38.204740 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/571420ac-fc46-4194-9a92-515eb6937cec-metrics-cert\") pod \"lws-controller-manager-5b7494b68d-72vwg\" (UID: \"571420ac-fc46-4194-9a92-515eb6937cec\") " pod="openshift-lws-operator/lws-controller-manager-5b7494b68d-72vwg" Jul 29 18:40:38.204878 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:40:38.204849 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/571420ac-fc46-4194-9a92-515eb6937cec-manager-config\") pod \"lws-controller-manager-5b7494b68d-72vwg\" (UID: \"571420ac-fc46-4194-9a92-515eb6937cec\") " pod="openshift-lws-operator/lws-controller-manager-5b7494b68d-72vwg" Jul 29 18:40:38.305467 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:40:38.305433 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/571420ac-fc46-4194-9a92-515eb6937cec-cert\") pod \"lws-controller-manager-5b7494b68d-72vwg\" (UID: \"571420ac-fc46-4194-9a92-515eb6937cec\") " pod="openshift-lws-operator/lws-controller-manager-5b7494b68d-72vwg" Jul 29 18:40:38.305598 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:40:38.305475 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8lhd\" (UniqueName: \"kubernetes.io/projected/571420ac-fc46-4194-9a92-515eb6937cec-kube-api-access-b8lhd\") pod \"lws-controller-manager-5b7494b68d-72vwg\" (UID: \"571420ac-fc46-4194-9a92-515eb6937cec\") " pod="openshift-lws-operator/lws-controller-manager-5b7494b68d-72vwg" Jul 29 18:40:38.305598 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:40:38.305500 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/571420ac-fc46-4194-9a92-515eb6937cec-metrics-cert\") pod \"lws-controller-manager-5b7494b68d-72vwg\" (UID: \"571420ac-fc46-4194-9a92-515eb6937cec\") " pod="openshift-lws-operator/lws-controller-manager-5b7494b68d-72vwg" Jul 29 18:40:38.305598 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:40:38.305571 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/571420ac-fc46-4194-9a92-515eb6937cec-manager-config\") pod \"lws-controller-manager-5b7494b68d-72vwg\" (UID: \"571420ac-fc46-4194-9a92-515eb6937cec\") " pod="openshift-lws-operator/lws-controller-manager-5b7494b68d-72vwg" Jul 29 18:40:38.306234 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:40:38.306211 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/571420ac-fc46-4194-9a92-515eb6937cec-manager-config\") pod \"lws-controller-manager-5b7494b68d-72vwg\" (UID: \"571420ac-fc46-4194-9a92-515eb6937cec\") " pod="openshift-lws-operator/lws-controller-manager-5b7494b68d-72vwg" Jul 29 18:40:38.308011 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:40:38.307991 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/571420ac-fc46-4194-9a92-515eb6937cec-cert\") pod \"lws-controller-manager-5b7494b68d-72vwg\" (UID: \"571420ac-fc46-4194-9a92-515eb6937cec\") " pod="openshift-lws-operator/lws-controller-manager-5b7494b68d-72vwg" Jul 29 18:40:38.308180 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:40:38.308160 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/571420ac-fc46-4194-9a92-515eb6937cec-metrics-cert\") pod \"lws-controller-manager-5b7494b68d-72vwg\" (UID: \"571420ac-fc46-4194-9a92-515eb6937cec\") " pod="openshift-lws-operator/lws-controller-manager-5b7494b68d-72vwg" Jul 29 18:40:38.314272 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:40:38.314252 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8lhd\" (UniqueName: \"kubernetes.io/projected/571420ac-fc46-4194-9a92-515eb6937cec-kube-api-access-b8lhd\") pod \"lws-controller-manager-5b7494b68d-72vwg\" (UID: \"571420ac-fc46-4194-9a92-515eb6937cec\") " pod="openshift-lws-operator/lws-controller-manager-5b7494b68d-72vwg" Jul 29 18:40:38.456995 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:40:38.456920 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5b7494b68d-72vwg" Jul 29 18:40:38.507268 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:40:38.507236 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c677dc98-e3ee-497f-ae69-7ce442751820-cert\") pod \"odh-model-controller-7998c67dc4-cc9zh\" (UID: \"c677dc98-e3ee-497f-ae69-7ce442751820\") " pod="opendatahub/odh-model-controller-7998c67dc4-cc9zh" Jul 29 18:40:38.510067 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:40:38.510046 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c677dc98-e3ee-497f-ae69-7ce442751820-cert\") pod \"odh-model-controller-7998c67dc4-cc9zh\" (UID: \"c677dc98-e3ee-497f-ae69-7ce442751820\") " pod="opendatahub/odh-model-controller-7998c67dc4-cc9zh" Jul 29 18:40:38.592533 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:40:38.592509 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5b7494b68d-72vwg"] Jul 29 18:40:38.594520 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:40:38.594488 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod571420ac_fc46_4194_9a92_515eb6937cec.slice/crio-5db2b7f4d045fc758eb25f0258df7002a88b5bc642fc60e4c83164f87fcfee64 WatchSource:0}: Error finding container 5db2b7f4d045fc758eb25f0258df7002a88b5bc642fc60e4c83164f87fcfee64: Status 404 returned error can't find the container with id 5db2b7f4d045fc758eb25f0258df7002a88b5bc642fc60e4c83164f87fcfee64 Jul 29 18:40:38.739004 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:40:38.738932 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-cc9zh" Jul 29 18:40:38.753215 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:40:38.753184 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:40:38.891075 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:40:38.891045 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-cc9zh"] Jul 29 18:40:38.891872 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:40:38.891842 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc677dc98_e3ee_497f_ae69_7ce442751820.slice/crio-0bb7deb7cff9d44525f65eeabc3be31041cf3ed9a9702e76bf58785091c3f010 WatchSource:0}: Error finding container 0bb7deb7cff9d44525f65eeabc3be31041cf3ed9a9702e76bf58785091c3f010: Status 404 returned error can't find the container with id 0bb7deb7cff9d44525f65eeabc3be31041cf3ed9a9702e76bf58785091c3f010 Jul 29 18:40:38.914098 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:40:38.914065 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:40:39.045181 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:40:39.045143 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5b7494b68d-72vwg" event={"ID":"571420ac-fc46-4194-9a92-515eb6937cec","Type":"ContainerStarted","Data":"5db2b7f4d045fc758eb25f0258df7002a88b5bc642fc60e4c83164f87fcfee64"} Jul 29 18:40:39.046709 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:40:39.046665 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-54cd75bc79-ns79p" event={"ID":"2d9798e3-f217-4a46-ab4d-9f2bd22e6516","Type":"ContainerStarted","Data":"9027fc4a6505272a0877b382843dd8e9138fd319f3baba9e00336e28c7da50cf"} Jul 29 18:40:39.048101 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:40:39.048073 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-cc9zh" event={"ID":"c677dc98-e3ee-497f-ae69-7ce442751820","Type":"ContainerStarted","Data":"0bb7deb7cff9d44525f65eeabc3be31041cf3ed9a9702e76bf58785091c3f010"} Jul 29 18:40:39.076642 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:40:39.076615 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:40:39.139179 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:40:39.139096 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:40:40.069448 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:40:40.069398 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:40:43.071267 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:40:43.071235 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5b7494b68d-72vwg" event={"ID":"571420ac-fc46-4194-9a92-515eb6937cec","Type":"ContainerStarted","Data":"48afa89a2241a6b832ffd72d1971f7b72bfdd63d2f150469c086eb3f649af1fe"} Jul 29 18:40:43.071703 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:40:43.071386 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-5b7494b68d-72vwg" Jul 29 18:40:43.072923 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:40:43.072889 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-54cd75bc79-ns79p" event={"ID":"2d9798e3-f217-4a46-ab4d-9f2bd22e6516","Type":"ContainerStarted","Data":"0bcb9a7cfe0d31a69c94e9f0e233378e04e62b9d4a7869211c8ef71c73c427e2"} Jul 29 18:40:43.074260 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:40:43.074240 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-cc9zh" event={"ID":"c677dc98-e3ee-497f-ae69-7ce442751820","Type":"ContainerStarted","Data":"72b76ce3f04ff710f1e8cfd105f22d1acffb1376313395d0bf754d0baf75cf1e"} Jul 29 18:40:43.074392 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:40:43.074276 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-cc9zh" Jul 29 18:40:43.090045 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:40:43.089994 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-5b7494b68d-72vwg" podStartSLOduration=1.597734969 podStartE2EDuration="5.089975023s" totalImagesPullingTime="3.492240054s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 18:40:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 18:40:43.089734042 +0000 UTC m=+378.315402915" watchObservedRunningTime="2026-07-29 18:40:43.089975023 +0000 UTC m=+378.315643896" Jul 29 18:40:43.105629 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:40:43.105587 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-54cd75bc79-ns79p" podStartSLOduration=2.069718215 podStartE2EDuration="6.105573127s" totalImagesPullingTime="4.035854912s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 18:40:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 18:40:43.104692332 +0000 UTC m=+378.330361204" watchObservedRunningTime="2026-07-29 18:40:43.105573127 +0000 UTC m=+378.331241999" Jul 29 18:40:43.121573 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:40:43.121530 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-cc9zh" podStartSLOduration=2.840140946 podStartE2EDuration="6.121517221s" totalImagesPullingTime="3.281376275s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 18:40:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 18:40:43.120175537 +0000 UTC m=+378.345844410" watchObservedRunningTime="2026-07-29 18:40:43.121517221 +0000 UTC m=+378.347186123" Jul 29 18:40:43.343827 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:40:43.343728 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:40:43.447796 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:40:43.447748 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:40:43.953641 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:40:43.953595 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:40:44.259834 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:40:44.259802 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:40:44.360654 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:40:44.360609 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:40:48.281508 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:40:48.281476 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:40:48.394735 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:40:48.394702 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:40:48.513258 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:40:48.513224 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:40:48.685478 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:40:48.685393 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:40:48.775607 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:40:48.775571 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:40:54.083259 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:40:54.083229 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-cc9zh" Jul 29 18:40:54.083670 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:40:54.083306 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-5b7494b68d-72vwg" Jul 29 18:41:07.044670 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:41:07.044637 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:41:38.724199 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:41:38.724163 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:41:38.823477 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:41:38.823443 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:41:39.012113 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:41:39.012080 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:41:39.678232 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:41:39.678197 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:41:39.879288 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:41:39.879254 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:41:49.446116 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:41:49.446032 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:41:51.393654 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:41:51.393614 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:41:51.494737 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:41:51.494703 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:41:54.683994 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:41:54.683963 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759s2bcr"] Jul 29 18:41:54.686765 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:41:54.686750 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759s2bcr" Jul 29 18:41:54.689139 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:41:54.689119 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-bnlth\"" Jul 29 18:41:54.689229 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:41:54.689146 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 29 18:41:54.689856 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:41:54.689841 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 29 18:41:54.696742 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:41:54.696722 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759s2bcr"] Jul 29 18:41:54.767202 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:41:54.767172 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e746889e-57fd-4aaf-bcec-4616fc712dc2-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759s2bcr\" (UID: \"e746889e-57fd-4aaf-bcec-4616fc712dc2\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759s2bcr" Jul 29 18:41:54.767317 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:41:54.767217 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hvvt7\" (UniqueName: \"kubernetes.io/projected/e746889e-57fd-4aaf-bcec-4616fc712dc2-kube-api-access-hvvt7\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759s2bcr\" (UID: \"e746889e-57fd-4aaf-bcec-4616fc712dc2\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759s2bcr" Jul 29 18:41:54.767317 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:41:54.767240 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e746889e-57fd-4aaf-bcec-4616fc712dc2-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759s2bcr\" (UID: \"e746889e-57fd-4aaf-bcec-4616fc712dc2\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759s2bcr" Jul 29 18:41:54.868490 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:41:54.868463 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e746889e-57fd-4aaf-bcec-4616fc712dc2-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759s2bcr\" (UID: \"e746889e-57fd-4aaf-bcec-4616fc712dc2\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759s2bcr" Jul 29 18:41:54.868643 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:41:54.868512 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvvt7\" (UniqueName: \"kubernetes.io/projected/e746889e-57fd-4aaf-bcec-4616fc712dc2-kube-api-access-hvvt7\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759s2bcr\" (UID: \"e746889e-57fd-4aaf-bcec-4616fc712dc2\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759s2bcr" Jul 29 18:41:54.868643 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:41:54.868536 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e746889e-57fd-4aaf-bcec-4616fc712dc2-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759s2bcr\" (UID: \"e746889e-57fd-4aaf-bcec-4616fc712dc2\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759s2bcr" Jul 29 18:41:54.868912 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:41:54.868888 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e746889e-57fd-4aaf-bcec-4616fc712dc2-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759s2bcr\" (UID: \"e746889e-57fd-4aaf-bcec-4616fc712dc2\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759s2bcr" Jul 29 18:41:54.868987 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:41:54.868923 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e746889e-57fd-4aaf-bcec-4616fc712dc2-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759s2bcr\" (UID: \"e746889e-57fd-4aaf-bcec-4616fc712dc2\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759s2bcr" Jul 29 18:41:54.877695 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:41:54.877669 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvvt7\" (UniqueName: \"kubernetes.io/projected/e746889e-57fd-4aaf-bcec-4616fc712dc2-kube-api-access-hvvt7\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759s2bcr\" (UID: \"e746889e-57fd-4aaf-bcec-4616fc712dc2\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759s2bcr" Jul 29 18:41:54.997045 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:41:54.997019 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759s2bcr" Jul 29 18:41:55.090249 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:41:55.090215 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gvtj8"] Jul 29 18:41:55.093682 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:41:55.093661 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gvtj8" Jul 29 18:41:55.103493 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:41:55.102548 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gvtj8"] Jul 29 18:41:55.136712 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:41:55.136687 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759s2bcr"] Jul 29 18:41:55.139059 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:41:55.139025 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode746889e_57fd_4aaf_bcec_4616fc712dc2.slice/crio-5c54aa5da3c063c97cf02a414a8a308ae18b2f355c02ce54e14ba6819b34a295 WatchSource:0}: Error finding container 5c54aa5da3c063c97cf02a414a8a308ae18b2f355c02ce54e14ba6819b34a295: Status 404 returned error can't find the container with id 5c54aa5da3c063c97cf02a414a8a308ae18b2f355c02ce54e14ba6819b34a295 Jul 29 18:41:55.171011 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:41:55.170986 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/62597727-1dca-4b09-a51f-ffbd5df32885-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gvtj8\" (UID: \"62597727-1dca-4b09-a51f-ffbd5df32885\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gvtj8" Jul 29 18:41:55.171127 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:41:55.171071 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/62597727-1dca-4b09-a51f-ffbd5df32885-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gvtj8\" (UID: \"62597727-1dca-4b09-a51f-ffbd5df32885\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gvtj8" Jul 29 18:41:55.171127 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:41:55.171099 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fh54r\" (UniqueName: \"kubernetes.io/projected/62597727-1dca-4b09-a51f-ffbd5df32885-kube-api-access-fh54r\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gvtj8\" (UID: \"62597727-1dca-4b09-a51f-ffbd5df32885\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gvtj8" Jul 29 18:41:55.272100 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:41:55.272042 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/62597727-1dca-4b09-a51f-ffbd5df32885-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gvtj8\" (UID: \"62597727-1dca-4b09-a51f-ffbd5df32885\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gvtj8" Jul 29 18:41:55.272208 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:41:55.272114 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/62597727-1dca-4b09-a51f-ffbd5df32885-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gvtj8\" (UID: \"62597727-1dca-4b09-a51f-ffbd5df32885\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gvtj8" Jul 29 18:41:55.272208 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:41:55.272135 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fh54r\" (UniqueName: \"kubernetes.io/projected/62597727-1dca-4b09-a51f-ffbd5df32885-kube-api-access-fh54r\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gvtj8\" (UID: \"62597727-1dca-4b09-a51f-ffbd5df32885\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gvtj8" Jul 29 18:41:55.272405 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:41:55.272383 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/62597727-1dca-4b09-a51f-ffbd5df32885-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gvtj8\" (UID: \"62597727-1dca-4b09-a51f-ffbd5df32885\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gvtj8" Jul 29 18:41:55.272454 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:41:55.272414 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/62597727-1dca-4b09-a51f-ffbd5df32885-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gvtj8\" (UID: \"62597727-1dca-4b09-a51f-ffbd5df32885\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gvtj8" Jul 29 18:41:55.280709 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:41:55.280690 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fh54r\" (UniqueName: \"kubernetes.io/projected/62597727-1dca-4b09-a51f-ffbd5df32885-kube-api-access-fh54r\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gvtj8\" (UID: \"62597727-1dca-4b09-a51f-ffbd5df32885\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gvtj8" Jul 29 18:41:55.407434 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:41:55.407409 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gvtj8" Jul 29 18:41:55.417055 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:41:55.417018 2560 generic.go:356] "Generic (PLEG): container finished" podID="e746889e-57fd-4aaf-bcec-4616fc712dc2" containerID="2892dc8e2641ac34b153818ca56d3ee1d73cee0416f75e3916e93d9b686e0b36" exitCode=0 Jul 29 18:41:55.417172 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:41:55.417073 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759s2bcr" event={"ID":"e746889e-57fd-4aaf-bcec-4616fc712dc2","Type":"ContainerDied","Data":"2892dc8e2641ac34b153818ca56d3ee1d73cee0416f75e3916e93d9b686e0b36"} Jul 29 18:41:55.417172 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:41:55.417113 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759s2bcr" event={"ID":"e746889e-57fd-4aaf-bcec-4616fc712dc2","Type":"ContainerStarted","Data":"5c54aa5da3c063c97cf02a414a8a308ae18b2f355c02ce54e14ba6819b34a295"} Jul 29 18:41:55.486802 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:41:55.486773 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0drl9r"] Jul 29 18:41:55.489907 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:41:55.489887 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0drl9r" Jul 29 18:41:55.501887 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:41:55.501862 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0drl9r"] Jul 29 18:41:55.543879 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:41:55.543857 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gvtj8"] Jul 29 18:41:55.546259 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:41:55.546234 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62597727_1dca_4b09_a51f_ffbd5df32885.slice/crio-b1ebb7576b3f2fb51b45e4c105d75d0ab8eb95bbd53a9caf3d88209534fe430b WatchSource:0}: Error finding container b1ebb7576b3f2fb51b45e4c105d75d0ab8eb95bbd53a9caf3d88209534fe430b: Status 404 returned error can't find the container with id b1ebb7576b3f2fb51b45e4c105d75d0ab8eb95bbd53a9caf3d88209534fe430b Jul 29 18:41:55.573910 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:41:55.573886 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b2a0edd1-d807-4803-b89e-1af7a9d2302d-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0drl9r\" (UID: \"b2a0edd1-d807-4803-b89e-1af7a9d2302d\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0drl9r" Jul 29 18:41:55.574029 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:41:55.573990 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxx7c\" (UniqueName: \"kubernetes.io/projected/b2a0edd1-d807-4803-b89e-1af7a9d2302d-kube-api-access-mxx7c\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0drl9r\" (UID: \"b2a0edd1-d807-4803-b89e-1af7a9d2302d\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0drl9r" Jul 29 18:41:55.574029 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:41:55.574025 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b2a0edd1-d807-4803-b89e-1af7a9d2302d-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0drl9r\" (UID: \"b2a0edd1-d807-4803-b89e-1af7a9d2302d\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0drl9r" Jul 29 18:41:55.674725 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:41:55.674688 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxx7c\" (UniqueName: \"kubernetes.io/projected/b2a0edd1-d807-4803-b89e-1af7a9d2302d-kube-api-access-mxx7c\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0drl9r\" (UID: \"b2a0edd1-d807-4803-b89e-1af7a9d2302d\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0drl9r" Jul 29 18:41:55.674725 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:41:55.674724 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b2a0edd1-d807-4803-b89e-1af7a9d2302d-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0drl9r\" (UID: \"b2a0edd1-d807-4803-b89e-1af7a9d2302d\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0drl9r" Jul 29 18:41:55.674931 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:41:55.674790 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b2a0edd1-d807-4803-b89e-1af7a9d2302d-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0drl9r\" (UID: \"b2a0edd1-d807-4803-b89e-1af7a9d2302d\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0drl9r" Jul 29 18:41:55.675220 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:41:55.675198 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b2a0edd1-d807-4803-b89e-1af7a9d2302d-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0drl9r\" (UID: \"b2a0edd1-d807-4803-b89e-1af7a9d2302d\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0drl9r" Jul 29 18:41:55.675261 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:41:55.675198 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b2a0edd1-d807-4803-b89e-1af7a9d2302d-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0drl9r\" (UID: \"b2a0edd1-d807-4803-b89e-1af7a9d2302d\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0drl9r" Jul 29 18:41:55.683207 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:41:55.683182 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxx7c\" (UniqueName: \"kubernetes.io/projected/b2a0edd1-d807-4803-b89e-1af7a9d2302d-kube-api-access-mxx7c\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0drl9r\" (UID: \"b2a0edd1-d807-4803-b89e-1af7a9d2302d\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0drl9r" Jul 29 18:41:55.802528 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:41:55.802452 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0drl9r" Jul 29 18:41:55.934989 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:41:55.934964 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0drl9r"] Jul 29 18:41:55.936420 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:41:55.936393 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2a0edd1_d807_4803_b89e_1af7a9d2302d.slice/crio-51d1b041b1c201536cf9982b3537ac0ffb19bb0ca6c969d51bfef00d0a3ea45f WatchSource:0}: Error finding container 51d1b041b1c201536cf9982b3537ac0ffb19bb0ca6c969d51bfef00d0a3ea45f: Status 404 returned error can't find the container with id 51d1b041b1c201536cf9982b3537ac0ffb19bb0ca6c969d51bfef00d0a3ea45f Jul 29 18:41:56.082006 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:41:56.081941 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73btqth"] Jul 29 18:41:56.084892 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:41:56.084871 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73btqth" Jul 29 18:41:56.093873 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:41:56.093848 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73btqth"] Jul 29 18:41:56.178662 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:41:56.178626 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/970cab5f-702a-4e4c-8ea5-b4743aa8718c-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73btqth\" (UID: \"970cab5f-702a-4e4c-8ea5-b4743aa8718c\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73btqth" Jul 29 18:41:56.178809 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:41:56.178683 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xlkbc\" (UniqueName: \"kubernetes.io/projected/970cab5f-702a-4e4c-8ea5-b4743aa8718c-kube-api-access-xlkbc\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73btqth\" (UID: \"970cab5f-702a-4e4c-8ea5-b4743aa8718c\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73btqth" Jul 29 18:41:56.178809 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:41:56.178751 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/970cab5f-702a-4e4c-8ea5-b4743aa8718c-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73btqth\" (UID: \"970cab5f-702a-4e4c-8ea5-b4743aa8718c\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73btqth" Jul 29 18:41:56.279293 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:41:56.279264 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/970cab5f-702a-4e4c-8ea5-b4743aa8718c-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73btqth\" (UID: \"970cab5f-702a-4e4c-8ea5-b4743aa8718c\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73btqth" Jul 29 18:41:56.279411 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:41:56.279376 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/970cab5f-702a-4e4c-8ea5-b4743aa8718c-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73btqth\" (UID: \"970cab5f-702a-4e4c-8ea5-b4743aa8718c\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73btqth" Jul 29 18:41:56.279463 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:41:56.279445 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xlkbc\" (UniqueName: \"kubernetes.io/projected/970cab5f-702a-4e4c-8ea5-b4743aa8718c-kube-api-access-xlkbc\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73btqth\" (UID: \"970cab5f-702a-4e4c-8ea5-b4743aa8718c\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73btqth" Jul 29 18:41:56.279704 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:41:56.279680 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/970cab5f-702a-4e4c-8ea5-b4743aa8718c-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73btqth\" (UID: \"970cab5f-702a-4e4c-8ea5-b4743aa8718c\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73btqth" Jul 29 18:41:56.279749 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:41:56.279708 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/970cab5f-702a-4e4c-8ea5-b4743aa8718c-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73btqth\" (UID: \"970cab5f-702a-4e4c-8ea5-b4743aa8718c\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73btqth" Jul 29 18:41:56.288514 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:41:56.288472 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xlkbc\" (UniqueName: \"kubernetes.io/projected/970cab5f-702a-4e4c-8ea5-b4743aa8718c-kube-api-access-xlkbc\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73btqth\" (UID: \"970cab5f-702a-4e4c-8ea5-b4743aa8718c\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73btqth" Jul 29 18:41:56.395993 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:41:56.395937 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73btqth" Jul 29 18:41:56.423728 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:41:56.423692 2560 generic.go:356] "Generic (PLEG): container finished" podID="b2a0edd1-d807-4803-b89e-1af7a9d2302d" containerID="43d34d819c234ec095b7ac411affbf8044952e56a997e45b5491c39edfa6f9a9" exitCode=0 Jul 29 18:41:56.423900 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:41:56.423850 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0drl9r" event={"ID":"b2a0edd1-d807-4803-b89e-1af7a9d2302d","Type":"ContainerDied","Data":"43d34d819c234ec095b7ac411affbf8044952e56a997e45b5491c39edfa6f9a9"} Jul 29 18:41:56.423977 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:41:56.423958 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0drl9r" event={"ID":"b2a0edd1-d807-4803-b89e-1af7a9d2302d","Type":"ContainerStarted","Data":"51d1b041b1c201536cf9982b3537ac0ffb19bb0ca6c969d51bfef00d0a3ea45f"} Jul 29 18:41:56.425592 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:41:56.425564 2560 generic.go:356] "Generic (PLEG): container finished" podID="62597727-1dca-4b09-a51f-ffbd5df32885" containerID="a4b0843ddc2f986239ee01fac3d8a8f91002256211b77874357e9be2d7c1bf48" exitCode=0 Jul 29 18:41:56.425706 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:41:56.425649 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gvtj8" event={"ID":"62597727-1dca-4b09-a51f-ffbd5df32885","Type":"ContainerDied","Data":"a4b0843ddc2f986239ee01fac3d8a8f91002256211b77874357e9be2d7c1bf48"} Jul 29 18:41:56.425706 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:41:56.425695 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gvtj8" event={"ID":"62597727-1dca-4b09-a51f-ffbd5df32885","Type":"ContainerStarted","Data":"b1ebb7576b3f2fb51b45e4c105d75d0ab8eb95bbd53a9caf3d88209534fe430b"} Jul 29 18:41:56.428007 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:41:56.427983 2560 generic.go:356] "Generic (PLEG): container finished" podID="e746889e-57fd-4aaf-bcec-4616fc712dc2" containerID="fdb3e76de9890e2986dd551728516c49cf2491cc644b80a2f8797569e1073824" exitCode=0 Jul 29 18:41:56.428098 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:41:56.428033 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759s2bcr" event={"ID":"e746889e-57fd-4aaf-bcec-4616fc712dc2","Type":"ContainerDied","Data":"fdb3e76de9890e2986dd551728516c49cf2491cc644b80a2f8797569e1073824"} Jul 29 18:41:56.532417 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:41:56.532371 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73btqth"] Jul 29 18:41:56.534729 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:41:56.534705 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod970cab5f_702a_4e4c_8ea5_b4743aa8718c.slice/crio-261c80dd318f54ca000623767c99591e1d989f1f7117469464c1683bdf3139fe WatchSource:0}: Error finding container 261c80dd318f54ca000623767c99591e1d989f1f7117469464c1683bdf3139fe: Status 404 returned error can't find the container with id 261c80dd318f54ca000623767c99591e1d989f1f7117469464c1683bdf3139fe Jul 29 18:41:57.436000 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:41:57.435969 2560 generic.go:356] "Generic (PLEG): container finished" podID="e746889e-57fd-4aaf-bcec-4616fc712dc2" containerID="6be06e45e182b532a9b2bce719d219a3d17931fe2313d7d4aa8ccf7f9f8dddd4" exitCode=0 Jul 29 18:41:57.436323 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:41:57.436099 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759s2bcr" event={"ID":"e746889e-57fd-4aaf-bcec-4616fc712dc2","Type":"ContainerDied","Data":"6be06e45e182b532a9b2bce719d219a3d17931fe2313d7d4aa8ccf7f9f8dddd4"} Jul 29 18:41:57.437571 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:41:57.437547 2560 generic.go:356] "Generic (PLEG): container finished" podID="970cab5f-702a-4e4c-8ea5-b4743aa8718c" containerID="821c3083ed57ed7bf0f29983ce7eee81169e220aef9276a305b665b0afdc5abe" exitCode=0 Jul 29 18:41:57.437678 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:41:57.437627 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73btqth" event={"ID":"970cab5f-702a-4e4c-8ea5-b4743aa8718c","Type":"ContainerDied","Data":"821c3083ed57ed7bf0f29983ce7eee81169e220aef9276a305b665b0afdc5abe"} Jul 29 18:41:57.437678 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:41:57.437671 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73btqth" event={"ID":"970cab5f-702a-4e4c-8ea5-b4743aa8718c","Type":"ContainerStarted","Data":"261c80dd318f54ca000623767c99591e1d989f1f7117469464c1683bdf3139fe"} Jul 29 18:41:57.439667 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:41:57.439644 2560 generic.go:356] "Generic (PLEG): container finished" podID="62597727-1dca-4b09-a51f-ffbd5df32885" containerID="da5a83b15eec619bb2fb15821f3018eb1a63916a37a29ce91c2e3e1ec1156760" exitCode=0 Jul 29 18:41:57.439775 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:41:57.439693 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gvtj8" event={"ID":"62597727-1dca-4b09-a51f-ffbd5df32885","Type":"ContainerDied","Data":"da5a83b15eec619bb2fb15821f3018eb1a63916a37a29ce91c2e3e1ec1156760"} Jul 29 18:41:58.453057 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:41:58.453015 2560 generic.go:356] "Generic (PLEG): container finished" podID="62597727-1dca-4b09-a51f-ffbd5df32885" containerID="ded2c5cc9369f958fa1895c98caba8dcecc3d6fd71dcde7e26aa0b27e9e2506d" exitCode=0 Jul 29 18:41:58.453492 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:41:58.453112 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gvtj8" event={"ID":"62597727-1dca-4b09-a51f-ffbd5df32885","Type":"ContainerDied","Data":"ded2c5cc9369f958fa1895c98caba8dcecc3d6fd71dcde7e26aa0b27e9e2506d"} Jul 29 18:41:58.455693 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:41:58.455664 2560 generic.go:356] "Generic (PLEG): container finished" podID="b2a0edd1-d807-4803-b89e-1af7a9d2302d" containerID="b414abab234741bd7b685997acdb36440211ad5bfa671f86d9019812ef5a6c05" exitCode=0 Jul 29 18:41:58.455930 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:41:58.455727 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0drl9r" event={"ID":"b2a0edd1-d807-4803-b89e-1af7a9d2302d","Type":"ContainerDied","Data":"b414abab234741bd7b685997acdb36440211ad5bfa671f86d9019812ef5a6c05"} Jul 29 18:41:58.619770 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:41:58.619306 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759s2bcr" Jul 29 18:41:58.701159 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:41:58.701137 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e746889e-57fd-4aaf-bcec-4616fc712dc2-util\") pod \"e746889e-57fd-4aaf-bcec-4616fc712dc2\" (UID: \"e746889e-57fd-4aaf-bcec-4616fc712dc2\") " Jul 29 18:41:58.701274 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:41:58.701219 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e746889e-57fd-4aaf-bcec-4616fc712dc2-bundle\") pod \"e746889e-57fd-4aaf-bcec-4616fc712dc2\" (UID: \"e746889e-57fd-4aaf-bcec-4616fc712dc2\") " Jul 29 18:41:58.701274 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:41:58.701257 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hvvt7\" (UniqueName: \"kubernetes.io/projected/e746889e-57fd-4aaf-bcec-4616fc712dc2-kube-api-access-hvvt7\") pod \"e746889e-57fd-4aaf-bcec-4616fc712dc2\" (UID: \"e746889e-57fd-4aaf-bcec-4616fc712dc2\") " Jul 29 18:41:58.701686 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:41:58.701655 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e746889e-57fd-4aaf-bcec-4616fc712dc2-bundle" (OuterVolumeSpecName: "bundle") pod "e746889e-57fd-4aaf-bcec-4616fc712dc2" (UID: "e746889e-57fd-4aaf-bcec-4616fc712dc2"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 18:41:58.703525 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:41:58.703478 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e746889e-57fd-4aaf-bcec-4616fc712dc2-kube-api-access-hvvt7" (OuterVolumeSpecName: "kube-api-access-hvvt7") pod "e746889e-57fd-4aaf-bcec-4616fc712dc2" (UID: "e746889e-57fd-4aaf-bcec-4616fc712dc2"). InnerVolumeSpecName "kube-api-access-hvvt7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 18:41:58.707228 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:41:58.707205 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e746889e-57fd-4aaf-bcec-4616fc712dc2-util" (OuterVolumeSpecName: "util") pod "e746889e-57fd-4aaf-bcec-4616fc712dc2" (UID: "e746889e-57fd-4aaf-bcec-4616fc712dc2"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 18:41:58.802096 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:41:58.802072 2560 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e746889e-57fd-4aaf-bcec-4616fc712dc2-bundle\") on node \"ip-10-0-140-205.ec2.internal\" DevicePath \"\"" Jul 29 18:41:58.802096 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:41:58.802094 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hvvt7\" (UniqueName: \"kubernetes.io/projected/e746889e-57fd-4aaf-bcec-4616fc712dc2-kube-api-access-hvvt7\") on node \"ip-10-0-140-205.ec2.internal\" DevicePath \"\"" Jul 29 18:41:58.802223 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:41:58.802105 2560 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e746889e-57fd-4aaf-bcec-4616fc712dc2-util\") on node \"ip-10-0-140-205.ec2.internal\" DevicePath \"\"" Jul 29 18:41:59.463501 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:41:59.463472 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759s2bcr" Jul 29 18:41:59.463501 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:41:59.463496 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759s2bcr" event={"ID":"e746889e-57fd-4aaf-bcec-4616fc712dc2","Type":"ContainerDied","Data":"5c54aa5da3c063c97cf02a414a8a308ae18b2f355c02ce54e14ba6819b34a295"} Jul 29 18:41:59.463911 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:41:59.463542 2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5c54aa5da3c063c97cf02a414a8a308ae18b2f355c02ce54e14ba6819b34a295" Jul 29 18:41:59.465863 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:41:59.465836 2560 generic.go:356] "Generic (PLEG): container finished" podID="b2a0edd1-d807-4803-b89e-1af7a9d2302d" containerID="b0c5a1cecce204a2e67d5c0faff201bc31684165af2dd672f19f5d83aba63e31" exitCode=0 Jul 29 18:41:59.465994 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:41:59.465868 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0drl9r" event={"ID":"b2a0edd1-d807-4803-b89e-1af7a9d2302d","Type":"ContainerDied","Data":"b0c5a1cecce204a2e67d5c0faff201bc31684165af2dd672f19f5d83aba63e31"} Jul 29 18:41:59.467659 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:41:59.467638 2560 generic.go:356] "Generic (PLEG): container finished" podID="970cab5f-702a-4e4c-8ea5-b4743aa8718c" containerID="f818ef6dfbf91c47858d443630b8e8acc74d08ab735ecaae83e1a1c39ba5865b" exitCode=0 Jul 29 18:41:59.467755 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:41:59.467719 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73btqth" event={"ID":"970cab5f-702a-4e4c-8ea5-b4743aa8718c","Type":"ContainerDied","Data":"f818ef6dfbf91c47858d443630b8e8acc74d08ab735ecaae83e1a1c39ba5865b"} Jul 29 18:41:59.603798 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:41:59.603776 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gvtj8" Jul 29 18:41:59.710454 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:41:59.710433 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fh54r\" (UniqueName: \"kubernetes.io/projected/62597727-1dca-4b09-a51f-ffbd5df32885-kube-api-access-fh54r\") pod \"62597727-1dca-4b09-a51f-ffbd5df32885\" (UID: \"62597727-1dca-4b09-a51f-ffbd5df32885\") " Jul 29 18:41:59.710562 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:41:59.710465 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/62597727-1dca-4b09-a51f-ffbd5df32885-util\") pod \"62597727-1dca-4b09-a51f-ffbd5df32885\" (UID: \"62597727-1dca-4b09-a51f-ffbd5df32885\") " Jul 29 18:41:59.710598 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:41:59.710560 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/62597727-1dca-4b09-a51f-ffbd5df32885-bundle\") pod \"62597727-1dca-4b09-a51f-ffbd5df32885\" (UID: \"62597727-1dca-4b09-a51f-ffbd5df32885\") " Jul 29 18:41:59.711156 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:41:59.711086 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62597727-1dca-4b09-a51f-ffbd5df32885-bundle" (OuterVolumeSpecName: "bundle") pod "62597727-1dca-4b09-a51f-ffbd5df32885" (UID: "62597727-1dca-4b09-a51f-ffbd5df32885"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 18:41:59.712537 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:41:59.712508 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62597727-1dca-4b09-a51f-ffbd5df32885-kube-api-access-fh54r" (OuterVolumeSpecName: "kube-api-access-fh54r") pod "62597727-1dca-4b09-a51f-ffbd5df32885" (UID: "62597727-1dca-4b09-a51f-ffbd5df32885"). InnerVolumeSpecName "kube-api-access-fh54r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 18:41:59.716163 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:41:59.716140 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62597727-1dca-4b09-a51f-ffbd5df32885-util" (OuterVolumeSpecName: "util") pod "62597727-1dca-4b09-a51f-ffbd5df32885" (UID: "62597727-1dca-4b09-a51f-ffbd5df32885"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 18:41:59.812046 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:41:59.812016 2560 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/62597727-1dca-4b09-a51f-ffbd5df32885-bundle\") on node \"ip-10-0-140-205.ec2.internal\" DevicePath \"\"" Jul 29 18:41:59.812136 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:41:59.812054 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fh54r\" (UniqueName: \"kubernetes.io/projected/62597727-1dca-4b09-a51f-ffbd5df32885-kube-api-access-fh54r\") on node \"ip-10-0-140-205.ec2.internal\" DevicePath \"\"" Jul 29 18:41:59.812136 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:41:59.812069 2560 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/62597727-1dca-4b09-a51f-ffbd5df32885-util\") on node \"ip-10-0-140-205.ec2.internal\" DevicePath \"\"" Jul 29 18:42:00.474556 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:42:00.474527 2560 generic.go:356] "Generic (PLEG): container finished" podID="970cab5f-702a-4e4c-8ea5-b4743aa8718c" containerID="25b529062c62cc0230b5d8a5639725392b83305b1b9fecfa1db33d4b23dfd11b" exitCode=0 Jul 29 18:42:00.474931 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:42:00.474596 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73btqth" event={"ID":"970cab5f-702a-4e4c-8ea5-b4743aa8718c","Type":"ContainerDied","Data":"25b529062c62cc0230b5d8a5639725392b83305b1b9fecfa1db33d4b23dfd11b"} Jul 29 18:42:00.476510 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:42:00.476488 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gvtj8" Jul 29 18:42:00.476653 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:42:00.476525 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gvtj8" event={"ID":"62597727-1dca-4b09-a51f-ffbd5df32885","Type":"ContainerDied","Data":"b1ebb7576b3f2fb51b45e4c105d75d0ab8eb95bbd53a9caf3d88209534fe430b"} Jul 29 18:42:00.476653 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:42:00.476558 2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b1ebb7576b3f2fb51b45e4c105d75d0ab8eb95bbd53a9caf3d88209534fe430b" Jul 29 18:42:00.604511 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:42:00.604485 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0drl9r" Jul 29 18:42:00.625455 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:42:00.625429 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:42:00.718594 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:42:00.718559 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mxx7c\" (UniqueName: \"kubernetes.io/projected/b2a0edd1-d807-4803-b89e-1af7a9d2302d-kube-api-access-mxx7c\") pod \"b2a0edd1-d807-4803-b89e-1af7a9d2302d\" (UID: \"b2a0edd1-d807-4803-b89e-1af7a9d2302d\") " Jul 29 18:42:00.718755 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:42:00.718604 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b2a0edd1-d807-4803-b89e-1af7a9d2302d-bundle\") pod \"b2a0edd1-d807-4803-b89e-1af7a9d2302d\" (UID: \"b2a0edd1-d807-4803-b89e-1af7a9d2302d\") " Jul 29 18:42:00.718755 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:42:00.718691 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b2a0edd1-d807-4803-b89e-1af7a9d2302d-util\") pod \"b2a0edd1-d807-4803-b89e-1af7a9d2302d\" (UID: \"b2a0edd1-d807-4803-b89e-1af7a9d2302d\") " Jul 29 18:42:00.719119 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:42:00.719088 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2a0edd1-d807-4803-b89e-1af7a9d2302d-bundle" (OuterVolumeSpecName: "bundle") pod "b2a0edd1-d807-4803-b89e-1af7a9d2302d" (UID: "b2a0edd1-d807-4803-b89e-1af7a9d2302d"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 18:42:00.720858 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:42:00.720837 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2a0edd1-d807-4803-b89e-1af7a9d2302d-kube-api-access-mxx7c" (OuterVolumeSpecName: "kube-api-access-mxx7c") pod "b2a0edd1-d807-4803-b89e-1af7a9d2302d" (UID: "b2a0edd1-d807-4803-b89e-1af7a9d2302d"). InnerVolumeSpecName "kube-api-access-mxx7c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 18:42:00.724406 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:42:00.724385 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2a0edd1-d807-4803-b89e-1af7a9d2302d-util" (OuterVolumeSpecName: "util") pod "b2a0edd1-d807-4803-b89e-1af7a9d2302d" (UID: "b2a0edd1-d807-4803-b89e-1af7a9d2302d"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 18:42:00.736580 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:42:00.736558 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:42:00.820005 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:42:00.819973 2560 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b2a0edd1-d807-4803-b89e-1af7a9d2302d-util\") on node \"ip-10-0-140-205.ec2.internal\" DevicePath \"\"" Jul 29 18:42:00.820005 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:42:00.819999 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mxx7c\" (UniqueName: \"kubernetes.io/projected/b2a0edd1-d807-4803-b89e-1af7a9d2302d-kube-api-access-mxx7c\") on node \"ip-10-0-140-205.ec2.internal\" DevicePath \"\"" Jul 29 18:42:00.820180 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:42:00.820015 2560 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b2a0edd1-d807-4803-b89e-1af7a9d2302d-bundle\") on node \"ip-10-0-140-205.ec2.internal\" DevicePath \"\"" Jul 29 18:42:01.483174 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:42:01.483146 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0drl9r" Jul 29 18:42:01.483174 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:42:01.483152 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0drl9r" event={"ID":"b2a0edd1-d807-4803-b89e-1af7a9d2302d","Type":"ContainerDied","Data":"51d1b041b1c201536cf9982b3537ac0ffb19bb0ca6c969d51bfef00d0a3ea45f"} Jul 29 18:42:01.483643 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:42:01.483202 2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="51d1b041b1c201536cf9982b3537ac0ffb19bb0ca6c969d51bfef00d0a3ea45f" Jul 29 18:42:01.614595 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:42:01.614541 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73btqth" Jul 29 18:42:01.727575 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:42:01.727547 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xlkbc\" (UniqueName: \"kubernetes.io/projected/970cab5f-702a-4e4c-8ea5-b4743aa8718c-kube-api-access-xlkbc\") pod \"970cab5f-702a-4e4c-8ea5-b4743aa8718c\" (UID: \"970cab5f-702a-4e4c-8ea5-b4743aa8718c\") " Jul 29 18:42:01.727575 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:42:01.727577 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/970cab5f-702a-4e4c-8ea5-b4743aa8718c-util\") pod \"970cab5f-702a-4e4c-8ea5-b4743aa8718c\" (UID: \"970cab5f-702a-4e4c-8ea5-b4743aa8718c\") " Jul 29 18:42:01.727760 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:42:01.727619 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/970cab5f-702a-4e4c-8ea5-b4743aa8718c-bundle\") pod \"970cab5f-702a-4e4c-8ea5-b4743aa8718c\" (UID: \"970cab5f-702a-4e4c-8ea5-b4743aa8718c\") " Jul 29 18:42:01.728228 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:42:01.728199 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/970cab5f-702a-4e4c-8ea5-b4743aa8718c-bundle" (OuterVolumeSpecName: "bundle") pod "970cab5f-702a-4e4c-8ea5-b4743aa8718c" (UID: "970cab5f-702a-4e4c-8ea5-b4743aa8718c"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 18:42:01.729711 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:42:01.729694 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/970cab5f-702a-4e4c-8ea5-b4743aa8718c-kube-api-access-xlkbc" (OuterVolumeSpecName: "kube-api-access-xlkbc") pod "970cab5f-702a-4e4c-8ea5-b4743aa8718c" (UID: "970cab5f-702a-4e4c-8ea5-b4743aa8718c"). InnerVolumeSpecName "kube-api-access-xlkbc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 18:42:01.735422 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:42:01.735399 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/970cab5f-702a-4e4c-8ea5-b4743aa8718c-util" (OuterVolumeSpecName: "util") pod "970cab5f-702a-4e4c-8ea5-b4743aa8718c" (UID: "970cab5f-702a-4e4c-8ea5-b4743aa8718c"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 18:42:01.828407 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:42:01.828375 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xlkbc\" (UniqueName: \"kubernetes.io/projected/970cab5f-702a-4e4c-8ea5-b4743aa8718c-kube-api-access-xlkbc\") on node \"ip-10-0-140-205.ec2.internal\" DevicePath \"\"" Jul 29 18:42:01.828407 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:42:01.828401 2560 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/970cab5f-702a-4e4c-8ea5-b4743aa8718c-util\") on node \"ip-10-0-140-205.ec2.internal\" DevicePath \"\"" Jul 29 18:42:01.828559 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:42:01.828415 2560 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/970cab5f-702a-4e4c-8ea5-b4743aa8718c-bundle\") on node \"ip-10-0-140-205.ec2.internal\" DevicePath \"\"" Jul 29 18:42:02.489449 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:42:02.489422 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73btqth" Jul 29 18:42:02.489818 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:42:02.489449 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73btqth" event={"ID":"970cab5f-702a-4e4c-8ea5-b4743aa8718c","Type":"ContainerDied","Data":"261c80dd318f54ca000623767c99591e1d989f1f7117469464c1683bdf3139fe"} Jul 29 18:42:02.489818 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:42:02.489492 2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="261c80dd318f54ca000623767c99591e1d989f1f7117469464c1683bdf3139fe" Jul 29 18:42:07.789942 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:42:07.789906 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:42:11.298307 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:42:11.298271 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:42:11.699157 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:42:11.699077 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6f54987646-w92xj"] Jul 29 18:42:11.810859 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:42:11.810828 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:42:11.907704 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:42:11.907674 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:42:11.958701 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:42:11.958621 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:42:12.794800 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:42:12.794771 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:42:13.189456 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:42:13.189384 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:42:14.260494 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:42:14.260459 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:42:15.807619 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:42:15.807584 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:42:16.841305 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:42:16.841265 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:42:18.219336 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:42:18.219301 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:42:19.427669 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:42:19.427635 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:42:19.878122 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:42:19.878089 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-pdbsk"] Jul 29 18:42:19.878425 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:42:19.878413 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e746889e-57fd-4aaf-bcec-4616fc712dc2" containerName="util" Jul 29 18:42:19.878472 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:42:19.878427 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="e746889e-57fd-4aaf-bcec-4616fc712dc2" containerName="util" Jul 29 18:42:19.878472 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:42:19.878436 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b2a0edd1-d807-4803-b89e-1af7a9d2302d" containerName="extract" Jul 29 18:42:19.878472 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:42:19.878442 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="b2a0edd1-d807-4803-b89e-1af7a9d2302d" containerName="extract" Jul 29 18:42:19.878472 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:42:19.878450 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62597727-1dca-4b09-a51f-ffbd5df32885" containerName="util" Jul 29 18:42:19.878472 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:42:19.878456 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="62597727-1dca-4b09-a51f-ffbd5df32885" containerName="util" Jul 29 18:42:19.878472 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:42:19.878466 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62597727-1dca-4b09-a51f-ffbd5df32885" containerName="pull" Jul 29 18:42:19.878472 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:42:19.878471 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="62597727-1dca-4b09-a51f-ffbd5df32885" containerName="pull" Jul 29 18:42:19.878678 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:42:19.878477 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62597727-1dca-4b09-a51f-ffbd5df32885" containerName="extract" Jul 29 18:42:19.878678 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:42:19.878483 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="62597727-1dca-4b09-a51f-ffbd5df32885" containerName="extract" Jul 29 18:42:19.878678 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:42:19.878490 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e746889e-57fd-4aaf-bcec-4616fc712dc2" containerName="extract" Jul 29 18:42:19.878678 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:42:19.878495 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="e746889e-57fd-4aaf-bcec-4616fc712dc2" containerName="extract" Jul 29 18:42:19.878678 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:42:19.878501 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="970cab5f-702a-4e4c-8ea5-b4743aa8718c" containerName="util" Jul 29 18:42:19.878678 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:42:19.878506 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="970cab5f-702a-4e4c-8ea5-b4743aa8718c" containerName="util" Jul 29 18:42:19.878678 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:42:19.878512 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b2a0edd1-d807-4803-b89e-1af7a9d2302d" containerName="util" Jul 29 18:42:19.878678 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:42:19.878517 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="b2a0edd1-d807-4803-b89e-1af7a9d2302d" containerName="util" Jul 29 18:42:19.878678 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:42:19.878525 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e746889e-57fd-4aaf-bcec-4616fc712dc2" containerName="pull" Jul 29 18:42:19.878678 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:42:19.878529 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="e746889e-57fd-4aaf-bcec-4616fc712dc2" containerName="pull" Jul 29 18:42:19.878678 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:42:19.878538 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b2a0edd1-d807-4803-b89e-1af7a9d2302d" containerName="pull" Jul 29 18:42:19.878678 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:42:19.878542 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="b2a0edd1-d807-4803-b89e-1af7a9d2302d" containerName="pull" Jul 29 18:42:19.878678 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:42:19.878547 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="970cab5f-702a-4e4c-8ea5-b4743aa8718c" containerName="pull" Jul 29 18:42:19.878678 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:42:19.878552 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="970cab5f-702a-4e4c-8ea5-b4743aa8718c" containerName="pull" Jul 29 18:42:19.878678 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:42:19.878561 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="970cab5f-702a-4e4c-8ea5-b4743aa8718c" containerName="extract" Jul 29 18:42:19.878678 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:42:19.878566 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="970cab5f-702a-4e4c-8ea5-b4743aa8718c" containerName="extract" Jul 29 18:42:19.878678 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:42:19.878615 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="e746889e-57fd-4aaf-bcec-4616fc712dc2" containerName="extract" Jul 29 18:42:19.878678 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:42:19.878624 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="970cab5f-702a-4e4c-8ea5-b4743aa8718c" containerName="extract" Jul 29 18:42:19.878678 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:42:19.878632 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="b2a0edd1-d807-4803-b89e-1af7a9d2302d" containerName="extract" Jul 29 18:42:19.878678 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:42:19.878639 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="62597727-1dca-4b09-a51f-ffbd5df32885" containerName="extract" Jul 29 18:42:19.881269 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:42:19.881254 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-pdbsk" Jul 29 18:42:19.883775 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:42:19.883750 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Jul 29 18:42:19.883921 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:42:19.883753 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-k9jcm\"" Jul 29 18:42:19.883921 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:42:19.883755 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 29 18:42:19.884770 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:42:19.884751 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 29 18:42:19.894014 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:42:19.893993 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-pdbsk"] Jul 29 18:42:19.981171 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:42:19.981137 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qt4x6\" (UniqueName: \"kubernetes.io/projected/1f048c37-df77-43f1-94ba-b2c4cdc10458-kube-api-access-qt4x6\") pod \"dns-operator-controller-manager-864dd78c4c-pdbsk\" (UID: \"1f048c37-df77-43f1-94ba-b2c4cdc10458\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-pdbsk" Jul 29 18:42:20.082068 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:42:20.082035 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qt4x6\" (UniqueName: \"kubernetes.io/projected/1f048c37-df77-43f1-94ba-b2c4cdc10458-kube-api-access-qt4x6\") pod \"dns-operator-controller-manager-864dd78c4c-pdbsk\" (UID: \"1f048c37-df77-43f1-94ba-b2c4cdc10458\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-pdbsk" Jul 29 18:42:20.090876 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:42:20.090858 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qt4x6\" (UniqueName: \"kubernetes.io/projected/1f048c37-df77-43f1-94ba-b2c4cdc10458-kube-api-access-qt4x6\") pod \"dns-operator-controller-manager-864dd78c4c-pdbsk\" (UID: \"1f048c37-df77-43f1-94ba-b2c4cdc10458\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-pdbsk" Jul 29 18:42:20.191266 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:42:20.191201 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-pdbsk" Jul 29 18:42:20.329057 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:42:20.329034 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-pdbsk"] Jul 29 18:42:20.331187 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:42:20.331158 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f048c37_df77_43f1_94ba_b2c4cdc10458.slice/crio-7528b5491b34cf65fb4a227ba5460b3575cabbcbb1eed065c861616bafcefbc3 WatchSource:0}: Error finding container 7528b5491b34cf65fb4a227ba5460b3575cabbcbb1eed065c861616bafcefbc3: Status 404 returned error can't find the container with id 7528b5491b34cf65fb4a227ba5460b3575cabbcbb1eed065c861616bafcefbc3 Jul 29 18:42:20.584036 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:42:20.584005 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-pdbsk" event={"ID":"1f048c37-df77-43f1-94ba-b2c4cdc10458","Type":"ContainerStarted","Data":"7528b5491b34cf65fb4a227ba5460b3575cabbcbb1eed065c861616bafcefbc3"} Jul 29 18:42:21.933489 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:42:21.933458 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-l4f29"] Jul 29 18:42:21.936865 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:42:21.936830 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-l4f29" Jul 29 18:42:21.939433 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:42:21.939398 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Jul 29 18:42:21.939433 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:42:21.939419 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Jul 29 18:42:21.939433 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:42:21.939402 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-bnlth\"" Jul 29 18:42:21.944127 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:42:21.944105 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-l4f29"] Jul 29 18:42:21.958556 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:42:21.958528 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:42:21.999141 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:42:21.999105 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/4fb47286-f9fe-449f-9a3d-bda17d693921-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-l4f29\" (UID: \"4fb47286-f9fe-449f-9a3d-bda17d693921\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-l4f29" Jul 29 18:42:21.999272 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:42:21.999168 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/4fb47286-f9fe-449f-9a3d-bda17d693921-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-l4f29\" (UID: \"4fb47286-f9fe-449f-9a3d-bda17d693921\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-l4f29" Jul 29 18:42:21.999272 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:42:21.999228 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xk7m\" (UniqueName: \"kubernetes.io/projected/4fb47286-f9fe-449f-9a3d-bda17d693921-kube-api-access-9xk7m\") pod \"kuadrant-console-plugin-66b5774985-l4f29\" (UID: \"4fb47286-f9fe-449f-9a3d-bda17d693921\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-l4f29" Jul 29 18:42:22.068696 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:42:22.068665 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:42:22.099872 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:42:22.099840 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/4fb47286-f9fe-449f-9a3d-bda17d693921-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-l4f29\" (UID: \"4fb47286-f9fe-449f-9a3d-bda17d693921\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-l4f29" Jul 29 18:42:22.100034 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:42:22.099899 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/4fb47286-f9fe-449f-9a3d-bda17d693921-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-l4f29\" (UID: \"4fb47286-f9fe-449f-9a3d-bda17d693921\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-l4f29" Jul 29 18:42:22.100034 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:42:22.099944 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xk7m\" (UniqueName: \"kubernetes.io/projected/4fb47286-f9fe-449f-9a3d-bda17d693921-kube-api-access-9xk7m\") pod \"kuadrant-console-plugin-66b5774985-l4f29\" (UID: \"4fb47286-f9fe-449f-9a3d-bda17d693921\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-l4f29" Jul 29 18:42:22.100703 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:42:22.100682 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/4fb47286-f9fe-449f-9a3d-bda17d693921-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-l4f29\" (UID: \"4fb47286-f9fe-449f-9a3d-bda17d693921\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-l4f29" Jul 29 18:42:22.102696 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:42:22.102667 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/4fb47286-f9fe-449f-9a3d-bda17d693921-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-l4f29\" (UID: \"4fb47286-f9fe-449f-9a3d-bda17d693921\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-l4f29" Jul 29 18:42:22.114717 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:42:22.114693 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xk7m\" (UniqueName: \"kubernetes.io/projected/4fb47286-f9fe-449f-9a3d-bda17d693921-kube-api-access-9xk7m\") pod \"kuadrant-console-plugin-66b5774985-l4f29\" (UID: \"4fb47286-f9fe-449f-9a3d-bda17d693921\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-l4f29" Jul 29 18:42:22.156318 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:42:22.156272 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:42:22.249210 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:42:22.249178 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-l4f29" Jul 29 18:42:22.526038 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:42:22.525946 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-l4f29"] Jul 29 18:42:22.527978 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:42:22.527952 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4fb47286_f9fe_449f_9a3d_bda17d693921.slice/crio-3cbf367f6b67f50b27e799b10f848a51bae11459d7c9ab55915e72a0236d6113 WatchSource:0}: Error finding container 3cbf367f6b67f50b27e799b10f848a51bae11459d7c9ab55915e72a0236d6113: Status 404 returned error can't find the container with id 3cbf367f6b67f50b27e799b10f848a51bae11459d7c9ab55915e72a0236d6113 Jul 29 18:42:22.598545 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:42:22.598511 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-pdbsk" event={"ID":"1f048c37-df77-43f1-94ba-b2c4cdc10458","Type":"ContainerStarted","Data":"308248b71d1cd2d1a0b9b1df23abe7a7fe21cc14c496e559a8cc4482faae3a57"} Jul 29 18:42:22.598729 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:42:22.598695 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-pdbsk" Jul 29 18:42:22.599751 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:42:22.599731 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-l4f29" event={"ID":"4fb47286-f9fe-449f-9a3d-bda17d693921","Type":"ContainerStarted","Data":"3cbf367f6b67f50b27e799b10f848a51bae11459d7c9ab55915e72a0236d6113"} Jul 29 18:42:22.620801 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:42:22.620752 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-pdbsk" podStartSLOduration=1.50949905 podStartE2EDuration="3.620740164s" totalImagesPullingTime="2.111241114s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 18:42:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 18:42:22.6186543 +0000 UTC m=+477.844323169" watchObservedRunningTime="2026-07-29 18:42:22.620740164 +0000 UTC m=+477.846409036" Jul 29 18:42:32.092610 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:42:32.092571 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:42:32.256160 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:42:32.256121 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:42:32.454630 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:42:32.454550 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:42:33.608789 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:42:33.608755 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-pdbsk" Jul 29 18:42:35.063935 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:42:35.063879 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:42:35.162971 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:42:35.162940 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:42:36.718186 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:42:36.718146 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-6f54987646-w92xj" podUID="96478f22-2f3c-44c6-b499-6906f50cc643" containerName="console" containerID="cri-o://e532ca10a1e45b3532d6cd1ff76bb9600456a14713a14250378d56a4de2e90be" gracePeriod=15 Jul 29 18:42:37.506668 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:42:37.506629 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:42:37.608591 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:42:37.608559 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:42:43.224434 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:42:43.223847 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:42:43.324580 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:42:43.324539 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:42:45.392795 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:42:45.392755 2560 patch_prober.go:28] interesting pod/console-6f54987646-w92xj container/console namespace/openshift-console: Readiness probe status=failure output="Get \"https://10.134.0.18:8443/health\": dial tcp 10.134.0.18:8443: connect: connection refused" start-of-body= Jul 29 18:42:45.393278 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:42:45.392809 2560 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console/console-6f54987646-w92xj" podUID="96478f22-2f3c-44c6-b499-6906f50cc643" containerName="console" probeResult="failure" output="Get \"https://10.134.0.18:8443/health\": dial tcp 10.134.0.18:8443: connect: connection refused" Jul 29 18:42:45.720622 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:42:45.720601 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6f54987646-w92xj" Jul 29 18:42:45.736484 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:42:45.736459 2560 generic.go:356] "Generic (PLEG): container finished" podID="96478f22-2f3c-44c6-b499-6906f50cc643" containerID="e532ca10a1e45b3532d6cd1ff76bb9600456a14713a14250378d56a4de2e90be" exitCode=0 Jul 29 18:42:45.736610 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:42:45.736499 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6f54987646-w92xj" Jul 29 18:42:45.736610 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:42:45.736539 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6f54987646-w92xj" event={"ID":"96478f22-2f3c-44c6-b499-6906f50cc643","Type":"ContainerDied","Data":"e532ca10a1e45b3532d6cd1ff76bb9600456a14713a14250378d56a4de2e90be"} Jul 29 18:42:45.736610 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:42:45.736583 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6f54987646-w92xj" event={"ID":"96478f22-2f3c-44c6-b499-6906f50cc643","Type":"ContainerDied","Data":"4ddec03e9ddb0a312c4775841a188866d1f42cc1db38388da496cbc121364ed0"} Jul 29 18:42:45.736610 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:42:45.736606 2560 scope.go:117] "RemoveContainer" containerID="e532ca10a1e45b3532d6cd1ff76bb9600456a14713a14250378d56a4de2e90be" Jul 29 18:42:45.738230 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:42:45.738185 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-l4f29" event={"ID":"4fb47286-f9fe-449f-9a3d-bda17d693921","Type":"ContainerStarted","Data":"cd168041540210c6413fa74ab65209d6345ac8b14f1ed6fb9e5cac0c580a9442"} Jul 29 18:42:45.746184 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:42:45.746159 2560 scope.go:117] "RemoveContainer" containerID="e532ca10a1e45b3532d6cd1ff76bb9600456a14713a14250378d56a4de2e90be" Jul 29 18:42:45.746486 ip-10-0-140-205 kubenswrapper[2560]: E0729 18:42:45.746463 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e532ca10a1e45b3532d6cd1ff76bb9600456a14713a14250378d56a4de2e90be\": container with ID starting with e532ca10a1e45b3532d6cd1ff76bb9600456a14713a14250378d56a4de2e90be not found: ID does not exist" containerID="e532ca10a1e45b3532d6cd1ff76bb9600456a14713a14250378d56a4de2e90be" Jul 29 18:42:45.746575 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:42:45.746495 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e532ca10a1e45b3532d6cd1ff76bb9600456a14713a14250378d56a4de2e90be"} err="failed to get container status \"e532ca10a1e45b3532d6cd1ff76bb9600456a14713a14250378d56a4de2e90be\": rpc error: code = NotFound desc = could not find container \"e532ca10a1e45b3532d6cd1ff76bb9600456a14713a14250378d56a4de2e90be\": container with ID starting with e532ca10a1e45b3532d6cd1ff76bb9600456a14713a14250378d56a4de2e90be not found: ID does not exist" Jul 29 18:42:45.764091 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:42:45.764046 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-l4f29" podStartSLOduration=1.6592393269999999 podStartE2EDuration="24.764033915s" totalImagesPullingTime="23.104794588s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 18:42:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 18:42:45.763013847 +0000 UTC m=+500.988682721" watchObservedRunningTime="2026-07-29 18:42:45.764033915 +0000 UTC m=+500.989702786" Jul 29 18:42:45.824757 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:42:45.824733 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-llfj6\" (UniqueName: \"kubernetes.io/projected/96478f22-2f3c-44c6-b499-6906f50cc643-kube-api-access-llfj6\") pod \"96478f22-2f3c-44c6-b499-6906f50cc643\" (UID: \"96478f22-2f3c-44c6-b499-6906f50cc643\") " Jul 29 18:42:45.824925 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:42:45.824765 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/96478f22-2f3c-44c6-b499-6906f50cc643-console-serving-cert\") pod \"96478f22-2f3c-44c6-b499-6906f50cc643\" (UID: \"96478f22-2f3c-44c6-b499-6906f50cc643\") " Jul 29 18:42:45.824925 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:42:45.824785 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/96478f22-2f3c-44c6-b499-6906f50cc643-service-ca\") pod \"96478f22-2f3c-44c6-b499-6906f50cc643\" (UID: \"96478f22-2f3c-44c6-b499-6906f50cc643\") " Jul 29 18:42:45.824925 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:42:45.824802 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/96478f22-2f3c-44c6-b499-6906f50cc643-console-config\") pod \"96478f22-2f3c-44c6-b499-6906f50cc643\" (UID: \"96478f22-2f3c-44c6-b499-6906f50cc643\") " Jul 29 18:42:45.824925 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:42:45.824886 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/96478f22-2f3c-44c6-b499-6906f50cc643-oauth-serving-cert\") pod \"96478f22-2f3c-44c6-b499-6906f50cc643\" (UID: \"96478f22-2f3c-44c6-b499-6906f50cc643\") " Jul 29 18:42:45.825139 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:42:45.824997 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/96478f22-2f3c-44c6-b499-6906f50cc643-trusted-ca-bundle\") pod \"96478f22-2f3c-44c6-b499-6906f50cc643\" (UID: \"96478f22-2f3c-44c6-b499-6906f50cc643\") " Jul 29 18:42:45.825139 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:42:45.825046 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/96478f22-2f3c-44c6-b499-6906f50cc643-console-oauth-config\") pod \"96478f22-2f3c-44c6-b499-6906f50cc643\" (UID: \"96478f22-2f3c-44c6-b499-6906f50cc643\") " Jul 29 18:42:45.825242 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:42:45.825187 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96478f22-2f3c-44c6-b499-6906f50cc643-console-config" (OuterVolumeSpecName: "console-config") pod "96478f22-2f3c-44c6-b499-6906f50cc643" (UID: "96478f22-2f3c-44c6-b499-6906f50cc643"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 18:42:45.825242 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:42:45.825194 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96478f22-2f3c-44c6-b499-6906f50cc643-service-ca" (OuterVolumeSpecName: "service-ca") pod "96478f22-2f3c-44c6-b499-6906f50cc643" (UID: "96478f22-2f3c-44c6-b499-6906f50cc643"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 18:42:45.825465 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:42:45.825436 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96478f22-2f3c-44c6-b499-6906f50cc643-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "96478f22-2f3c-44c6-b499-6906f50cc643" (UID: "96478f22-2f3c-44c6-b499-6906f50cc643"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 18:42:45.825652 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:42:45.825603 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96478f22-2f3c-44c6-b499-6906f50cc643-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "96478f22-2f3c-44c6-b499-6906f50cc643" (UID: "96478f22-2f3c-44c6-b499-6906f50cc643"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 18:42:45.825652 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:42:45.825617 2560 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/96478f22-2f3c-44c6-b499-6906f50cc643-service-ca\") on node \"ip-10-0-140-205.ec2.internal\" DevicePath \"\"" Jul 29 18:42:45.825652 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:42:45.825638 2560 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/96478f22-2f3c-44c6-b499-6906f50cc643-console-config\") on node \"ip-10-0-140-205.ec2.internal\" DevicePath \"\"" Jul 29 18:42:45.827239 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:42:45.827207 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/96478f22-2f3c-44c6-b499-6906f50cc643-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "96478f22-2f3c-44c6-b499-6906f50cc643" (UID: "96478f22-2f3c-44c6-b499-6906f50cc643"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 18:42:45.827601 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:42:45.827579 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/96478f22-2f3c-44c6-b499-6906f50cc643-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "96478f22-2f3c-44c6-b499-6906f50cc643" (UID: "96478f22-2f3c-44c6-b499-6906f50cc643"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 18:42:45.827694 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:42:45.827667 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96478f22-2f3c-44c6-b499-6906f50cc643-kube-api-access-llfj6" (OuterVolumeSpecName: "kube-api-access-llfj6") pod "96478f22-2f3c-44c6-b499-6906f50cc643" (UID: "96478f22-2f3c-44c6-b499-6906f50cc643"). InnerVolumeSpecName "kube-api-access-llfj6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 18:42:45.926998 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:42:45.926932 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-llfj6\" (UniqueName: \"kubernetes.io/projected/96478f22-2f3c-44c6-b499-6906f50cc643-kube-api-access-llfj6\") on node \"ip-10-0-140-205.ec2.internal\" DevicePath \"\"" Jul 29 18:42:45.926998 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:42:45.926959 2560 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/96478f22-2f3c-44c6-b499-6906f50cc643-console-serving-cert\") on node \"ip-10-0-140-205.ec2.internal\" DevicePath \"\"" Jul 29 18:42:45.926998 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:42:45.926972 2560 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/96478f22-2f3c-44c6-b499-6906f50cc643-oauth-serving-cert\") on node \"ip-10-0-140-205.ec2.internal\" DevicePath \"\"" Jul 29 18:42:45.926998 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:42:45.926984 2560 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/96478f22-2f3c-44c6-b499-6906f50cc643-trusted-ca-bundle\") on node \"ip-10-0-140-205.ec2.internal\" DevicePath \"\"" Jul 29 18:42:45.926998 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:42:45.926998 2560 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/96478f22-2f3c-44c6-b499-6906f50cc643-console-oauth-config\") on node \"ip-10-0-140-205.ec2.internal\" DevicePath \"\"" Jul 29 18:42:46.060356 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:42:46.060326 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6f54987646-w92xj"] Jul 29 18:42:46.063442 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:42:46.063421 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6f54987646-w92xj"] Jul 29 18:42:47.354252 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:42:47.354218 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="96478f22-2f3c-44c6-b499-6906f50cc643" path="/var/lib/kubelet/pods/96478f22-2f3c-44c6-b499-6906f50cc643/volumes" Jul 29 18:42:53.415743 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:42:53.415710 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:42:53.517820 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:42:53.517787 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:42:53.663081 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:42:53.663048 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:42:53.765173 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:42:53.765138 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:42:55.425752 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:42:55.425709 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:42:55.527471 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:42:55.527440 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:42:57.866563 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:42:57.866530 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:42:57.929642 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:42:57.929607 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:43:01.373215 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:43:01.373183 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:43:01.466678 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:43:01.466646 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:43:02.991660 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:43:02.991622 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:43:03.004528 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:43:03.004493 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-mnfn2"] Jul 29 18:43:03.004918 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:43:03.004901 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="96478f22-2f3c-44c6-b499-6906f50cc643" containerName="console" Jul 29 18:43:03.005005 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:43:03.004920 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="96478f22-2f3c-44c6-b499-6906f50cc643" containerName="console" Jul 29 18:43:03.005057 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:43:03.005020 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="96478f22-2f3c-44c6-b499-6906f50cc643" containerName="console" Jul 29 18:43:03.036621 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:43:03.036580 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-mnfn2"] Jul 29 18:43:03.036739 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:43:03.036696 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-mnfn2" Jul 29 18:43:03.039401 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:43:03.039379 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 29 18:43:03.095082 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:43:03.095050 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-mnfn2"] Jul 29 18:43:03.102793 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:43:03.102767 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:43:03.154846 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:43:03.154820 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:43:03.174140 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:43:03.174118 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/00599f7d-1b85-456c-968f-fb8870a6fa3a-config-file\") pod \"limitador-limitador-64c94fd64f-mnfn2\" (UID: \"00599f7d-1b85-456c-968f-fb8870a6fa3a\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-mnfn2" Jul 29 18:43:03.174220 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:43:03.174152 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xcnll\" (UniqueName: \"kubernetes.io/projected/00599f7d-1b85-456c-968f-fb8870a6fa3a-kube-api-access-xcnll\") pod \"limitador-limitador-64c94fd64f-mnfn2\" (UID: \"00599f7d-1b85-456c-968f-fb8870a6fa3a\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-mnfn2" Jul 29 18:43:03.275123 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:43:03.275100 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/00599f7d-1b85-456c-968f-fb8870a6fa3a-config-file\") pod \"limitador-limitador-64c94fd64f-mnfn2\" (UID: \"00599f7d-1b85-456c-968f-fb8870a6fa3a\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-mnfn2" Jul 29 18:43:03.275243 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:43:03.275135 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xcnll\" (UniqueName: \"kubernetes.io/projected/00599f7d-1b85-456c-968f-fb8870a6fa3a-kube-api-access-xcnll\") pod \"limitador-limitador-64c94fd64f-mnfn2\" (UID: \"00599f7d-1b85-456c-968f-fb8870a6fa3a\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-mnfn2" Jul 29 18:43:03.275730 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:43:03.275714 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/00599f7d-1b85-456c-968f-fb8870a6fa3a-config-file\") pod \"limitador-limitador-64c94fd64f-mnfn2\" (UID: \"00599f7d-1b85-456c-968f-fb8870a6fa3a\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-mnfn2" Jul 29 18:43:03.283801 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:43:03.283778 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xcnll\" (UniqueName: \"kubernetes.io/projected/00599f7d-1b85-456c-968f-fb8870a6fa3a-kube-api-access-xcnll\") pod \"limitador-limitador-64c94fd64f-mnfn2\" (UID: \"00599f7d-1b85-456c-968f-fb8870a6fa3a\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-mnfn2" Jul 29 18:43:03.346641 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:43:03.346618 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-mnfn2" Jul 29 18:43:03.479054 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:43:03.479030 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-mnfn2"] Jul 29 18:43:03.480719 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:43:03.480692 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00599f7d_1b85_456c_968f_fb8870a6fa3a.slice/crio-594aa8154aa32c5440a962eeb43c2aad8d3975ce906b57b09ac1ffb596d9657d WatchSource:0}: Error finding container 594aa8154aa32c5440a962eeb43c2aad8d3975ce906b57b09ac1ffb596d9657d: Status 404 returned error can't find the container with id 594aa8154aa32c5440a962eeb43c2aad8d3975ce906b57b09ac1ffb596d9657d Jul 29 18:43:03.793303 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:43:03.793270 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-58cd955675-grndh"] Jul 29 18:43:03.796722 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:43:03.796698 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-grndh" Jul 29 18:43:03.798742 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:43:03.798722 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-7x5zw\"" Jul 29 18:43:03.806271 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:43:03.806250 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-grndh"] Jul 29 18:43:03.826162 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:43:03.826134 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:43:03.830824 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:43:03.830797 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-mnfn2" event={"ID":"00599f7d-1b85-456c-968f-fb8870a6fa3a","Type":"ContainerStarted","Data":"594aa8154aa32c5440a962eeb43c2aad8d3975ce906b57b09ac1ffb596d9657d"} Jul 29 18:43:03.887854 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:43:03.887821 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:43:03.981793 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:43:03.981760 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmn9r\" (UniqueName: \"kubernetes.io/projected/bc28c584-5d58-43ac-ba0b-5e684874068b-kube-api-access-jmn9r\") pod \"authorino-58cd955675-grndh\" (UID: \"bc28c584-5d58-43ac-ba0b-5e684874068b\") " pod="kuadrant-system/authorino-58cd955675-grndh" Jul 29 18:43:04.082399 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:43:04.082274 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmn9r\" (UniqueName: \"kubernetes.io/projected/bc28c584-5d58-43ac-ba0b-5e684874068b-kube-api-access-jmn9r\") pod \"authorino-58cd955675-grndh\" (UID: \"bc28c584-5d58-43ac-ba0b-5e684874068b\") " pod="kuadrant-system/authorino-58cd955675-grndh" Jul 29 18:43:04.090107 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:43:04.090080 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmn9r\" (UniqueName: \"kubernetes.io/projected/bc28c584-5d58-43ac-ba0b-5e684874068b-kube-api-access-jmn9r\") pod \"authorino-58cd955675-grndh\" (UID: \"bc28c584-5d58-43ac-ba0b-5e684874068b\") " pod="kuadrant-system/authorino-58cd955675-grndh" Jul 29 18:43:04.105832 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:43:04.105807 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-grndh" Jul 29 18:43:04.234186 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:43:04.234165 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-grndh"] Jul 29 18:43:04.235932 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:43:04.235900 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbc28c584_5d58_43ac_ba0b_5e684874068b.slice/crio-a9cb339e58aa4bd838ebdcdbfb8c2b682ecbfc62a8312c99f60b324e3bdae89a WatchSource:0}: Error finding container a9cb339e58aa4bd838ebdcdbfb8c2b682ecbfc62a8312c99f60b324e3bdae89a: Status 404 returned error can't find the container with id a9cb339e58aa4bd838ebdcdbfb8c2b682ecbfc62a8312c99f60b324e3bdae89a Jul 29 18:43:04.590754 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:43:04.590724 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:43:04.646395 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:43:04.646363 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:43:04.836584 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:43:04.836554 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-grndh" event={"ID":"bc28c584-5d58-43ac-ba0b-5e684874068b","Type":"ContainerStarted","Data":"a9cb339e58aa4bd838ebdcdbfb8c2b682ecbfc62a8312c99f60b324e3bdae89a"} Jul 29 18:43:07.949486 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:43:07.949445 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-grndh"] Jul 29 18:43:09.875167 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:43:09.875124 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-grndh" event={"ID":"bc28c584-5d58-43ac-ba0b-5e684874068b","Type":"ContainerStarted","Data":"f805fa9a2197751a46eda2b797ebdcb3e55356f6bb73152fef18e839de1509be"} Jul 29 18:43:09.875585 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:43:09.875131 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-58cd955675-grndh" podUID="bc28c584-5d58-43ac-ba0b-5e684874068b" containerName="authorino" containerID="cri-o://f805fa9a2197751a46eda2b797ebdcb3e55356f6bb73152fef18e839de1509be" gracePeriod=30 Jul 29 18:43:09.876759 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:43:09.876724 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-mnfn2" event={"ID":"00599f7d-1b85-456c-968f-fb8870a6fa3a","Type":"ContainerStarted","Data":"14dc2ca08a01f3c95bcb98755bb796536239df88457fea4b512cb124a7f33c55"} Jul 29 18:43:09.876881 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:43:09.876793 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-mnfn2" Jul 29 18:43:09.888989 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:43:09.888937 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-58cd955675-grndh" podStartSLOduration=2.184663109 podStartE2EDuration="6.888922399s" totalImagesPullingTime="4.70425929s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 18:43:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 18:43:09.88843098 +0000 UTC m=+525.114099852" watchObservedRunningTime="2026-07-29 18:43:09.888922399 +0000 UTC m=+525.114591268" Jul 29 18:43:09.904320 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:43:09.904276 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-64c94fd64f-mnfn2" podStartSLOduration=2.452248953 podStartE2EDuration="7.904263844s" totalImagesPullingTime="5.452014891s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 18:43:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 18:43:09.90302818 +0000 UTC m=+525.128697056" watchObservedRunningTime="2026-07-29 18:43:09.904263844 +0000 UTC m=+525.129932715" Jul 29 18:43:10.127841 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:43:10.127789 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-grndh" Jul 29 18:43:10.236118 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:43:10.236084 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jmn9r\" (UniqueName: \"kubernetes.io/projected/bc28c584-5d58-43ac-ba0b-5e684874068b-kube-api-access-jmn9r\") pod \"bc28c584-5d58-43ac-ba0b-5e684874068b\" (UID: \"bc28c584-5d58-43ac-ba0b-5e684874068b\") " Jul 29 18:43:10.238237 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:43:10.238204 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc28c584-5d58-43ac-ba0b-5e684874068b-kube-api-access-jmn9r" (OuterVolumeSpecName: "kube-api-access-jmn9r") pod "bc28c584-5d58-43ac-ba0b-5e684874068b" (UID: "bc28c584-5d58-43ac-ba0b-5e684874068b"). InnerVolumeSpecName "kube-api-access-jmn9r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 18:43:10.336939 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:43:10.336905 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jmn9r\" (UniqueName: \"kubernetes.io/projected/bc28c584-5d58-43ac-ba0b-5e684874068b-kube-api-access-jmn9r\") on node \"ip-10-0-140-205.ec2.internal\" DevicePath \"\"" Jul 29 18:43:10.882678 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:43:10.882646 2560 generic.go:356] "Generic (PLEG): container finished" podID="bc28c584-5d58-43ac-ba0b-5e684874068b" containerID="f805fa9a2197751a46eda2b797ebdcb3e55356f6bb73152fef18e839de1509be" exitCode=0 Jul 29 18:43:10.883057 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:43:10.882683 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-grndh" Jul 29 18:43:10.883057 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:43:10.882735 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-grndh" event={"ID":"bc28c584-5d58-43ac-ba0b-5e684874068b","Type":"ContainerDied","Data":"f805fa9a2197751a46eda2b797ebdcb3e55356f6bb73152fef18e839de1509be"} Jul 29 18:43:10.883057 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:43:10.882773 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-grndh" event={"ID":"bc28c584-5d58-43ac-ba0b-5e684874068b","Type":"ContainerDied","Data":"a9cb339e58aa4bd838ebdcdbfb8c2b682ecbfc62a8312c99f60b324e3bdae89a"} Jul 29 18:43:10.883057 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:43:10.882790 2560 scope.go:117] "RemoveContainer" containerID="f805fa9a2197751a46eda2b797ebdcb3e55356f6bb73152fef18e839de1509be" Jul 29 18:43:10.891955 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:43:10.891937 2560 scope.go:117] "RemoveContainer" containerID="f805fa9a2197751a46eda2b797ebdcb3e55356f6bb73152fef18e839de1509be" Jul 29 18:43:10.892219 ip-10-0-140-205 kubenswrapper[2560]: E0729 18:43:10.892201 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f805fa9a2197751a46eda2b797ebdcb3e55356f6bb73152fef18e839de1509be\": container with ID starting with f805fa9a2197751a46eda2b797ebdcb3e55356f6bb73152fef18e839de1509be not found: ID does not exist" containerID="f805fa9a2197751a46eda2b797ebdcb3e55356f6bb73152fef18e839de1509be" Jul 29 18:43:10.892261 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:43:10.892226 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f805fa9a2197751a46eda2b797ebdcb3e55356f6bb73152fef18e839de1509be"} err="failed to get container status \"f805fa9a2197751a46eda2b797ebdcb3e55356f6bb73152fef18e839de1509be\": rpc error: code = NotFound desc = could not find container \"f805fa9a2197751a46eda2b797ebdcb3e55356f6bb73152fef18e839de1509be\": container with ID starting with f805fa9a2197751a46eda2b797ebdcb3e55356f6bb73152fef18e839de1509be not found: ID does not exist" Jul 29 18:43:10.903140 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:43:10.903118 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-grndh"] Jul 29 18:43:10.906915 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:43:10.906896 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-58cd955675-grndh"] Jul 29 18:43:11.353688 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:43:11.353653 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bc28c584-5d58-43ac-ba0b-5e684874068b" path="/var/lib/kubelet/pods/bc28c584-5d58-43ac-ba0b-5e684874068b/volumes" Jul 29 18:43:17.940176 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:43:17.940097 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-mnfn2"] Jul 29 18:43:17.940604 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:43:17.940262 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-64c94fd64f-mnfn2" podUID="00599f7d-1b85-456c-968f-fb8870a6fa3a" containerName="limitador" containerID="cri-o://14dc2ca08a01f3c95bcb98755bb796536239df88457fea4b512cb124a7f33c55" gracePeriod=30 Jul 29 18:43:17.942553 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:43:17.942524 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-mnfn2" Jul 29 18:43:18.524593 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:43:18.524530 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-mnfn2" Jul 29 18:43:18.602891 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:43:18.602863 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/00599f7d-1b85-456c-968f-fb8870a6fa3a-config-file\") pod \"00599f7d-1b85-456c-968f-fb8870a6fa3a\" (UID: \"00599f7d-1b85-456c-968f-fb8870a6fa3a\") " Jul 29 18:43:18.602891 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:43:18.602898 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xcnll\" (UniqueName: \"kubernetes.io/projected/00599f7d-1b85-456c-968f-fb8870a6fa3a-kube-api-access-xcnll\") pod \"00599f7d-1b85-456c-968f-fb8870a6fa3a\" (UID: \"00599f7d-1b85-456c-968f-fb8870a6fa3a\") " Jul 29 18:43:18.603288 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:43:18.603262 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00599f7d-1b85-456c-968f-fb8870a6fa3a-config-file" (OuterVolumeSpecName: "config-file") pod "00599f7d-1b85-456c-968f-fb8870a6fa3a" (UID: "00599f7d-1b85-456c-968f-fb8870a6fa3a"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 18:43:18.605105 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:43:18.605080 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00599f7d-1b85-456c-968f-fb8870a6fa3a-kube-api-access-xcnll" (OuterVolumeSpecName: "kube-api-access-xcnll") pod "00599f7d-1b85-456c-968f-fb8870a6fa3a" (UID: "00599f7d-1b85-456c-968f-fb8870a6fa3a"). InnerVolumeSpecName "kube-api-access-xcnll". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 18:43:18.704074 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:43:18.704045 2560 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/00599f7d-1b85-456c-968f-fb8870a6fa3a-config-file\") on node \"ip-10-0-140-205.ec2.internal\" DevicePath \"\"" Jul 29 18:43:18.704074 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:43:18.704068 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xcnll\" (UniqueName: \"kubernetes.io/projected/00599f7d-1b85-456c-968f-fb8870a6fa3a-kube-api-access-xcnll\") on node \"ip-10-0-140-205.ec2.internal\" DevicePath \"\"" Jul 29 18:43:18.925483 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:43:18.925403 2560 generic.go:356] "Generic (PLEG): container finished" podID="00599f7d-1b85-456c-968f-fb8870a6fa3a" containerID="14dc2ca08a01f3c95bcb98755bb796536239df88457fea4b512cb124a7f33c55" exitCode=0 Jul 29 18:43:18.925483 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:43:18.925449 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-mnfn2" Jul 29 18:43:18.925680 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:43:18.925489 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-mnfn2" event={"ID":"00599f7d-1b85-456c-968f-fb8870a6fa3a","Type":"ContainerDied","Data":"14dc2ca08a01f3c95bcb98755bb796536239df88457fea4b512cb124a7f33c55"} Jul 29 18:43:18.925680 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:43:18.925571 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-mnfn2" event={"ID":"00599f7d-1b85-456c-968f-fb8870a6fa3a","Type":"ContainerDied","Data":"594aa8154aa32c5440a962eeb43c2aad8d3975ce906b57b09ac1ffb596d9657d"} Jul 29 18:43:18.925680 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:43:18.925598 2560 scope.go:117] "RemoveContainer" containerID="14dc2ca08a01f3c95bcb98755bb796536239df88457fea4b512cb124a7f33c55" Jul 29 18:43:18.934890 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:43:18.934873 2560 scope.go:117] "RemoveContainer" containerID="14dc2ca08a01f3c95bcb98755bb796536239df88457fea4b512cb124a7f33c55" Jul 29 18:43:18.935143 ip-10-0-140-205 kubenswrapper[2560]: E0729 18:43:18.935124 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14dc2ca08a01f3c95bcb98755bb796536239df88457fea4b512cb124a7f33c55\": container with ID starting with 14dc2ca08a01f3c95bcb98755bb796536239df88457fea4b512cb124a7f33c55 not found: ID does not exist" containerID="14dc2ca08a01f3c95bcb98755bb796536239df88457fea4b512cb124a7f33c55" Jul 29 18:43:18.935201 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:43:18.935150 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14dc2ca08a01f3c95bcb98755bb796536239df88457fea4b512cb124a7f33c55"} err="failed to get container status \"14dc2ca08a01f3c95bcb98755bb796536239df88457fea4b512cb124a7f33c55\": rpc error: code = NotFound desc = could not find container \"14dc2ca08a01f3c95bcb98755bb796536239df88457fea4b512cb124a7f33c55\": container with ID starting with 14dc2ca08a01f3c95bcb98755bb796536239df88457fea4b512cb124a7f33c55 not found: ID does not exist" Jul 29 18:43:18.946313 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:43:18.946288 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-mnfn2"] Jul 29 18:43:18.949681 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:43:18.949663 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-mnfn2"] Jul 29 18:43:19.346227 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:43:19.346187 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:43:19.353760 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:43:19.353719 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="00599f7d-1b85-456c-968f-fb8870a6fa3a" path="/var/lib/kubelet/pods/00599f7d-1b85-456c-968f-fb8870a6fa3a/volumes" Jul 29 18:43:19.465400 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:43:19.465308 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:43:19.862094 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:43:19.862058 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-kxnkt"] Jul 29 18:43:19.862639 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:43:19.862615 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="00599f7d-1b85-456c-968f-fb8870a6fa3a" containerName="limitador" Jul 29 18:43:19.862639 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:43:19.862641 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="00599f7d-1b85-456c-968f-fb8870a6fa3a" containerName="limitador" Jul 29 18:43:19.862789 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:43:19.862670 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bc28c584-5d58-43ac-ba0b-5e684874068b" containerName="authorino" Jul 29 18:43:19.862789 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:43:19.862679 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc28c584-5d58-43ac-ba0b-5e684874068b" containerName="authorino" Jul 29 18:43:19.862789 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:43:19.862764 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="bc28c584-5d58-43ac-ba0b-5e684874068b" containerName="authorino" Jul 29 18:43:19.862789 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:43:19.862774 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="00599f7d-1b85-456c-968f-fb8870a6fa3a" containerName="limitador" Jul 29 18:43:19.878673 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:43:19.878649 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-kxnkt"] Jul 29 18:43:19.878827 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:43:19.878734 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-kxnkt" Jul 29 18:43:19.881464 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:43:19.881441 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-48k5p\"" Jul 29 18:43:19.881582 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:43:19.881513 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Jul 29 18:43:19.882254 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:43:19.882234 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 29 18:43:19.882340 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:43:19.882242 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 29 18:43:19.913785 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:43:19.913761 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cqr6m\" (UniqueName: \"kubernetes.io/projected/7b8c910c-6556-420a-a28d-5fe83264e80c-kube-api-access-cqr6m\") pod \"postgres-69fb4594cf-kxnkt\" (UID: \"7b8c910c-6556-420a-a28d-5fe83264e80c\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-kxnkt" Jul 29 18:43:19.913880 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:43:19.913811 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/7b8c910c-6556-420a-a28d-5fe83264e80c-data\") pod \"postgres-69fb4594cf-kxnkt\" (UID: \"7b8c910c-6556-420a-a28d-5fe83264e80c\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-kxnkt" Jul 29 18:43:19.931321 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:43:19.931297 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:43:20.014627 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:43:20.014602 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/7b8c910c-6556-420a-a28d-5fe83264e80c-data\") pod \"postgres-69fb4594cf-kxnkt\" (UID: \"7b8c910c-6556-420a-a28d-5fe83264e80c\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-kxnkt" Jul 29 18:43:20.014922 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:43:20.014682 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqr6m\" (UniqueName: \"kubernetes.io/projected/7b8c910c-6556-420a-a28d-5fe83264e80c-kube-api-access-cqr6m\") pod \"postgres-69fb4594cf-kxnkt\" (UID: \"7b8c910c-6556-420a-a28d-5fe83264e80c\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-kxnkt" Jul 29 18:43:20.014968 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:43:20.014921 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/7b8c910c-6556-420a-a28d-5fe83264e80c-data\") pod \"postgres-69fb4594cf-kxnkt\" (UID: \"7b8c910c-6556-420a-a28d-5fe83264e80c\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-kxnkt" Jul 29 18:43:20.022732 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:43:20.022709 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqr6m\" (UniqueName: \"kubernetes.io/projected/7b8c910c-6556-420a-a28d-5fe83264e80c-kube-api-access-cqr6m\") pod \"postgres-69fb4594cf-kxnkt\" (UID: \"7b8c910c-6556-420a-a28d-5fe83264e80c\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-kxnkt" Jul 29 18:43:20.189126 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:43:20.189055 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-kxnkt" Jul 29 18:43:20.318561 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:43:20.318527 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-kxnkt"] Jul 29 18:43:20.321966 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:43:20.321944 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b8c910c_6556_420a_a28d_5fe83264e80c.slice/crio-9152603f0b30a2a5835a8d01a70f0db1dbcb18f01ef3b21627c4ceb11d3fb0b5 WatchSource:0}: Error finding container 9152603f0b30a2a5835a8d01a70f0db1dbcb18f01ef3b21627c4ceb11d3fb0b5: Status 404 returned error can't find the container with id 9152603f0b30a2a5835a8d01a70f0db1dbcb18f01ef3b21627c4ceb11d3fb0b5 Jul 29 18:43:20.936631 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:43:20.936600 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-kxnkt" event={"ID":"7b8c910c-6556-420a-a28d-5fe83264e80c","Type":"ContainerStarted","Data":"9152603f0b30a2a5835a8d01a70f0db1dbcb18f01ef3b21627c4ceb11d3fb0b5"} Jul 29 18:43:25.356981 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:43:25.356962 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Jul 29 18:43:25.969440 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:43:25.969398 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-kxnkt" event={"ID":"7b8c910c-6556-420a-a28d-5fe83264e80c","Type":"ContainerStarted","Data":"55b1da1be13a52f9e6836e8d0e2a48f8f1d8313c65d42a52bef0399ebf91e205"} Jul 29 18:43:25.969605 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:43:25.969552 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-kxnkt" Jul 29 18:43:25.985184 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:43:25.985140 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-kxnkt" podStartSLOduration=1.9542238250000001 podStartE2EDuration="6.985123804s" totalImagesPullingTime="5.030899979s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 18:43:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 18:43:25.983986706 +0000 UTC m=+541.209655577" watchObservedRunningTime="2026-07-29 18:43:25.985123804 +0000 UTC m=+541.210792677" Jul 29 18:43:26.213942 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:43:26.213910 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:43:32.002770 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:43:32.002746 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-kxnkt" Jul 29 18:43:32.331783 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:43:32.331677 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:43:32.453494 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:43:32.453451 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:43:32.569489 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:43:32.569451 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5d4b67c446-m4m64"] Jul 29 18:43:32.572979 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:43:32.572960 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-m4m64" Jul 29 18:43:32.575045 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:43:32.575025 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-7x5zw\"" Jul 29 18:43:32.579115 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:43:32.579086 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-m4m64"] Jul 29 18:43:32.740255 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:43:32.740175 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9h86h\" (UniqueName: \"kubernetes.io/projected/d0d0c222-509f-43c1-83c6-6013f8c731df-kube-api-access-9h86h\") pod \"authorino-5d4b67c446-m4m64\" (UID: \"d0d0c222-509f-43c1-83c6-6013f8c731df\") " pod="kuadrant-system/authorino-5d4b67c446-m4m64" Jul 29 18:43:32.803088 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:43:32.803063 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-m4m64"] Jul 29 18:43:32.803309 ip-10-0-140-205 kubenswrapper[2560]: E0729 18:43:32.803258 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-9h86h], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-5d4b67c446-m4m64" podUID="d0d0c222-509f-43c1-83c6-6013f8c731df" Jul 29 18:43:32.832010 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:43:32.831981 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-79599897b8-jj46z"] Jul 29 18:43:32.835315 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:43:32.835296 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-79599897b8-jj46z" Jul 29 18:43:32.837445 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:43:32.837427 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Jul 29 18:43:32.840598 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:43:32.840577 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9h86h\" (UniqueName: \"kubernetes.io/projected/d0d0c222-509f-43c1-83c6-6013f8c731df-kube-api-access-9h86h\") pod \"authorino-5d4b67c446-m4m64\" (UID: \"d0d0c222-509f-43c1-83c6-6013f8c731df\") " pod="kuadrant-system/authorino-5d4b67c446-m4m64" Jul 29 18:43:32.843805 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:43:32.843787 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-79599897b8-jj46z"] Jul 29 18:43:32.850771 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:43:32.850750 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9h86h\" (UniqueName: \"kubernetes.io/projected/d0d0c222-509f-43c1-83c6-6013f8c731df-kube-api-access-9h86h\") pod \"authorino-5d4b67c446-m4m64\" (UID: \"d0d0c222-509f-43c1-83c6-6013f8c731df\") " pod="kuadrant-system/authorino-5d4b67c446-m4m64" Jul 29 18:43:32.941244 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:43:32.941214 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4wwg\" (UniqueName: \"kubernetes.io/projected/70c8de05-3b10-4ff8-9a02-22b7738470e5-kube-api-access-m4wwg\") pod \"authorino-79599897b8-jj46z\" (UID: \"70c8de05-3b10-4ff8-9a02-22b7738470e5\") " pod="kuadrant-system/authorino-79599897b8-jj46z" Jul 29 18:43:32.941378 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:43:32.941247 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/70c8de05-3b10-4ff8-9a02-22b7738470e5-tls-cert\") pod \"authorino-79599897b8-jj46z\" (UID: \"70c8de05-3b10-4ff8-9a02-22b7738470e5\") " pod="kuadrant-system/authorino-79599897b8-jj46z" Jul 29 18:43:33.010582 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:43:33.010554 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-m4m64" Jul 29 18:43:33.017615 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:43:33.017594 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-m4m64" Jul 29 18:43:33.042128 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:43:33.042094 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4wwg\" (UniqueName: \"kubernetes.io/projected/70c8de05-3b10-4ff8-9a02-22b7738470e5-kube-api-access-m4wwg\") pod \"authorino-79599897b8-jj46z\" (UID: \"70c8de05-3b10-4ff8-9a02-22b7738470e5\") " pod="kuadrant-system/authorino-79599897b8-jj46z" Jul 29 18:43:33.042252 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:43:33.042140 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/70c8de05-3b10-4ff8-9a02-22b7738470e5-tls-cert\") pod \"authorino-79599897b8-jj46z\" (UID: \"70c8de05-3b10-4ff8-9a02-22b7738470e5\") " pod="kuadrant-system/authorino-79599897b8-jj46z" Jul 29 18:43:33.044632 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:43:33.044617 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/70c8de05-3b10-4ff8-9a02-22b7738470e5-tls-cert\") pod \"authorino-79599897b8-jj46z\" (UID: \"70c8de05-3b10-4ff8-9a02-22b7738470e5\") " pod="kuadrant-system/authorino-79599897b8-jj46z" Jul 29 18:43:33.049427 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:43:33.049410 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4wwg\" (UniqueName: \"kubernetes.io/projected/70c8de05-3b10-4ff8-9a02-22b7738470e5-kube-api-access-m4wwg\") pod \"authorino-79599897b8-jj46z\" (UID: \"70c8de05-3b10-4ff8-9a02-22b7738470e5\") " pod="kuadrant-system/authorino-79599897b8-jj46z" Jul 29 18:43:33.142888 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:43:33.142856 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9h86h\" (UniqueName: \"kubernetes.io/projected/d0d0c222-509f-43c1-83c6-6013f8c731df-kube-api-access-9h86h\") pod \"d0d0c222-509f-43c1-83c6-6013f8c731df\" (UID: \"d0d0c222-509f-43c1-83c6-6013f8c731df\") " Jul 29 18:43:33.144913 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:43:33.144891 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0d0c222-509f-43c1-83c6-6013f8c731df-kube-api-access-9h86h" (OuterVolumeSpecName: "kube-api-access-9h86h") pod "d0d0c222-509f-43c1-83c6-6013f8c731df" (UID: "d0d0c222-509f-43c1-83c6-6013f8c731df"). InnerVolumeSpecName "kube-api-access-9h86h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 18:43:33.145000 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:43:33.144921 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-79599897b8-jj46z" Jul 29 18:43:33.243900 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:43:33.243872 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9h86h\" (UniqueName: \"kubernetes.io/projected/d0d0c222-509f-43c1-83c6-6013f8c731df-kube-api-access-9h86h\") on node \"ip-10-0-140-205.ec2.internal\" DevicePath \"\"" Jul 29 18:43:33.273455 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:43:33.273431 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-79599897b8-jj46z"] Jul 29 18:43:33.275504 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:43:33.275473 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod70c8de05_3b10_4ff8_9a02_22b7738470e5.slice/crio-14e2b9e9d9c857d705711daf014d45b2e30c977e678a638f622673fdf635bffe WatchSource:0}: Error finding container 14e2b9e9d9c857d705711daf014d45b2e30c977e678a638f622673fdf635bffe: Status 404 returned error can't find the container with id 14e2b9e9d9c857d705711daf014d45b2e30c977e678a638f622673fdf635bffe Jul 29 18:43:34.016750 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:43:34.016715 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-79599897b8-jj46z" event={"ID":"70c8de05-3b10-4ff8-9a02-22b7738470e5","Type":"ContainerStarted","Data":"4ba12611ab38590646bad0c722b40b902a18904c18a46bd0fbb65c64d3327cd5"} Jul 29 18:43:34.016750 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:43:34.016754 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-79599897b8-jj46z" event={"ID":"70c8de05-3b10-4ff8-9a02-22b7738470e5","Type":"ContainerStarted","Data":"14e2b9e9d9c857d705711daf014d45b2e30c977e678a638f622673fdf635bffe"} Jul 29 18:43:34.017230 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:43:34.016756 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-m4m64" Jul 29 18:43:34.032822 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:43:34.032779 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-79599897b8-jj46z" podStartSLOduration=1.7127045669999998 podStartE2EDuration="2.032765973s" totalImagesPullingTime="320.061406ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 18:43:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 18:43:34.031220294 +0000 UTC m=+549.256889172" watchObservedRunningTime="2026-07-29 18:43:34.032765973 +0000 UTC m=+549.258434844" Jul 29 18:43:34.055250 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:43:34.055222 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-m4m64"] Jul 29 18:43:34.063048 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:43:34.063024 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-m4m64"] Jul 29 18:43:35.353592 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:43:35.353561 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d0d0c222-509f-43c1-83c6-6013f8c731df" path="/var/lib/kubelet/pods/d0d0c222-509f-43c1-83c6-6013f8c731df/volumes" Jul 29 18:43:39.118083 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:43:39.118043 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:43:39.215884 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:43:39.215847 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:43:39.308080 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:43:39.308050 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:43:39.468112 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:43:39.467997 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:43:42.285772 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:43:42.285734 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:43:42.390382 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:43:42.390337 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:43:50.882551 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:43:50.882510 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:43:50.936363 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:43:50.936274 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:44:13.895922 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:44:13.895888 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:44:14.002364 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:44:14.002320 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:44:14.568588 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:44:14.568560 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:44:14.668922 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:44:14.668889 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:44:14.772950 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:44:14.772923 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:44:14.854860 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:44:14.854771 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:44:15.242429 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:44:15.242289 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:44:15.324465 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:44:15.324416 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:44:18.911768 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:44:18.911724 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:44:18.981365 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:44:18.981297 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:44:25.342603 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:44:25.342508 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-v4n5x_1727fb40-fbed-4984-91d4-a5890d0f4103/ovn-acl-logging/0.log" Jul 29 18:44:25.345793 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:44:25.342909 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-v4n5x_1727fb40-fbed-4984-91d4-a5890d0f4103/ovn-acl-logging/0.log" Jul 29 18:44:27.363403 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:44:27.363338 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:44:29.486029 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:44:29.485992 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:44:33.888776 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:44:33.888742 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qcvt8"] Jul 29 18:44:33.892240 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:44:33.892224 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qcvt8" Jul 29 18:44:33.894840 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:44:33.894817 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 29 18:44:33.894973 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:44:33.894822 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 29 18:44:33.895717 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:44:33.895696 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Jul 29 18:44:33.895825 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:44:33.895743 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-479pt\"" Jul 29 18:44:33.901670 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:44:33.901646 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qcvt8"] Jul 29 18:44:33.904926 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:44:33.904902 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:44:33.959180 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:44:33.959150 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/031eec58-a223-4fd8-9faf-5f64c76554b1-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-qcvt8\" (UID: \"031eec58-a223-4fd8-9faf-5f64c76554b1\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qcvt8" Jul 29 18:44:33.959341 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:44:33.959198 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/031eec58-a223-4fd8-9faf-5f64c76554b1-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-qcvt8\" (UID: \"031eec58-a223-4fd8-9faf-5f64c76554b1\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qcvt8" Jul 29 18:44:33.959341 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:44:33.959287 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xn698\" (UniqueName: \"kubernetes.io/projected/031eec58-a223-4fd8-9faf-5f64c76554b1-kube-api-access-xn698\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-qcvt8\" (UID: \"031eec58-a223-4fd8-9faf-5f64c76554b1\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qcvt8" Jul 29 18:44:33.959341 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:44:33.959314 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/031eec58-a223-4fd8-9faf-5f64c76554b1-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-qcvt8\" (UID: \"031eec58-a223-4fd8-9faf-5f64c76554b1\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qcvt8" Jul 29 18:44:33.959341 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:44:33.959341 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/031eec58-a223-4fd8-9faf-5f64c76554b1-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-qcvt8\" (UID: \"031eec58-a223-4fd8-9faf-5f64c76554b1\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qcvt8" Jul 29 18:44:33.959537 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:44:33.959443 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/031eec58-a223-4fd8-9faf-5f64c76554b1-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-qcvt8\" (UID: \"031eec58-a223-4fd8-9faf-5f64c76554b1\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qcvt8" Jul 29 18:44:34.060628 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:44:34.060596 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/031eec58-a223-4fd8-9faf-5f64c76554b1-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-qcvt8\" (UID: \"031eec58-a223-4fd8-9faf-5f64c76554b1\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qcvt8" Jul 29 18:44:34.060817 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:44:34.060665 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/031eec58-a223-4fd8-9faf-5f64c76554b1-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-qcvt8\" (UID: \"031eec58-a223-4fd8-9faf-5f64c76554b1\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qcvt8" Jul 29 18:44:34.060817 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:44:34.060687 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/031eec58-a223-4fd8-9faf-5f64c76554b1-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-qcvt8\" (UID: \"031eec58-a223-4fd8-9faf-5f64c76554b1\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qcvt8" Jul 29 18:44:34.060817 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:44:34.060720 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/031eec58-a223-4fd8-9faf-5f64c76554b1-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-qcvt8\" (UID: \"031eec58-a223-4fd8-9faf-5f64c76554b1\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qcvt8" Jul 29 18:44:34.060817 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:44:34.060759 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xn698\" (UniqueName: \"kubernetes.io/projected/031eec58-a223-4fd8-9faf-5f64c76554b1-kube-api-access-xn698\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-qcvt8\" (UID: \"031eec58-a223-4fd8-9faf-5f64c76554b1\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qcvt8" Jul 29 18:44:34.060817 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:44:34.060784 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/031eec58-a223-4fd8-9faf-5f64c76554b1-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-qcvt8\" (UID: \"031eec58-a223-4fd8-9faf-5f64c76554b1\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qcvt8" Jul 29 18:44:34.061157 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:44:34.061137 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/031eec58-a223-4fd8-9faf-5f64c76554b1-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-qcvt8\" (UID: \"031eec58-a223-4fd8-9faf-5f64c76554b1\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qcvt8" Jul 29 18:44:34.061199 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:44:34.061162 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/031eec58-a223-4fd8-9faf-5f64c76554b1-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-qcvt8\" (UID: \"031eec58-a223-4fd8-9faf-5f64c76554b1\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qcvt8" Jul 29 18:44:34.061239 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:44:34.061196 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/031eec58-a223-4fd8-9faf-5f64c76554b1-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-qcvt8\" (UID: \"031eec58-a223-4fd8-9faf-5f64c76554b1\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qcvt8" Jul 29 18:44:34.063049 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:44:34.063028 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/031eec58-a223-4fd8-9faf-5f64c76554b1-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-qcvt8\" (UID: \"031eec58-a223-4fd8-9faf-5f64c76554b1\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qcvt8" Jul 29 18:44:34.063392 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:44:34.063375 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/031eec58-a223-4fd8-9faf-5f64c76554b1-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-qcvt8\" (UID: \"031eec58-a223-4fd8-9faf-5f64c76554b1\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qcvt8" Jul 29 18:44:34.068260 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:44:34.068236 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xn698\" (UniqueName: \"kubernetes.io/projected/031eec58-a223-4fd8-9faf-5f64c76554b1-kube-api-access-xn698\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-qcvt8\" (UID: \"031eec58-a223-4fd8-9faf-5f64c76554b1\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qcvt8" Jul 29 18:44:34.203911 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:44:34.203831 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qcvt8" Jul 29 18:44:34.336666 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:44:34.334279 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qcvt8"] Jul 29 18:44:34.339227 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:44:34.339198 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod031eec58_a223_4fd8_9faf_5f64c76554b1.slice/crio-40a9e20d3cfa0c554c84c936f8c7c1627de32d632e1239d8f9a8478764b2de0c WatchSource:0}: Error finding container 40a9e20d3cfa0c554c84c936f8c7c1627de32d632e1239d8f9a8478764b2de0c: Status 404 returned error can't find the container with id 40a9e20d3cfa0c554c84c936f8c7c1627de32d632e1239d8f9a8478764b2de0c Jul 29 18:44:34.340825 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:44:34.340797 2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 29 18:44:35.346381 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:44:35.346323 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qcvt8" event={"ID":"031eec58-a223-4fd8-9faf-5f64c76554b1","Type":"ContainerStarted","Data":"40a9e20d3cfa0c554c84c936f8c7c1627de32d632e1239d8f9a8478764b2de0c"} Jul 29 18:44:36.352766 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:44:36.352733 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qcvt8" event={"ID":"031eec58-a223-4fd8-9faf-5f64c76554b1","Type":"ContainerStarted","Data":"81b280eb30f1f2e22170eb5eadf2e9b1dbd3df56991992a26cee12e0bfe578b8"} Jul 29 18:44:36.372124 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:44:36.372074 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qcvt8" podStartSLOduration=2.022835294 podStartE2EDuration="3.372061687s" totalImagesPullingTime="1.349226393s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 18:44:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 18:44:36.37001996 +0000 UTC m=+611.595688831" watchObservedRunningTime="2026-07-29 18:44:36.372061687 +0000 UTC m=+611.597730558" Jul 29 18:44:44.204781 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:44:44.204741 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qcvt8" Jul 29 18:44:44.205289 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:44:44.204808 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qcvt8" Jul 29 18:44:44.217227 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:44:44.217199 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qcvt8" Jul 29 18:44:44.403301 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:44:44.403272 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qcvt8" Jul 29 18:44:52.300915 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:44:52.300833 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:45:00.144645 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:45:00.144610 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29755845-gfxfs"] Jul 29 18:45:00.151047 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:45:00.151030 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29755845-gfxfs" Jul 29 18:45:00.153540 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:45:00.153515 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vgb9z\"" Jul 29 18:45:00.155192 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:45:00.155167 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29755845-gfxfs"] Jul 29 18:45:00.162322 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:45:00.162303 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29755845-gfxfs" Jul 29 18:45:00.295168 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:45:00.295141 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29755845-gfxfs"] Jul 29 18:45:00.297189 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:45:00.297158 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd13589c_fe74_4332_8fdd_dec85585f01c.slice/crio-f4f9c857a4084f628b03a874bc9943cc364f14ba3f546156a938b11f9799ebbc WatchSource:0}: Error finding container f4f9c857a4084f628b03a874bc9943cc364f14ba3f546156a938b11f9799ebbc: Status 404 returned error can't find the container with id f4f9c857a4084f628b03a874bc9943cc364f14ba3f546156a938b11f9799ebbc Jul 29 18:45:00.482328 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:45:00.482250 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29755845-gfxfs" event={"ID":"bd13589c-fe74-4332-8fdd-dec85585f01c","Type":"ContainerStarted","Data":"f4f9c857a4084f628b03a874bc9943cc364f14ba3f546156a938b11f9799ebbc"} Jul 29 18:45:04.508398 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:45:04.508358 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29755845-gfxfs" event={"ID":"bd13589c-fe74-4332-8fdd-dec85585f01c","Type":"ContainerStarted","Data":"033263efa9fc9959670587674a24012eb160f24998f0b704ffe46f7fb6f43cdf"} Jul 29 18:45:04.525290 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:45:04.525244 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29755845-gfxfs" podStartSLOduration=1.077005274 podStartE2EDuration="4.525229145s" totalImagesPullingTime="3.448223871s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 18:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 18:45:04.522898825 +0000 UTC m=+639.748567699" watchObservedRunningTime="2026-07-29 18:45:04.525229145 +0000 UTC m=+639.750898017" Jul 29 18:45:05.515157 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:45:05.515116 2560 generic.go:356] "Generic (PLEG): container finished" podID="bd13589c-fe74-4332-8fdd-dec85585f01c" containerID="033263efa9fc9959670587674a24012eb160f24998f0b704ffe46f7fb6f43cdf" exitCode=0 Jul 29 18:45:05.515566 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:45:05.515185 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29755845-gfxfs" event={"ID":"bd13589c-fe74-4332-8fdd-dec85585f01c","Type":"ContainerDied","Data":"033263efa9fc9959670587674a24012eb160f24998f0b704ffe46f7fb6f43cdf"} Jul 29 18:45:06.645996 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:45:06.645972 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29755845-gfxfs" Jul 29 18:45:06.872363 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:45:06.872267 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-wbn2l"] Jul 29 18:45:06.872652 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:45:06.872640 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd13589c-fe74-4332-8fdd-dec85585f01c" containerName="cleanup" Jul 29 18:45:06.872695 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:45:06.872654 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd13589c-fe74-4332-8fdd-dec85585f01c" containerName="cleanup" Jul 29 18:45:06.872730 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:45:06.872715 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="bd13589c-fe74-4332-8fdd-dec85585f01c" containerName="cleanup" Jul 29 18:45:06.875706 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:45:06.875690 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-wbn2l" Jul 29 18:45:06.878192 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:45:06.878175 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Jul 29 18:45:06.885471 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:45:06.885444 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-wbn2l"] Jul 29 18:45:06.907697 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:45:06.907666 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:45:06.941788 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:45:06.941765 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b5f1c834-97e4-4e1c-9d03-f10adaa9c8b9-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-wbn2l\" (UID: \"b5f1c834-97e4-4e1c-9d03-f10adaa9c8b9\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-wbn2l" Jul 29 18:45:06.941919 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:45:06.941805 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cp7pm\" (UniqueName: \"kubernetes.io/projected/b5f1c834-97e4-4e1c-9d03-f10adaa9c8b9-kube-api-access-cp7pm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-wbn2l\" (UID: \"b5f1c834-97e4-4e1c-9d03-f10adaa9c8b9\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-wbn2l" Jul 29 18:45:06.941919 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:45:06.941881 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b5f1c834-97e4-4e1c-9d03-f10adaa9c8b9-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-wbn2l\" (UID: \"b5f1c834-97e4-4e1c-9d03-f10adaa9c8b9\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-wbn2l" Jul 29 18:45:06.942013 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:45:06.941949 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b5f1c834-97e4-4e1c-9d03-f10adaa9c8b9-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-wbn2l\" (UID: \"b5f1c834-97e4-4e1c-9d03-f10adaa9c8b9\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-wbn2l" Jul 29 18:45:06.942013 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:45:06.941972 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b5f1c834-97e4-4e1c-9d03-f10adaa9c8b9-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-wbn2l\" (UID: \"b5f1c834-97e4-4e1c-9d03-f10adaa9c8b9\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-wbn2l" Jul 29 18:45:06.942013 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:45:06.941994 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b5f1c834-97e4-4e1c-9d03-f10adaa9c8b9-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-wbn2l\" (UID: \"b5f1c834-97e4-4e1c-9d03-f10adaa9c8b9\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-wbn2l" Jul 29 18:45:07.042430 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:45:07.042397 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b5f1c834-97e4-4e1c-9d03-f10adaa9c8b9-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-wbn2l\" (UID: \"b5f1c834-97e4-4e1c-9d03-f10adaa9c8b9\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-wbn2l" Jul 29 18:45:07.042430 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:45:07.042436 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cp7pm\" (UniqueName: \"kubernetes.io/projected/b5f1c834-97e4-4e1c-9d03-f10adaa9c8b9-kube-api-access-cp7pm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-wbn2l\" (UID: \"b5f1c834-97e4-4e1c-9d03-f10adaa9c8b9\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-wbn2l" Jul 29 18:45:07.042659 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:45:07.042468 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b5f1c834-97e4-4e1c-9d03-f10adaa9c8b9-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-wbn2l\" (UID: \"b5f1c834-97e4-4e1c-9d03-f10adaa9c8b9\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-wbn2l" Jul 29 18:45:07.042659 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:45:07.042513 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b5f1c834-97e4-4e1c-9d03-f10adaa9c8b9-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-wbn2l\" (UID: \"b5f1c834-97e4-4e1c-9d03-f10adaa9c8b9\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-wbn2l" Jul 29 18:45:07.042659 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:45:07.042527 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b5f1c834-97e4-4e1c-9d03-f10adaa9c8b9-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-wbn2l\" (UID: \"b5f1c834-97e4-4e1c-9d03-f10adaa9c8b9\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-wbn2l" Jul 29 18:45:07.042659 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:45:07.042544 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b5f1c834-97e4-4e1c-9d03-f10adaa9c8b9-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-wbn2l\" (UID: \"b5f1c834-97e4-4e1c-9d03-f10adaa9c8b9\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-wbn2l" Jul 29 18:45:07.042892 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:45:07.042868 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b5f1c834-97e4-4e1c-9d03-f10adaa9c8b9-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-wbn2l\" (UID: \"b5f1c834-97e4-4e1c-9d03-f10adaa9c8b9\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-wbn2l" Jul 29 18:45:07.042944 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:45:07.042898 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b5f1c834-97e4-4e1c-9d03-f10adaa9c8b9-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-wbn2l\" (UID: \"b5f1c834-97e4-4e1c-9d03-f10adaa9c8b9\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-wbn2l" Jul 29 18:45:07.042944 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:45:07.042918 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b5f1c834-97e4-4e1c-9d03-f10adaa9c8b9-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-wbn2l\" (UID: \"b5f1c834-97e4-4e1c-9d03-f10adaa9c8b9\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-wbn2l" Jul 29 18:45:07.044764 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:45:07.044741 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b5f1c834-97e4-4e1c-9d03-f10adaa9c8b9-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-wbn2l\" (UID: \"b5f1c834-97e4-4e1c-9d03-f10adaa9c8b9\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-wbn2l" Jul 29 18:45:07.045019 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:45:07.045001 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b5f1c834-97e4-4e1c-9d03-f10adaa9c8b9-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-wbn2l\" (UID: \"b5f1c834-97e4-4e1c-9d03-f10adaa9c8b9\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-wbn2l" Jul 29 18:45:07.051243 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:45:07.051225 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cp7pm\" (UniqueName: \"kubernetes.io/projected/b5f1c834-97e4-4e1c-9d03-f10adaa9c8b9-kube-api-access-cp7pm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-wbn2l\" (UID: \"b5f1c834-97e4-4e1c-9d03-f10adaa9c8b9\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-wbn2l" Jul 29 18:45:07.186790 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:45:07.186703 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-wbn2l" Jul 29 18:45:07.319059 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:45:07.318985 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-wbn2l"] Jul 29 18:45:07.321333 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:45:07.321308 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5f1c834_97e4_4e1c_9d03_f10adaa9c8b9.slice/crio-b7779047b4aa4a175672790415c2afa43a512a647e6d4cbee387ea4d6ec6fda9 WatchSource:0}: Error finding container b7779047b4aa4a175672790415c2afa43a512a647e6d4cbee387ea4d6ec6fda9: Status 404 returned error can't find the container with id b7779047b4aa4a175672790415c2afa43a512a647e6d4cbee387ea4d6ec6fda9 Jul 29 18:45:07.527497 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:45:07.527467 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29755845-gfxfs" Jul 29 18:45:07.527691 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:45:07.527466 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29755845-gfxfs" event={"ID":"bd13589c-fe74-4332-8fdd-dec85585f01c","Type":"ContainerDied","Data":"f4f9c857a4084f628b03a874bc9943cc364f14ba3f546156a938b11f9799ebbc"} Jul 29 18:45:07.527691 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:45:07.527589 2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f4f9c857a4084f628b03a874bc9943cc364f14ba3f546156a938b11f9799ebbc" Jul 29 18:45:07.528629 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:45:07.528610 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-wbn2l" event={"ID":"b5f1c834-97e4-4e1c-9d03-f10adaa9c8b9","Type":"ContainerStarted","Data":"b7779047b4aa4a175672790415c2afa43a512a647e6d4cbee387ea4d6ec6fda9"} Jul 29 18:45:08.536748 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:45:08.536712 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-wbn2l" event={"ID":"b5f1c834-97e4-4e1c-9d03-f10adaa9c8b9","Type":"ContainerStarted","Data":"36ae64a48650a947658e9e433840b4097874809fff76daca4a4cc5f4ff016619"} Jul 29 18:45:08.556032 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:45:08.555984 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-wbn2l" podStartSLOduration=2.074454943 podStartE2EDuration="2.555968129s" totalImagesPullingTime="481.513186ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 18:45:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 18:45:08.553470263 +0000 UTC m=+643.779139134" watchObservedRunningTime="2026-07-29 18:45:08.555968129 +0000 UTC m=+643.781637000" Jul 29 18:45:16.002829 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:45:16.002798 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:45:17.187661 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:45:17.187618 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-wbn2l" Jul 29 18:45:17.188149 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:45:17.187700 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-wbn2l" Jul 29 18:45:17.200633 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:45:17.200607 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-wbn2l" Jul 29 18:45:17.593568 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:45:17.593542 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-wbn2l" Jul 29 18:45:35.282047 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:45:35.282018 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-rbqwc"] Jul 29 18:45:35.288628 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:45:35.288610 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-rbqwc" Jul 29 18:45:35.292086 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:45:35.292057 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Jul 29 18:45:35.294532 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:45:35.294503 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-rbqwc"] Jul 29 18:45:35.311120 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:45:35.311094 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:45:35.388784 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:45:35.388753 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc3ba607-5106-4d50-9f19-2d558de8404b-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-rbqwc\" (UID: \"dc3ba607-5106-4d50-9f19-2d558de8404b\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-rbqwc" Jul 29 18:45:35.388913 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:45:35.388788 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc3ba607-5106-4d50-9f19-2d558de8404b-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-rbqwc\" (UID: \"dc3ba607-5106-4d50-9f19-2d558de8404b\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-rbqwc" Jul 29 18:45:35.388913 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:45:35.388877 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qfmf\" (UniqueName: \"kubernetes.io/projected/dc3ba607-5106-4d50-9f19-2d558de8404b-kube-api-access-5qfmf\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-rbqwc\" (UID: \"dc3ba607-5106-4d50-9f19-2d558de8404b\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-rbqwc" Jul 29 18:45:35.388913 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:45:35.388910 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc3ba607-5106-4d50-9f19-2d558de8404b-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-rbqwc\" (UID: \"dc3ba607-5106-4d50-9f19-2d558de8404b\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-rbqwc" Jul 29 18:45:35.389077 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:45:35.388937 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc3ba607-5106-4d50-9f19-2d558de8404b-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-rbqwc\" (UID: \"dc3ba607-5106-4d50-9f19-2d558de8404b\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-rbqwc" Jul 29 18:45:35.389077 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:45:35.388972 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dc3ba607-5106-4d50-9f19-2d558de8404b-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-rbqwc\" (UID: \"dc3ba607-5106-4d50-9f19-2d558de8404b\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-rbqwc" Jul 29 18:45:35.489488 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:45:35.489459 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dc3ba607-5106-4d50-9f19-2d558de8404b-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-rbqwc\" (UID: \"dc3ba607-5106-4d50-9f19-2d558de8404b\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-rbqwc" Jul 29 18:45:35.489626 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:45:35.489510 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc3ba607-5106-4d50-9f19-2d558de8404b-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-rbqwc\" (UID: \"dc3ba607-5106-4d50-9f19-2d558de8404b\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-rbqwc" Jul 29 18:45:35.489669 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:45:35.489622 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc3ba607-5106-4d50-9f19-2d558de8404b-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-rbqwc\" (UID: \"dc3ba607-5106-4d50-9f19-2d558de8404b\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-rbqwc" Jul 29 18:45:35.489724 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:45:35.489709 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qfmf\" (UniqueName: \"kubernetes.io/projected/dc3ba607-5106-4d50-9f19-2d558de8404b-kube-api-access-5qfmf\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-rbqwc\" (UID: \"dc3ba607-5106-4d50-9f19-2d558de8404b\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-rbqwc" Jul 29 18:45:35.489796 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:45:35.489744 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc3ba607-5106-4d50-9f19-2d558de8404b-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-rbqwc\" (UID: \"dc3ba607-5106-4d50-9f19-2d558de8404b\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-rbqwc" Jul 29 18:45:35.489796 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:45:35.489780 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc3ba607-5106-4d50-9f19-2d558de8404b-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-rbqwc\" (UID: \"dc3ba607-5106-4d50-9f19-2d558de8404b\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-rbqwc" Jul 29 18:45:35.489905 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:45:35.489878 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dc3ba607-5106-4d50-9f19-2d558de8404b-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-rbqwc\" (UID: \"dc3ba607-5106-4d50-9f19-2d558de8404b\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-rbqwc" Jul 29 18:45:35.490063 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:45:35.490038 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc3ba607-5106-4d50-9f19-2d558de8404b-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-rbqwc\" (UID: \"dc3ba607-5106-4d50-9f19-2d558de8404b\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-rbqwc" Jul 29 18:45:35.490134 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:45:35.490072 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc3ba607-5106-4d50-9f19-2d558de8404b-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-rbqwc\" (UID: \"dc3ba607-5106-4d50-9f19-2d558de8404b\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-rbqwc" Jul 29 18:45:35.491757 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:45:35.491736 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc3ba607-5106-4d50-9f19-2d558de8404b-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-rbqwc\" (UID: \"dc3ba607-5106-4d50-9f19-2d558de8404b\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-rbqwc" Jul 29 18:45:35.492143 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:45:35.492122 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc3ba607-5106-4d50-9f19-2d558de8404b-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-rbqwc\" (UID: \"dc3ba607-5106-4d50-9f19-2d558de8404b\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-rbqwc" Jul 29 18:45:35.497521 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:45:35.497500 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qfmf\" (UniqueName: \"kubernetes.io/projected/dc3ba607-5106-4d50-9f19-2d558de8404b-kube-api-access-5qfmf\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-rbqwc\" (UID: \"dc3ba607-5106-4d50-9f19-2d558de8404b\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-rbqwc" Jul 29 18:45:35.599941 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:45:35.599863 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-rbqwc" Jul 29 18:45:35.730124 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:45:35.730098 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-rbqwc"] Jul 29 18:45:35.731931 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:45:35.731906 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc3ba607_5106_4d50_9f19_2d558de8404b.slice/crio-ab26c1fb36ad450ca6210d9362282710ae52dcfbdea43bf324d28f030f17f6ea WatchSource:0}: Error finding container ab26c1fb36ad450ca6210d9362282710ae52dcfbdea43bf324d28f030f17f6ea: Status 404 returned error can't find the container with id ab26c1fb36ad450ca6210d9362282710ae52dcfbdea43bf324d28f030f17f6ea Jul 29 18:45:36.687807 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:45:36.687719 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-rbqwc" event={"ID":"dc3ba607-5106-4d50-9f19-2d558de8404b","Type":"ContainerStarted","Data":"e3900d0e83390ab2df0edb7c67f458b203a18e3352eb5e377eec182f7507cc0d"} Jul 29 18:45:36.687807 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:45:36.687770 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-rbqwc" event={"ID":"dc3ba607-5106-4d50-9f19-2d558de8404b","Type":"ContainerStarted","Data":"ab26c1fb36ad450ca6210d9362282710ae52dcfbdea43bf324d28f030f17f6ea"} Jul 29 18:45:36.705604 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:45:36.705551 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-rbqwc" podStartSLOduration=1.127437604 podStartE2EDuration="1.705535273s" totalImagesPullingTime="578.097669ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 18:45:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 18:45:36.70398144 +0000 UTC m=+671.929650313" watchObservedRunningTime="2026-07-29 18:45:36.705535273 +0000 UTC m=+671.931204177" Jul 29 18:45:39.261395 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:45:39.261341 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:45:42.272028 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:45:42.271998 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-bbkz4"] Jul 29 18:45:42.275738 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:45:42.275720 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bbkz4" Jul 29 18:45:42.279429 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:45:42.279413 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Jul 29 18:45:42.289803 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:45:42.289783 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-bbkz4"] Jul 29 18:45:42.300978 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:45:42.300955 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:45:42.351697 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:45:42.351667 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bd51f594-611f-44e7-a8e9-6e02aed86875-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-bbkz4\" (UID: \"bd51f594-611f-44e7-a8e9-6e02aed86875\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bbkz4" Jul 29 18:45:42.351810 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:45:42.351723 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bd51f594-611f-44e7-a8e9-6e02aed86875-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-bbkz4\" (UID: \"bd51f594-611f-44e7-a8e9-6e02aed86875\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bbkz4" Jul 29 18:45:42.351810 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:45:42.351776 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bd51f594-611f-44e7-a8e9-6e02aed86875-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-bbkz4\" (UID: \"bd51f594-611f-44e7-a8e9-6e02aed86875\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bbkz4" Jul 29 18:45:42.351920 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:45:42.351844 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68d55\" (UniqueName: \"kubernetes.io/projected/bd51f594-611f-44e7-a8e9-6e02aed86875-kube-api-access-68d55\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-bbkz4\" (UID: \"bd51f594-611f-44e7-a8e9-6e02aed86875\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bbkz4" Jul 29 18:45:42.351920 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:45:42.351870 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bd51f594-611f-44e7-a8e9-6e02aed86875-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-bbkz4\" (UID: \"bd51f594-611f-44e7-a8e9-6e02aed86875\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bbkz4" Jul 29 18:45:42.352015 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:45:42.351922 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bd51f594-611f-44e7-a8e9-6e02aed86875-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-bbkz4\" (UID: \"bd51f594-611f-44e7-a8e9-6e02aed86875\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bbkz4" Jul 29 18:45:42.453136 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:45:42.453104 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bd51f594-611f-44e7-a8e9-6e02aed86875-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-bbkz4\" (UID: \"bd51f594-611f-44e7-a8e9-6e02aed86875\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bbkz4" Jul 29 18:45:42.453136 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:45:42.453138 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bd51f594-611f-44e7-a8e9-6e02aed86875-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-bbkz4\" (UID: \"bd51f594-611f-44e7-a8e9-6e02aed86875\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bbkz4" Jul 29 18:45:42.453340 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:45:42.453192 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68d55\" (UniqueName: \"kubernetes.io/projected/bd51f594-611f-44e7-a8e9-6e02aed86875-kube-api-access-68d55\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-bbkz4\" (UID: \"bd51f594-611f-44e7-a8e9-6e02aed86875\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bbkz4" Jul 29 18:45:42.453340 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:45:42.453219 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bd51f594-611f-44e7-a8e9-6e02aed86875-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-bbkz4\" (UID: \"bd51f594-611f-44e7-a8e9-6e02aed86875\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bbkz4" Jul 29 18:45:42.453340 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:45:42.453257 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bd51f594-611f-44e7-a8e9-6e02aed86875-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-bbkz4\" (UID: \"bd51f594-611f-44e7-a8e9-6e02aed86875\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bbkz4" Jul 29 18:45:42.453554 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:45:42.453371 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bd51f594-611f-44e7-a8e9-6e02aed86875-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-bbkz4\" (UID: \"bd51f594-611f-44e7-a8e9-6e02aed86875\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bbkz4" Jul 29 18:45:42.453611 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:45:42.453574 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bd51f594-611f-44e7-a8e9-6e02aed86875-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-bbkz4\" (UID: \"bd51f594-611f-44e7-a8e9-6e02aed86875\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bbkz4" Jul 29 18:45:42.453676 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:45:42.453606 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bd51f594-611f-44e7-a8e9-6e02aed86875-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-bbkz4\" (UID: \"bd51f594-611f-44e7-a8e9-6e02aed86875\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bbkz4" Jul 29 18:45:42.453780 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:45:42.453760 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bd51f594-611f-44e7-a8e9-6e02aed86875-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-bbkz4\" (UID: \"bd51f594-611f-44e7-a8e9-6e02aed86875\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bbkz4" Jul 29 18:45:42.455783 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:45:42.455754 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bd51f594-611f-44e7-a8e9-6e02aed86875-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-bbkz4\" (UID: \"bd51f594-611f-44e7-a8e9-6e02aed86875\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bbkz4" Jul 29 18:45:42.456000 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:45:42.455984 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bd51f594-611f-44e7-a8e9-6e02aed86875-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-bbkz4\" (UID: \"bd51f594-611f-44e7-a8e9-6e02aed86875\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bbkz4" Jul 29 18:45:42.461092 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:45:42.461076 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-68d55\" (UniqueName: \"kubernetes.io/projected/bd51f594-611f-44e7-a8e9-6e02aed86875-kube-api-access-68d55\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-bbkz4\" (UID: \"bd51f594-611f-44e7-a8e9-6e02aed86875\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bbkz4" Jul 29 18:45:42.584858 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:45:42.584778 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bbkz4" Jul 29 18:45:42.717676 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:45:42.717650 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-bbkz4"] Jul 29 18:45:42.719829 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:45:42.719797 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbd51f594_611f_44e7_a8e9_6e02aed86875.slice/crio-346cf83b76dcc8a417431a07db6037e73e94eb67c8ea6ec3cc17f6eb824afbbd WatchSource:0}: Error finding container 346cf83b76dcc8a417431a07db6037e73e94eb67c8ea6ec3cc17f6eb824afbbd: Status 404 returned error can't find the container with id 346cf83b76dcc8a417431a07db6037e73e94eb67c8ea6ec3cc17f6eb824afbbd Jul 29 18:45:43.727408 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:45:43.727336 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bbkz4" event={"ID":"bd51f594-611f-44e7-a8e9-6e02aed86875","Type":"ContainerStarted","Data":"27ee260b3917a370a5fec3b31fe3aaf910aa375baf3d3edcec34bfeec887f17a"} Jul 29 18:45:43.727408 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:45:43.727415 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bbkz4" event={"ID":"bd51f594-611f-44e7-a8e9-6e02aed86875","Type":"ContainerStarted","Data":"346cf83b76dcc8a417431a07db6037e73e94eb67c8ea6ec3cc17f6eb824afbbd"} Jul 29 18:45:43.744724 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:45:43.744675 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bbkz4" podStartSLOduration=1.4521020789999999 podStartE2EDuration="1.744662163s" totalImagesPullingTime="292.560084ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 18:45:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 18:45:43.743104834 +0000 UTC m=+678.968773731" watchObservedRunningTime="2026-07-29 18:45:43.744662163 +0000 UTC m=+678.970331034" Jul 29 18:45:45.600752 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:45:45.600712 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-rbqwc" Jul 29 18:45:45.601156 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:45:45.600778 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-rbqwc" Jul 29 18:45:45.613900 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:45:45.613875 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-rbqwc" Jul 29 18:45:45.760558 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:45:45.760523 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-rbqwc" Jul 29 18:45:52.585185 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:45:52.585150 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bbkz4" Jul 29 18:45:52.585694 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:45:52.585202 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bbkz4" Jul 29 18:45:52.597863 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:45:52.597839 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bbkz4" Jul 29 18:45:52.790667 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:45:52.790643 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bbkz4" Jul 29 18:46:23.453493 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:46:23.453404 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-79b6457d9c-8ckk2"] Jul 29 18:46:23.457241 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:46:23.457219 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-79b6457d9c-8ckk2" Jul 29 18:46:23.465388 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:46:23.465333 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-79b6457d9c-8ckk2"] Jul 29 18:46:23.511368 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:46:23.511324 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k95gr\" (UniqueName: \"kubernetes.io/projected/58c2b854-b8b9-4217-8fec-a260faf5ab20-kube-api-access-k95gr\") pod \"authorino-79b6457d9c-8ckk2\" (UID: \"58c2b854-b8b9-4217-8fec-a260faf5ab20\") " pod="kuadrant-system/authorino-79b6457d9c-8ckk2" Jul 29 18:46:23.511484 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:46:23.511385 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/58c2b854-b8b9-4217-8fec-a260faf5ab20-tls-cert\") pod \"authorino-79b6457d9c-8ckk2\" (UID: \"58c2b854-b8b9-4217-8fec-a260faf5ab20\") " pod="kuadrant-system/authorino-79b6457d9c-8ckk2" Jul 29 18:46:23.612927 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:46:23.612898 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k95gr\" (UniqueName: \"kubernetes.io/projected/58c2b854-b8b9-4217-8fec-a260faf5ab20-kube-api-access-k95gr\") pod \"authorino-79b6457d9c-8ckk2\" (UID: \"58c2b854-b8b9-4217-8fec-a260faf5ab20\") " pod="kuadrant-system/authorino-79b6457d9c-8ckk2" Jul 29 18:46:23.613043 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:46:23.612940 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/58c2b854-b8b9-4217-8fec-a260faf5ab20-tls-cert\") pod \"authorino-79b6457d9c-8ckk2\" (UID: \"58c2b854-b8b9-4217-8fec-a260faf5ab20\") " pod="kuadrant-system/authorino-79b6457d9c-8ckk2" Jul 29 18:46:23.615443 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:46:23.615423 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/58c2b854-b8b9-4217-8fec-a260faf5ab20-tls-cert\") pod \"authorino-79b6457d9c-8ckk2\" (UID: \"58c2b854-b8b9-4217-8fec-a260faf5ab20\") " pod="kuadrant-system/authorino-79b6457d9c-8ckk2" Jul 29 18:46:23.620670 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:46:23.620646 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k95gr\" (UniqueName: \"kubernetes.io/projected/58c2b854-b8b9-4217-8fec-a260faf5ab20-kube-api-access-k95gr\") pod \"authorino-79b6457d9c-8ckk2\" (UID: \"58c2b854-b8b9-4217-8fec-a260faf5ab20\") " pod="kuadrant-system/authorino-79b6457d9c-8ckk2" Jul 29 18:46:23.767828 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:46:23.767800 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-79b6457d9c-8ckk2" Jul 29 18:46:23.901645 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:46:23.901607 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-79b6457d9c-8ckk2"] Jul 29 18:46:23.903599 ip-10-0-140-205 kubenswrapper[2560]: W0729 18:46:23.903566 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod58c2b854_b8b9_4217_8fec_a260faf5ab20.slice/crio-90cef20488e08dee81b4be78260c36b439ba524af726b6f226599f3bee43d2ac WatchSource:0}: Error finding container 90cef20488e08dee81b4be78260c36b439ba524af726b6f226599f3bee43d2ac: Status 404 returned error can't find the container with id 90cef20488e08dee81b4be78260c36b439ba524af726b6f226599f3bee43d2ac Jul 29 18:46:23.940952 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:46:23.940917 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-79b6457d9c-8ckk2" event={"ID":"58c2b854-b8b9-4217-8fec-a260faf5ab20","Type":"ContainerStarted","Data":"90cef20488e08dee81b4be78260c36b439ba524af726b6f226599f3bee43d2ac"} Jul 29 18:46:23.966802 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:46:23.966775 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:46:24.059379 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:46:24.059293 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:46:24.950386 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:46:24.950331 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-79b6457d9c-8ckk2" event={"ID":"58c2b854-b8b9-4217-8fec-a260faf5ab20","Type":"ContainerStarted","Data":"9176994cff59fc852ef8bc40fe1621576cc56579b04594f4565349c1286d6b9b"} Jul 29 18:46:24.971865 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:46:24.971809 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-79b6457d9c-8ckk2" podStartSLOduration=1.559542035 podStartE2EDuration="1.971787132s" totalImagesPullingTime="412.245097ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 18:46:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 18:46:24.968498773 +0000 UTC m=+720.194167646" watchObservedRunningTime="2026-07-29 18:46:24.971787132 +0000 UTC m=+720.197456006" Jul 29 18:46:25.004615 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:46:25.004575 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-79599897b8-jj46z"] Jul 29 18:46:25.005312 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:46:25.005050 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-79599897b8-jj46z" podUID="70c8de05-3b10-4ff8-9a02-22b7738470e5" containerName="authorino" containerID="cri-o://4ba12611ab38590646bad0c722b40b902a18904c18a46bd0fbb65c64d3327cd5" gracePeriod=30 Jul 29 18:46:25.273734 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:46:25.273709 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-79599897b8-jj46z" Jul 29 18:46:25.334820 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:46:25.334785 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/70c8de05-3b10-4ff8-9a02-22b7738470e5-tls-cert\") pod \"70c8de05-3b10-4ff8-9a02-22b7738470e5\" (UID: \"70c8de05-3b10-4ff8-9a02-22b7738470e5\") " Jul 29 18:46:25.334991 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:46:25.334845 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:46:25.334991 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:46:25.334882 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m4wwg\" (UniqueName: \"kubernetes.io/projected/70c8de05-3b10-4ff8-9a02-22b7738470e5-kube-api-access-m4wwg\") pod \"70c8de05-3b10-4ff8-9a02-22b7738470e5\" (UID: \"70c8de05-3b10-4ff8-9a02-22b7738470e5\") " Jul 29 18:46:25.337331 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:46:25.337297 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70c8de05-3b10-4ff8-9a02-22b7738470e5-kube-api-access-m4wwg" (OuterVolumeSpecName: "kube-api-access-m4wwg") pod "70c8de05-3b10-4ff8-9a02-22b7738470e5" (UID: "70c8de05-3b10-4ff8-9a02-22b7738470e5"). InnerVolumeSpecName "kube-api-access-m4wwg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 18:46:25.347332 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:46:25.347307 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70c8de05-3b10-4ff8-9a02-22b7738470e5-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "70c8de05-3b10-4ff8-9a02-22b7738470e5" (UID: "70c8de05-3b10-4ff8-9a02-22b7738470e5"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 18:46:25.436384 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:46:25.436336 2560 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/70c8de05-3b10-4ff8-9a02-22b7738470e5-tls-cert\") on node \"ip-10-0-140-205.ec2.internal\" DevicePath \"\"" Jul 29 18:46:25.436384 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:46:25.436372 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m4wwg\" (UniqueName: \"kubernetes.io/projected/70c8de05-3b10-4ff8-9a02-22b7738470e5-kube-api-access-m4wwg\") on node \"ip-10-0-140-205.ec2.internal\" DevicePath \"\"" Jul 29 18:46:25.488428 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:46:25.488396 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:46:25.957773 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:46:25.957684 2560 generic.go:356] "Generic (PLEG): container finished" podID="70c8de05-3b10-4ff8-9a02-22b7738470e5" containerID="4ba12611ab38590646bad0c722b40b902a18904c18a46bd0fbb65c64d3327cd5" exitCode=0 Jul 29 18:46:25.957773 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:46:25.957728 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-79599897b8-jj46z" Jul 29 18:46:25.958264 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:46:25.957774 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-79599897b8-jj46z" event={"ID":"70c8de05-3b10-4ff8-9a02-22b7738470e5","Type":"ContainerDied","Data":"4ba12611ab38590646bad0c722b40b902a18904c18a46bd0fbb65c64d3327cd5"} Jul 29 18:46:25.958264 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:46:25.957817 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-79599897b8-jj46z" event={"ID":"70c8de05-3b10-4ff8-9a02-22b7738470e5","Type":"ContainerDied","Data":"14e2b9e9d9c857d705711daf014d45b2e30c977e678a638f622673fdf635bffe"} Jul 29 18:46:25.958264 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:46:25.957843 2560 scope.go:117] "RemoveContainer" containerID="4ba12611ab38590646bad0c722b40b902a18904c18a46bd0fbb65c64d3327cd5" Jul 29 18:46:25.967905 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:46:25.967887 2560 scope.go:117] "RemoveContainer" containerID="4ba12611ab38590646bad0c722b40b902a18904c18a46bd0fbb65c64d3327cd5" Jul 29 18:46:25.968155 ip-10-0-140-205 kubenswrapper[2560]: E0729 18:46:25.968138 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ba12611ab38590646bad0c722b40b902a18904c18a46bd0fbb65c64d3327cd5\": container with ID starting with 4ba12611ab38590646bad0c722b40b902a18904c18a46bd0fbb65c64d3327cd5 not found: ID does not exist" containerID="4ba12611ab38590646bad0c722b40b902a18904c18a46bd0fbb65c64d3327cd5" Jul 29 18:46:25.968213 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:46:25.968165 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ba12611ab38590646bad0c722b40b902a18904c18a46bd0fbb65c64d3327cd5"} err="failed to get container status \"4ba12611ab38590646bad0c722b40b902a18904c18a46bd0fbb65c64d3327cd5\": rpc error: code = NotFound desc = could not find container \"4ba12611ab38590646bad0c722b40b902a18904c18a46bd0fbb65c64d3327cd5\": container with ID starting with 4ba12611ab38590646bad0c722b40b902a18904c18a46bd0fbb65c64d3327cd5 not found: ID does not exist" Jul 29 18:46:25.975816 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:46:25.975793 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-79599897b8-jj46z"] Jul 29 18:46:25.981629 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:46:25.981607 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-79599897b8-jj46z"] Jul 29 18:46:27.354222 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:46:27.354184 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="70c8de05-3b10-4ff8-9a02-22b7738470e5" path="/var/lib/kubelet/pods/70c8de05-3b10-4ff8-9a02-22b7738470e5/volumes" Jul 29 18:47:33.309229 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:47:33.309201 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:47:35.884815 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:47:35.884780 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:48:06.309274 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:48:06.309184 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:48:32.987714 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:48:32.987678 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:48:34.520257 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:48:34.520226 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:49:08.423954 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:49:08.423915 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:49:08.522892 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:49:08.522859 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:49:08.684484 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:49:08.684376 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:49:14.351928 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:49:14.351893 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:49:14.468967 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:49:14.468936 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:49:14.555983 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:49:14.555938 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:49:14.745698 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:49:14.745673 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:49:20.423114 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:49:20.423084 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:49:20.523174 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:49:20.523142 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:49:25.385454 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:49:25.385336 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-v4n5x_1727fb40-fbed-4984-91d4-a5890d0f4103/ovn-acl-logging/0.log" Jul 29 18:49:25.393105 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:49:25.388274 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-v4n5x_1727fb40-fbed-4984-91d4-a5890d0f4103/ovn-acl-logging/0.log" Jul 29 18:50:13.454426 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:50:13.454399 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:52:00.455374 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:52:00.455330 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:54:25.426233 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:54:25.426137 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-v4n5x_1727fb40-fbed-4984-91d4-a5890d0f4103/ovn-acl-logging/0.log" Jul 29 18:54:25.430785 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:54:25.430760 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-v4n5x_1727fb40-fbed-4984-91d4-a5890d0f4103/ovn-acl-logging/0.log" Jul 29 18:55:58.580780 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:55:58.580741 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:55:59.846532 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:55:59.846499 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:57:41.232644 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:57:41.232607 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:57:50.989525 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:57:50.989485 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:58:26.384813 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:58:26.384779 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:58:26.485269 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:58:26.485242 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:58:27.477560 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:58:27.477526 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:59:02.395453 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:59:02.395422 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 18:59:25.461461 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:59:25.461336 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-v4n5x_1727fb40-fbed-4984-91d4-a5890d0f4103/ovn-acl-logging/0.log" Jul 29 18:59:25.468989 ip-10-0-140-205 kubenswrapper[2560]: I0729 18:59:25.468966 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-v4n5x_1727fb40-fbed-4984-91d4-a5890d0f4103/ovn-acl-logging/0.log" Jul 29 19:01:39.791404 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:01:39.791370 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:01:44.211262 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:01:44.211232 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:01:44.312458 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:01:44.312416 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:02:01.933588 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:02:01.933556 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:02:05.377232 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:02:05.377187 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:02:05.474766 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:02:05.474738 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:02:05.931147 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:02:05.931109 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:02:12.088476 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:02:12.088440 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:02:13.647251 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:02:13.647214 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:02:13.710448 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:02:13.710418 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:02:14.697729 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:02:14.697682 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:02:14.785158 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:02:14.785121 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:02:14.990784 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:02:14.990699 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:02:15.089588 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:02:15.089552 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:02:15.595499 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:02:15.595463 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:02:15.695472 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:02:15.695435 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:02:45.381043 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:02:45.380933 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:02:45.454114 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:02:45.454087 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:02:45.520189 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:02:45.520157 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:02:45.579999 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:02:45.579966 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:02:45.736409 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:02:45.736316 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:02:46.101695 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:02:46.101663 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:02:46.313364 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:02:46.313313 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:02:46.405301 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:02:46.405215 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:02:46.561205 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:02:46.561180 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:02:46.924764 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:02:46.924731 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:02:47.274667 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:02:47.274635 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:02:49.040201 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:02:49.040168 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:02:49.305061 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:02:49.304956 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:02:49.411764 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:02:49.411730 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:02:49.505988 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:02:49.505959 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:02:49.784834 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:02:49.784798 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:02:49.881257 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:02:49.881228 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:03:00.361464 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:03:00.361429 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:03:20.485166 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:03:20.485135 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:03:20.550611 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:03:20.550574 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:03:20.649803 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:03:20.649773 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:03:20.707138 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:03:20.707108 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:03:20.808614 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:03:20.808585 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:03:21.065271 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:03:21.065192 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:03:21.317402 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:03:21.317300 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:03:21.669054 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:03:21.668970 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:03:21.784586 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:03:21.784562 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:03:24.102136 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:03:24.102101 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:03:24.319796 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:03:24.319765 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:03:24.449321 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:03:24.449237 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:03:24.890028 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:03:24.889992 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:03:24.994419 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:03:24.994390 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:03:54.946302 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:03:54.946265 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:03:55.024191 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:03:55.024156 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:03:55.084638 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:03:55.084606 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:03:55.186777 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:03:55.186738 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:03:56.078243 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:03:56.078207 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:03:56.443918 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:03:56.443832 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:03:56.674201 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:03:56.674163 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:03:56.918407 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:03:56.918370 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:03:57.251820 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:03:57.251787 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:03:58.127501 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:03:58.127473 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:03:59.280827 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:03:59.280787 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:03:59.380626 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:03:59.380596 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:03:59.466571 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:03:59.466534 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:03:59.574909 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:03:59.574814 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:04:00.105887 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:04:00.105822 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:04:00.196832 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:04:00.196806 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:04:25.504711 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:04:25.504684 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-v4n5x_1727fb40-fbed-4984-91d4-a5890d0f4103/ovn-acl-logging/0.log" Jul 29 19:04:25.511902 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:04:25.511881 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-v4n5x_1727fb40-fbed-4984-91d4-a5890d0f4103/ovn-acl-logging/0.log" Jul 29 19:04:30.001745 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:04:30.001710 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:04:30.104250 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:04:30.104220 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:04:30.196836 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:04:30.196797 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:04:30.264408 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:04:30.264294 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:04:31.396449 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:04:31.396411 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:04:31.525064 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:04:31.525029 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:04:32.424304 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:04:32.424274 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:04:32.930821 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:04:32.930789 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:04:39.137636 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:04:39.137592 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:04:39.244554 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:04:39.244507 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:04:39.392013 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:04:39.391918 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:04:39.492770 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:04:39.492736 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:04:39.929882 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:04:39.929846 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:05:06.010553 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:05:06.010521 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:05:12.826588 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:05:12.826550 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:05:12.936873 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:05:12.936844 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:05:13.027789 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:05:13.027755 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:05:29.017325 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:05:29.017287 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:05:34.978108 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:05:34.978078 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:05:35.081021 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:05:35.080976 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:05:35.178450 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:05:35.178410 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:05:45.782756 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:05:45.782667 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:05:46.679072 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:05:46.679036 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:05:52.864337 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:05:52.864295 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:05:52.971667 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:05:52.971623 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:05:53.063091 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:05:53.063054 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:05:53.163397 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:05:53.163286 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:05:58.398834 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:05:58.398805 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:05:58.499365 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:05:58.499305 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:05:59.127694 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:05:59.127659 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:06:00.427540 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:06:00.427513 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:06:06.441951 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:06:06.441923 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:06:06.552255 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:06:06.552200 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:06:07.159009 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:06:07.158961 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:06:14.096871 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:06:14.096836 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:06:14.200694 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:06:14.200642 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:06:16.984176 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:06:16.984136 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:06:17.089703 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:06:17.089658 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:06:17.169290 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:06:17.169256 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:06:17.299855 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:06:17.299811 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:06:17.846828 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:06:17.846785 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:06:17.946796 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:06:17.946764 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:06:42.101705 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:06:42.101640 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:06:42.780367 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:06:42.780305 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:06:45.407514 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:06:45.407481 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:06:48.991624 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:06:48.991577 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:06:49.080877 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:06:49.080839 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:06:59.896009 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:06:59.895966 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:06:59.995839 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:06:59.995807 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:07:00.060942 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:07:00.060915 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:07:00.119299 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:07:00.119269 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:07:01.155216 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:07:01.155175 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:07:02.768663 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:07:02.768631 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:07:06.604571 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:07:06.604540 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:07:06.706154 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:07:06.706118 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:07:06.900971 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:07:06.900884 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:07:07.124076 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:07:07.124048 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:07:07.254782 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:07:07.254744 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:07:07.353640 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:07:07.353603 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:07:07.488693 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:07:07.488658 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:07:13.486151 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:07:13.486113 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:07:13.573843 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:07:13.573804 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:07:13.666974 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:07:13.666925 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:07:13.780042 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:07:13.780001 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:07:13.923579 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:07:13.923547 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:07:19.629166 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:07:19.629081 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:07:19.728811 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:07:19.728782 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:07:26.901910 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:07:26.901870 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:07:27.939786 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:07:27.939752 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:07:28.040278 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:07:28.040240 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:07:29.957213 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:07:29.957183 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:07:30.055752 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:07:30.055701 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:07:30.202468 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:07:30.202434 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:07:30.294363 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:07:30.294311 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:07:30.508459 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:07:30.508423 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:07:30.742055 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:07:30.741947 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:07:30.851630 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:07:30.851587 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:08:00.156251 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:08:00.156219 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:08:07.222583 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:08:07.222546 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:08:45.132711 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:08:45.132676 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:08:45.210509 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:08:45.210425 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:08:45.305207 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:08:45.305178 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:08:45.369754 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:08:45.369711 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:08:48.755594 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:08:48.755553 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:08:52.453211 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:08:52.453166 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:08:52.525523 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:08:52.525488 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:08:53.777906 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:08:53.777872 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:08:56.980552 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:08:56.980517 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:09:25.542487 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:09:25.542381 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-v4n5x_1727fb40-fbed-4984-91d4-a5890d0f4103/ovn-acl-logging/0.log" Jul 29 19:09:25.553237 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:09:25.553216 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-v4n5x_1727fb40-fbed-4984-91d4-a5890d0f4103/ovn-acl-logging/0.log" Jul 29 19:09:30.565672 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:09:30.565624 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:09:30.754124 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:09:30.754089 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:09:36.523655 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:09:36.523618 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:09:36.601595 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:09:36.601565 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:09:36.833786 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:09:36.833685 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:09:37.105475 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:09:37.105392 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:09:38.249108 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:09:38.249067 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:09:38.365500 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:09:38.365445 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:09:38.469688 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:09:38.469644 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:09:39.314959 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:09:39.314929 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:09:39.413888 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:09:39.413826 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:09:39.546864 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:09:39.546832 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:09:41.019964 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:09:41.019922 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:09:46.670140 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:09:46.670106 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:09:46.770123 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:09:46.770089 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:09:47.342861 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:09:47.342818 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:09:54.443254 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:09:54.443171 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:09:54.541506 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:09:54.541426 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:10:04.831521 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:10:04.831490 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:10:05.200600 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:10:05.200509 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:10:26.286935 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:10:26.286891 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:10:32.847828 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:10:32.847792 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:10:32.927094 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:10:32.927064 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:10:33.208981 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:10:33.208894 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:10:34.465382 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:10:34.465300 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:10:34.528545 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:10:34.528504 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:10:36.885329 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:10:36.885283 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:10:36.992645 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:10:36.992614 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:10:37.120052 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:10:37.120019 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:10:37.227082 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:10:37.226999 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:10:37.327080 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:10:37.327052 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:10:37.850369 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:10:37.850313 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:10:37.960011 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:10:37.959968 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:11:01.955901 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:11:01.955866 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:11:02.324162 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:11:02.324127 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:11:02.748199 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:11:02.748163 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:11:03.001751 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:11:03.001656 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:11:04.361209 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:11:04.361178 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:11:06.019601 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:11:06.019565 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:11:06.121610 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:11:06.121576 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:11:06.221314 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:11:06.221280 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:11:06.538283 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:11:06.538252 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:11:06.753900 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:11:06.753867 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:11:06.851714 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:11:06.851625 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:11:11.763551 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:11:11.763518 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:11:11.831848 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:11:11.831820 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:11:31.117001 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:11:31.116968 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:11:37.721755 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:11:37.721718 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:12:04.792152 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:12:04.792064 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-5b4a8e/test-model-5b4a8e-kserve-7b5cb5f6cf-c7jd5"] Jul 29 19:12:04.792698 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:12:04.792670 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="70c8de05-3b10-4ff8-9a02-22b7738470e5" containerName="authorino" Jul 29 19:12:04.792698 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:12:04.792689 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="70c8de05-3b10-4ff8-9a02-22b7738470e5" containerName="authorino" Jul 29 19:12:04.792855 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:12:04.792780 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="70c8de05-3b10-4ff8-9a02-22b7738470e5" containerName="authorino" Jul 29 19:12:04.795650 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:12:04.795629 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-5b4a8e/test-model-5b4a8e-kserve-7b5cb5f6cf-c7jd5" Jul 29 19:12:04.799139 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:12:04.799115 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-5b4a8e\"/\"test-model-5b4a8e-kserve-self-signed-certs\"" Jul 29 19:12:04.799260 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:12:04.799115 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-5b4a8e\"/\"openshift-service-ca.crt\"" Jul 29 19:12:04.799260 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:12:04.799117 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-5b4a8e\"/\"default-dockercfg-p8ksp\"" Jul 29 19:12:04.799406 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:12:04.799129 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-5b4a8e\"/\"kube-root-ca.crt\"" Jul 29 19:12:04.804681 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:12:04.804614 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-5b4a8e/test-model-5b4a8e-kserve-7b5cb5f6cf-c7jd5"] Jul 29 19:12:04.821554 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:12:04.821530 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:12:04.879721 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:12:04.879697 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4479ff91-eaa7-4329-9e86-6ecacc061f59-tls-certs\") pod \"test-model-5b4a8e-kserve-7b5cb5f6cf-c7jd5\" (UID: \"4479ff91-eaa7-4329-9e86-6ecacc061f59\") " pod="ai-tenant-e2e-inf-b-5b4a8e/test-model-5b4a8e-kserve-7b5cb5f6cf-c7jd5" Jul 29 19:12:04.879814 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:12:04.879736 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4479ff91-eaa7-4329-9e86-6ecacc061f59-home\") pod \"test-model-5b4a8e-kserve-7b5cb5f6cf-c7jd5\" (UID: \"4479ff91-eaa7-4329-9e86-6ecacc061f59\") " pod="ai-tenant-e2e-inf-b-5b4a8e/test-model-5b4a8e-kserve-7b5cb5f6cf-c7jd5" Jul 29 19:12:04.879814 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:12:04.879753 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d67h2\" (UniqueName: \"kubernetes.io/projected/4479ff91-eaa7-4329-9e86-6ecacc061f59-kube-api-access-d67h2\") pod \"test-model-5b4a8e-kserve-7b5cb5f6cf-c7jd5\" (UID: \"4479ff91-eaa7-4329-9e86-6ecacc061f59\") " pod="ai-tenant-e2e-inf-b-5b4a8e/test-model-5b4a8e-kserve-7b5cb5f6cf-c7jd5" Jul 29 19:12:04.879897 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:12:04.879840 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4479ff91-eaa7-4329-9e86-6ecacc061f59-dshm\") pod \"test-model-5b4a8e-kserve-7b5cb5f6cf-c7jd5\" (UID: \"4479ff91-eaa7-4329-9e86-6ecacc061f59\") " pod="ai-tenant-e2e-inf-b-5b4a8e/test-model-5b4a8e-kserve-7b5cb5f6cf-c7jd5" Jul 29 19:12:04.879897 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:12:04.879882 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4479ff91-eaa7-4329-9e86-6ecacc061f59-model-cache\") pod \"test-model-5b4a8e-kserve-7b5cb5f6cf-c7jd5\" (UID: \"4479ff91-eaa7-4329-9e86-6ecacc061f59\") " pod="ai-tenant-e2e-inf-b-5b4a8e/test-model-5b4a8e-kserve-7b5cb5f6cf-c7jd5" Jul 29 19:12:04.879970 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:12:04.879926 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4479ff91-eaa7-4329-9e86-6ecacc061f59-tmp-dir\") pod \"test-model-5b4a8e-kserve-7b5cb5f6cf-c7jd5\" (UID: \"4479ff91-eaa7-4329-9e86-6ecacc061f59\") " pod="ai-tenant-e2e-inf-b-5b4a8e/test-model-5b4a8e-kserve-7b5cb5f6cf-c7jd5" Jul 29 19:12:04.981149 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:12:04.981126 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4479ff91-eaa7-4329-9e86-6ecacc061f59-home\") pod \"test-model-5b4a8e-kserve-7b5cb5f6cf-c7jd5\" (UID: \"4479ff91-eaa7-4329-9e86-6ecacc061f59\") " pod="ai-tenant-e2e-inf-b-5b4a8e/test-model-5b4a8e-kserve-7b5cb5f6cf-c7jd5" Jul 29 19:12:04.981269 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:12:04.981159 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d67h2\" (UniqueName: \"kubernetes.io/projected/4479ff91-eaa7-4329-9e86-6ecacc061f59-kube-api-access-d67h2\") pod \"test-model-5b4a8e-kserve-7b5cb5f6cf-c7jd5\" (UID: \"4479ff91-eaa7-4329-9e86-6ecacc061f59\") " pod="ai-tenant-e2e-inf-b-5b4a8e/test-model-5b4a8e-kserve-7b5cb5f6cf-c7jd5" Jul 29 19:12:04.981269 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:12:04.981205 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4479ff91-eaa7-4329-9e86-6ecacc061f59-dshm\") pod \"test-model-5b4a8e-kserve-7b5cb5f6cf-c7jd5\" (UID: \"4479ff91-eaa7-4329-9e86-6ecacc061f59\") " pod="ai-tenant-e2e-inf-b-5b4a8e/test-model-5b4a8e-kserve-7b5cb5f6cf-c7jd5" Jul 29 19:12:04.981269 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:12:04.981248 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4479ff91-eaa7-4329-9e86-6ecacc061f59-model-cache\") pod \"test-model-5b4a8e-kserve-7b5cb5f6cf-c7jd5\" (UID: \"4479ff91-eaa7-4329-9e86-6ecacc061f59\") " pod="ai-tenant-e2e-inf-b-5b4a8e/test-model-5b4a8e-kserve-7b5cb5f6cf-c7jd5" Jul 29 19:12:04.981453 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:12:04.981302 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4479ff91-eaa7-4329-9e86-6ecacc061f59-tmp-dir\") pod \"test-model-5b4a8e-kserve-7b5cb5f6cf-c7jd5\" (UID: \"4479ff91-eaa7-4329-9e86-6ecacc061f59\") " pod="ai-tenant-e2e-inf-b-5b4a8e/test-model-5b4a8e-kserve-7b5cb5f6cf-c7jd5" Jul 29 19:12:04.981504 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:12:04.981461 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4479ff91-eaa7-4329-9e86-6ecacc061f59-tls-certs\") pod \"test-model-5b4a8e-kserve-7b5cb5f6cf-c7jd5\" (UID: \"4479ff91-eaa7-4329-9e86-6ecacc061f59\") " pod="ai-tenant-e2e-inf-b-5b4a8e/test-model-5b4a8e-kserve-7b5cb5f6cf-c7jd5" Jul 29 19:12:04.981559 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:12:04.981532 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4479ff91-eaa7-4329-9e86-6ecacc061f59-home\") pod \"test-model-5b4a8e-kserve-7b5cb5f6cf-c7jd5\" (UID: \"4479ff91-eaa7-4329-9e86-6ecacc061f59\") " pod="ai-tenant-e2e-inf-b-5b4a8e/test-model-5b4a8e-kserve-7b5cb5f6cf-c7jd5" Jul 29 19:12:04.981647 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:12:04.981629 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4479ff91-eaa7-4329-9e86-6ecacc061f59-model-cache\") pod \"test-model-5b4a8e-kserve-7b5cb5f6cf-c7jd5\" (UID: \"4479ff91-eaa7-4329-9e86-6ecacc061f59\") " pod="ai-tenant-e2e-inf-b-5b4a8e/test-model-5b4a8e-kserve-7b5cb5f6cf-c7jd5" Jul 29 19:12:04.981730 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:12:04.981709 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4479ff91-eaa7-4329-9e86-6ecacc061f59-tmp-dir\") pod \"test-model-5b4a8e-kserve-7b5cb5f6cf-c7jd5\" (UID: \"4479ff91-eaa7-4329-9e86-6ecacc061f59\") " pod="ai-tenant-e2e-inf-b-5b4a8e/test-model-5b4a8e-kserve-7b5cb5f6cf-c7jd5" Jul 29 19:12:04.983556 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:12:04.983529 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4479ff91-eaa7-4329-9e86-6ecacc061f59-dshm\") pod \"test-model-5b4a8e-kserve-7b5cb5f6cf-c7jd5\" (UID: \"4479ff91-eaa7-4329-9e86-6ecacc061f59\") " pod="ai-tenant-e2e-inf-b-5b4a8e/test-model-5b4a8e-kserve-7b5cb5f6cf-c7jd5" Jul 29 19:12:04.983848 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:12:04.983832 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4479ff91-eaa7-4329-9e86-6ecacc061f59-tls-certs\") pod \"test-model-5b4a8e-kserve-7b5cb5f6cf-c7jd5\" (UID: \"4479ff91-eaa7-4329-9e86-6ecacc061f59\") " pod="ai-tenant-e2e-inf-b-5b4a8e/test-model-5b4a8e-kserve-7b5cb5f6cf-c7jd5" Jul 29 19:12:04.990899 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:12:04.990882 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d67h2\" (UniqueName: \"kubernetes.io/projected/4479ff91-eaa7-4329-9e86-6ecacc061f59-kube-api-access-d67h2\") pod \"test-model-5b4a8e-kserve-7b5cb5f6cf-c7jd5\" (UID: \"4479ff91-eaa7-4329-9e86-6ecacc061f59\") " pod="ai-tenant-e2e-inf-b-5b4a8e/test-model-5b4a8e-kserve-7b5cb5f6cf-c7jd5" Jul 29 19:12:05.105787 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:12:05.105716 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-5b4a8e/test-model-5b4a8e-kserve-7b5cb5f6cf-c7jd5" Jul 29 19:12:05.245316 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:12:05.243677 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-5b4a8e/test-model-5b4a8e-kserve-7b5cb5f6cf-c7jd5"] Jul 29 19:12:05.247380 ip-10-0-140-205 kubenswrapper[2560]: W0729 19:12:05.247336 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4479ff91_eaa7_4329_9e86_6ecacc061f59.slice/crio-bb0ddd72a08c38f1a40f7b49713409a836fdcc988c540902927ed69a0729b019 WatchSource:0}: Error finding container bb0ddd72a08c38f1a40f7b49713409a836fdcc988c540902927ed69a0729b019: Status 404 returned error can't find the container with id bb0ddd72a08c38f1a40f7b49713409a836fdcc988c540902927ed69a0729b019 Jul 29 19:12:06.245170 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:12:06.245138 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-5b4a8e/test-model-5b4a8e-kserve-7b5cb5f6cf-c7jd5" event={"ID":"4479ff91-eaa7-4329-9e86-6ecacc061f59","Type":"ContainerStarted","Data":"96836be905ddd07225c91304f0dc7ae05341f6495ad9b9badc282374e94e02cf"} Jul 29 19:12:06.245553 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:12:06.245181 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-5b4a8e/test-model-5b4a8e-kserve-7b5cb5f6cf-c7jd5" event={"ID":"4479ff91-eaa7-4329-9e86-6ecacc061f59","Type":"ContainerStarted","Data":"bb0ddd72a08c38f1a40f7b49713409a836fdcc988c540902927ed69a0729b019"} Jul 29 19:12:06.264200 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:12:06.264154 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-5b4a8e/test-model-5b4a8e-kserve-7b5cb5f6cf-c7jd5" podStartSLOduration=2.264141012 podStartE2EDuration="2.264141012s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:12:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:12:06.261607475 +0000 UTC m=+2261.487276346" watchObservedRunningTime="2026-07-29 19:12:06.264141012 +0000 UTC m=+2261.489809903" Jul 29 19:12:14.523109 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:12:14.523071 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:12:15.106327 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:12:15.106294 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-5b4a8e/test-model-5b4a8e-kserve-7b5cb5f6cf-c7jd5" Jul 29 19:12:15.106527 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:12:15.106383 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-5b4a8e/test-model-5b4a8e-kserve-7b5cb5f6cf-c7jd5" Jul 29 19:12:15.118639 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:12:15.118616 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-5b4a8e/test-model-5b4a8e-kserve-7b5cb5f6cf-c7jd5" Jul 29 19:12:15.310759 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:12:15.310724 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-5b4a8e/test-model-5b4a8e-kserve-7b5cb5f6cf-c7jd5" Jul 29 19:12:46.186116 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:12:46.186077 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:12:47.111064 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:12:47.111034 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-5b4a8e/test-model-5b4a8e-kserve-7b5cb5f6cf-c7jd5"] Jul 29 19:12:47.137301 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:12:47.137269 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:12:47.260373 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:12:47.260320 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e3eaaf-jgf5c"] Jul 29 19:12:47.263847 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:12:47.263831 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e3eaaf-jgf5c" Jul 29 19:12:47.266314 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:12:47.266289 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vgb9z\"" Jul 29 19:12:47.271837 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:12:47.271814 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e3eaaf-jgf5c"] Jul 29 19:12:47.355925 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:12:47.355886 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3093709-1f70-42c1-8a2d-e82ac699b83d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-e3eaaf-jgf5c\" (UID: \"d3093709-1f70-42c1-8a2d-e82ac699b83d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e3eaaf-jgf5c" Jul 29 19:12:47.357565 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:12:47.357537 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:12:47.456953 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:12:47.456887 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3093709-1f70-42c1-8a2d-e82ac699b83d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-e3eaaf-jgf5c\" (UID: \"d3093709-1f70-42c1-8a2d-e82ac699b83d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e3eaaf-jgf5c" Jul 29 19:12:47.457527 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:12:47.457510 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3093709-1f70-42c1-8a2d-e82ac699b83d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-e3eaaf-jgf5c\" (UID: \"d3093709-1f70-42c1-8a2d-e82ac699b83d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e3eaaf-jgf5c" Jul 29 19:12:47.575769 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:12:47.575745 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e3eaaf-jgf5c" Jul 29 19:12:47.922199 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:12:47.922139 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e3eaaf-jgf5c"] Jul 29 19:12:47.924616 ip-10-0-140-205 kubenswrapper[2560]: W0729 19:12:47.924583 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3093709_1f70_42c1_8a2d_e82ac699b83d.slice/crio-108a98900637b67bfc62c5cd0183a6f198e34905097e44a19915351948e86df6 WatchSource:0}: Error finding container 108a98900637b67bfc62c5cd0183a6f198e34905097e44a19915351948e86df6: Status 404 returned error can't find the container with id 108a98900637b67bfc62c5cd0183a6f198e34905097e44a19915351948e86df6 Jul 29 19:12:48.482232 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:12:48.482198 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e3eaaf-jgf5c" event={"ID":"d3093709-1f70-42c1-8a2d-e82ac699b83d","Type":"ContainerStarted","Data":"c028eef3bf37d38e1d7c1b82b9a023c360381099d0d2486ac0a1f447b5745d59"} Jul 29 19:12:48.482633 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:12:48.482287 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e3eaaf-jgf5c" event={"ID":"d3093709-1f70-42c1-8a2d-e82ac699b83d","Type":"ContainerStarted","Data":"108a98900637b67bfc62c5cd0183a6f198e34905097e44a19915351948e86df6"} Jul 29 19:12:48.498553 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:12:48.498509 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e3eaaf-jgf5c" podStartSLOduration=1.4984959899999999 podStartE2EDuration="1.49849599s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:12:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:12:48.496340314 +0000 UTC m=+2303.722009187" watchObservedRunningTime="2026-07-29 19:12:48.49849599 +0000 UTC m=+2303.724164888" Jul 29 19:12:49.488911 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:12:49.488869 2560 generic.go:356] "Generic (PLEG): container finished" podID="d3093709-1f70-42c1-8a2d-e82ac699b83d" containerID="c028eef3bf37d38e1d7c1b82b9a023c360381099d0d2486ac0a1f447b5745d59" exitCode=0 Jul 29 19:12:49.489333 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:12:49.488933 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e3eaaf-jgf5c" event={"ID":"d3093709-1f70-42c1-8a2d-e82ac699b83d","Type":"ContainerDied","Data":"c028eef3bf37d38e1d7c1b82b9a023c360381099d0d2486ac0a1f447b5745d59"} Jul 29 19:12:50.626341 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:12:50.626315 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e3eaaf-jgf5c" Jul 29 19:12:50.684734 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:12:50.684707 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3093709-1f70-42c1-8a2d-e82ac699b83d-maas-api-service-ca\") pod \"d3093709-1f70-42c1-8a2d-e82ac699b83d\" (UID: \"d3093709-1f70-42c1-8a2d-e82ac699b83d\") " Jul 29 19:12:50.685048 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:12:50.685024 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3093709-1f70-42c1-8a2d-e82ac699b83d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d3093709-1f70-42c1-8a2d-e82ac699b83d" (UID: "d3093709-1f70-42c1-8a2d-e82ac699b83d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 19:12:50.785781 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:12:50.785713 2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3093709-1f70-42c1-8a2d-e82ac699b83d-maas-api-service-ca\") on node \"ip-10-0-140-205.ec2.internal\" DevicePath \"\"" Jul 29 19:12:51.501739 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:12:51.501696 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e3eaaf-jgf5c" Jul 29 19:12:51.501739 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:12:51.501727 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e3eaaf-jgf5c" event={"ID":"d3093709-1f70-42c1-8a2d-e82ac699b83d","Type":"ContainerDied","Data":"108a98900637b67bfc62c5cd0183a6f198e34905097e44a19915351948e86df6"} Jul 29 19:12:51.501937 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:12:51.501779 2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="108a98900637b67bfc62c5cd0183a6f198e34905097e44a19915351948e86df6" Jul 29 19:12:57.716787 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:12:57.716746 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:12:57.785508 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:12:57.785468 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:12:57.847002 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:12:57.846946 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:12:57.895776 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:12:57.895740 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e3eaaf-jgf5c"] Jul 29 19:12:57.898730 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:12:57.898706 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e3eaaf-jgf5c"] Jul 29 19:12:57.910826 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:12:57.910799 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:12:58.326768 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:12:58.326726 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:12:59.354768 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:12:59.354733 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d3093709-1f70-42c1-8a2d-e82ac699b83d" path="/var/lib/kubelet/pods/d3093709-1f70-42c1-8a2d-e82ac699b83d/volumes" Jul 29 19:13:02.130104 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:13:02.130037 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-5b4a8e/test-model-5b4a8e-kserve-7b5cb5f6cf-c7jd5" podUID="4479ff91-eaa7-4329-9e86-6ecacc061f59" containerName="main" containerID="cri-o://96836be905ddd07225c91304f0dc7ae05341f6495ad9b9badc282374e94e02cf" gracePeriod=45 Jul 29 19:13:02.377596 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:13:02.377567 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-5b4a8e/test-model-5b4a8e-kserve-7b5cb5f6cf-c7jd5" Jul 29 19:13:02.489086 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:13:02.488993 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4479ff91-eaa7-4329-9e86-6ecacc061f59-tls-certs\") pod \"4479ff91-eaa7-4329-9e86-6ecacc061f59\" (UID: \"4479ff91-eaa7-4329-9e86-6ecacc061f59\") " Jul 29 19:13:02.489086 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:13:02.489041 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4479ff91-eaa7-4329-9e86-6ecacc061f59-home\") pod \"4479ff91-eaa7-4329-9e86-6ecacc061f59\" (UID: \"4479ff91-eaa7-4329-9e86-6ecacc061f59\") " Jul 29 19:13:02.489086 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:13:02.489058 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4479ff91-eaa7-4329-9e86-6ecacc061f59-model-cache\") pod \"4479ff91-eaa7-4329-9e86-6ecacc061f59\" (UID: \"4479ff91-eaa7-4329-9e86-6ecacc061f59\") " Jul 29 19:13:02.489396 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:13:02.489093 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4479ff91-eaa7-4329-9e86-6ecacc061f59-dshm\") pod \"4479ff91-eaa7-4329-9e86-6ecacc061f59\" (UID: \"4479ff91-eaa7-4329-9e86-6ecacc061f59\") " Jul 29 19:13:02.489396 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:13:02.489128 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d67h2\" (UniqueName: \"kubernetes.io/projected/4479ff91-eaa7-4329-9e86-6ecacc061f59-kube-api-access-d67h2\") pod \"4479ff91-eaa7-4329-9e86-6ecacc061f59\" (UID: \"4479ff91-eaa7-4329-9e86-6ecacc061f59\") " Jul 29 19:13:02.489396 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:13:02.489162 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4479ff91-eaa7-4329-9e86-6ecacc061f59-tmp-dir\") pod \"4479ff91-eaa7-4329-9e86-6ecacc061f59\" (UID: \"4479ff91-eaa7-4329-9e86-6ecacc061f59\") " Jul 29 19:13:02.489396 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:13:02.489290 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4479ff91-eaa7-4329-9e86-6ecacc061f59-home" (OuterVolumeSpecName: "home") pod "4479ff91-eaa7-4329-9e86-6ecacc061f59" (UID: "4479ff91-eaa7-4329-9e86-6ecacc061f59"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 19:13:02.489630 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:13:02.489413 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4479ff91-eaa7-4329-9e86-6ecacc061f59-model-cache" (OuterVolumeSpecName: "model-cache") pod "4479ff91-eaa7-4329-9e86-6ecacc061f59" (UID: "4479ff91-eaa7-4329-9e86-6ecacc061f59"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 19:13:02.489630 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:13:02.489481 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4479ff91-eaa7-4329-9e86-6ecacc061f59-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4479ff91-eaa7-4329-9e86-6ecacc061f59" (UID: "4479ff91-eaa7-4329-9e86-6ecacc061f59"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 19:13:02.489739 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:13:02.489626 2560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4479ff91-eaa7-4329-9e86-6ecacc061f59-tmp-dir\") on node \"ip-10-0-140-205.ec2.internal\" DevicePath \"\"" Jul 29 19:13:02.489739 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:13:02.489648 2560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4479ff91-eaa7-4329-9e86-6ecacc061f59-home\") on node \"ip-10-0-140-205.ec2.internal\" DevicePath \"\"" Jul 29 19:13:02.489739 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:13:02.489661 2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4479ff91-eaa7-4329-9e86-6ecacc061f59-model-cache\") on node \"ip-10-0-140-205.ec2.internal\" DevicePath \"\"" Jul 29 19:13:02.491431 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:13:02.491404 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4479ff91-eaa7-4329-9e86-6ecacc061f59-dshm" (OuterVolumeSpecName: "dshm") pod "4479ff91-eaa7-4329-9e86-6ecacc061f59" (UID: "4479ff91-eaa7-4329-9e86-6ecacc061f59"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 19:13:02.491538 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:13:02.491505 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4479ff91-eaa7-4329-9e86-6ecacc061f59-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4479ff91-eaa7-4329-9e86-6ecacc061f59" (UID: "4479ff91-eaa7-4329-9e86-6ecacc061f59"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 19:13:02.491580 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:13:02.491556 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4479ff91-eaa7-4329-9e86-6ecacc061f59-kube-api-access-d67h2" (OuterVolumeSpecName: "kube-api-access-d67h2") pod "4479ff91-eaa7-4329-9e86-6ecacc061f59" (UID: "4479ff91-eaa7-4329-9e86-6ecacc061f59"). InnerVolumeSpecName "kube-api-access-d67h2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 19:13:02.561970 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:13:02.561934 2560 generic.go:356] "Generic (PLEG): container finished" podID="4479ff91-eaa7-4329-9e86-6ecacc061f59" containerID="96836be905ddd07225c91304f0dc7ae05341f6495ad9b9badc282374e94e02cf" exitCode=0 Jul 29 19:13:02.562098 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:13:02.561982 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-5b4a8e/test-model-5b4a8e-kserve-7b5cb5f6cf-c7jd5" Jul 29 19:13:02.562098 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:13:02.561984 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-5b4a8e/test-model-5b4a8e-kserve-7b5cb5f6cf-c7jd5" event={"ID":"4479ff91-eaa7-4329-9e86-6ecacc061f59","Type":"ContainerDied","Data":"96836be905ddd07225c91304f0dc7ae05341f6495ad9b9badc282374e94e02cf"} Jul 29 19:13:02.562098 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:13:02.562079 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-5b4a8e/test-model-5b4a8e-kserve-7b5cb5f6cf-c7jd5" event={"ID":"4479ff91-eaa7-4329-9e86-6ecacc061f59","Type":"ContainerDied","Data":"bb0ddd72a08c38f1a40f7b49713409a836fdcc988c540902927ed69a0729b019"} Jul 29 19:13:02.562098 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:13:02.562099 2560 scope.go:117] "RemoveContainer" containerID="96836be905ddd07225c91304f0dc7ae05341f6495ad9b9badc282374e94e02cf" Jul 29 19:13:02.571632 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:13:02.571614 2560 scope.go:117] "RemoveContainer" containerID="96836be905ddd07225c91304f0dc7ae05341f6495ad9b9badc282374e94e02cf" Jul 29 19:13:02.571867 ip-10-0-140-205 kubenswrapper[2560]: E0729 19:13:02.571850 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96836be905ddd07225c91304f0dc7ae05341f6495ad9b9badc282374e94e02cf\": container with ID starting with 96836be905ddd07225c91304f0dc7ae05341f6495ad9b9badc282374e94e02cf not found: ID does not exist" containerID="96836be905ddd07225c91304f0dc7ae05341f6495ad9b9badc282374e94e02cf" Jul 29 19:13:02.571917 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:13:02.571875 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96836be905ddd07225c91304f0dc7ae05341f6495ad9b9badc282374e94e02cf"} err="failed to get container status \"96836be905ddd07225c91304f0dc7ae05341f6495ad9b9badc282374e94e02cf\": rpc error: code = NotFound desc = could not find container \"96836be905ddd07225c91304f0dc7ae05341f6495ad9b9badc282374e94e02cf\": container with ID starting with 96836be905ddd07225c91304f0dc7ae05341f6495ad9b9badc282374e94e02cf not found: ID does not exist" Jul 29 19:13:02.590034 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:13:02.590007 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d67h2\" (UniqueName: \"kubernetes.io/projected/4479ff91-eaa7-4329-9e86-6ecacc061f59-kube-api-access-d67h2\") on node \"ip-10-0-140-205.ec2.internal\" DevicePath \"\"" Jul 29 19:13:02.590034 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:13:02.590030 2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4479ff91-eaa7-4329-9e86-6ecacc061f59-tls-certs\") on node \"ip-10-0-140-205.ec2.internal\" DevicePath \"\"" Jul 29 19:13:02.590034 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:13:02.590038 2560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4479ff91-eaa7-4329-9e86-6ecacc061f59-dshm\") on node \"ip-10-0-140-205.ec2.internal\" DevicePath \"\"" Jul 29 19:13:02.591343 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:13:02.591324 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-5b4a8e/test-model-5b4a8e-kserve-7b5cb5f6cf-c7jd5"] Jul 29 19:13:02.599476 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:13:02.599458 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-5b4a8e/test-model-5b4a8e-kserve-7b5cb5f6cf-c7jd5"] Jul 29 19:13:03.353722 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:13:03.353691 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4479ff91-eaa7-4329-9e86-6ecacc061f59" path="/var/lib/kubelet/pods/4479ff91-eaa7-4329-9e86-6ecacc061f59/volumes" Jul 29 19:13:04.491236 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:13:04.491202 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:13:04.560595 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:13:04.560558 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:13:04.780836 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:13:04.780805 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:13:05.014126 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:13:05.014088 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:13:05.133806 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:13:05.133727 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5b4a8e-pg6qc"] Jul 29 19:13:05.134142 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:13:05.134129 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d3093709-1f70-42c1-8a2d-e82ac699b83d" containerName="revoke-keys" Jul 29 19:13:05.134187 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:13:05.134144 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="d3093709-1f70-42c1-8a2d-e82ac699b83d" containerName="revoke-keys" Jul 29 19:13:05.134187 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:13:05.134152 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4479ff91-eaa7-4329-9e86-6ecacc061f59" containerName="main" Jul 29 19:13:05.134187 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:13:05.134157 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="4479ff91-eaa7-4329-9e86-6ecacc061f59" containerName="main" Jul 29 19:13:05.134286 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:13:05.134219 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="4479ff91-eaa7-4329-9e86-6ecacc061f59" containerName="main" Jul 29 19:13:05.134286 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:13:05.134227 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="d3093709-1f70-42c1-8a2d-e82ac699b83d" containerName="revoke-keys" Jul 29 19:13:05.138482 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:13:05.138467 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5b4a8e-pg6qc" Jul 29 19:13:05.140996 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:13:05.140980 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vgb9z\"" Jul 29 19:13:05.147421 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:13:05.147389 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5b4a8e-pg6qc"] Jul 29 19:13:05.214473 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:13:05.214446 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/183c2db0-222b-4f8d-bf3c-141b0f0c0c79-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-5b4a8e-pg6qc\" (UID: \"183c2db0-222b-4f8d-bf3c-141b0f0c0c79\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5b4a8e-pg6qc" Jul 29 19:13:05.232382 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:13:05.232328 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:13:05.315298 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:13:05.315273 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/183c2db0-222b-4f8d-bf3c-141b0f0c0c79-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-5b4a8e-pg6qc\" (UID: \"183c2db0-222b-4f8d-bf3c-141b0f0c0c79\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5b4a8e-pg6qc" Jul 29 19:13:05.315885 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:13:05.315865 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/183c2db0-222b-4f8d-bf3c-141b0f0c0c79-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-5b4a8e-pg6qc\" (UID: \"183c2db0-222b-4f8d-bf3c-141b0f0c0c79\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5b4a8e-pg6qc" Jul 29 19:13:05.450053 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:13:05.449978 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5b4a8e-pg6qc" Jul 29 19:13:05.582076 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:13:05.582046 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5b4a8e-pg6qc"] Jul 29 19:13:05.584381 ip-10-0-140-205 kubenswrapper[2560]: W0729 19:13:05.584325 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod183c2db0_222b_4f8d_bf3c_141b0f0c0c79.slice/crio-2da6c8932072c28f5f32640a5fd1d79c04edff3fc3ebfb09efe31b7d67bd863d WatchSource:0}: Error finding container 2da6c8932072c28f5f32640a5fd1d79c04edff3fc3ebfb09efe31b7d67bd863d: Status 404 returned error can't find the container with id 2da6c8932072c28f5f32640a5fd1d79c04edff3fc3ebfb09efe31b7d67bd863d Jul 29 19:13:06.590158 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:13:06.590120 2560 generic.go:356] "Generic (PLEG): container finished" podID="183c2db0-222b-4f8d-bf3c-141b0f0c0c79" containerID="0bae1515f2808935a3ec0ad1df767ae90a47732bce0cdb9d6afcb3999c198665" exitCode=0 Jul 29 19:13:06.590543 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:13:06.590209 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5b4a8e-pg6qc" event={"ID":"183c2db0-222b-4f8d-bf3c-141b0f0c0c79","Type":"ContainerDied","Data":"0bae1515f2808935a3ec0ad1df767ae90a47732bce0cdb9d6afcb3999c198665"} Jul 29 19:13:06.590543 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:13:06.590256 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5b4a8e-pg6qc" event={"ID":"183c2db0-222b-4f8d-bf3c-141b0f0c0c79","Type":"ContainerStarted","Data":"2da6c8932072c28f5f32640a5fd1d79c04edff3fc3ebfb09efe31b7d67bd863d"} Jul 29 19:13:07.727265 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:13:07.727240 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5b4a8e-pg6qc" Jul 29 19:13:07.838297 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:13:07.838267 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/183c2db0-222b-4f8d-bf3c-141b0f0c0c79-maas-api-service-ca\") pod \"183c2db0-222b-4f8d-bf3c-141b0f0c0c79\" (UID: \"183c2db0-222b-4f8d-bf3c-141b0f0c0c79\") " Jul 29 19:13:07.838625 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:13:07.838603 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/183c2db0-222b-4f8d-bf3c-141b0f0c0c79-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "183c2db0-222b-4f8d-bf3c-141b0f0c0c79" (UID: "183c2db0-222b-4f8d-bf3c-141b0f0c0c79"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 19:13:07.939321 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:13:07.939254 2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/183c2db0-222b-4f8d-bf3c-141b0f0c0c79-maas-api-service-ca\") on node \"ip-10-0-140-205.ec2.internal\" DevicePath \"\"" Jul 29 19:13:08.602237 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:13:08.602198 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5b4a8e-pg6qc" event={"ID":"183c2db0-222b-4f8d-bf3c-141b0f0c0c79","Type":"ContainerDied","Data":"2da6c8932072c28f5f32640a5fd1d79c04edff3fc3ebfb09efe31b7d67bd863d"} Jul 29 19:13:08.602429 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:13:08.602253 2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2da6c8932072c28f5f32640a5fd1d79c04edff3fc3ebfb09efe31b7d67bd863d" Jul 29 19:13:08.602429 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:13:08.602217 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5b4a8e-pg6qc" Jul 29 19:13:15.648815 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:13:15.648734 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:13:15.777840 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:13:15.777802 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:13:15.853645 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:13:15.853610 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:13:15.913122 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:13:15.913023 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5b4a8e-pg6qc"] Jul 29 19:13:15.920466 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:13:15.918007 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5b4a8e-pg6qc"] Jul 29 19:13:17.037928 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:13:17.037894 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:13:17.106313 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:13:17.106277 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:13:17.354028 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:13:17.353950 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="183c2db0-222b-4f8d-bf3c-141b0f0c0c79" path="/var/lib/kubelet/pods/183c2db0-222b-4f8d-bf3c-141b0f0c0c79/volumes" Jul 29 19:13:22.329011 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:13:22.328977 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:13:22.400414 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:13:22.400381 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:13:22.618781 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:13:22.618696 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:13:22.868524 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:13:22.868488 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:13:24.199938 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:13:24.199904 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:13:24.302959 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:13:24.302924 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:13:26.753542 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:13:26.753501 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:13:26.842038 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:13:26.842008 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:13:27.030153 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:13:27.030072 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:13:27.155847 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:13:27.155811 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:13:27.468900 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:13:27.468824 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:13:27.573234 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:13:27.573204 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:13:27.778211 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:13:27.778169 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:13:51.826291 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:13:51.826261 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:13:52.357272 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:13:52.357223 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:13:52.787176 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:13:52.787144 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:13:53.136495 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:13:53.136409 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:13:53.328943 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:13:53.328913 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:13:55.930525 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:13:55.930488 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:13:56.031771 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:13:56.031739 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:13:56.229939 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:13:56.229841 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:13:56.333652 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:13:56.333619 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:13:56.475691 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:13:56.475652 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:13:56.964525 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:13:56.964496 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:13:57.107955 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:13:57.107918 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:13:57.939303 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:13:57.939274 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:14:13.919921 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:14:13.919884 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:14:17.449883 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:14:17.449854 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:14:21.002504 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:14:21.002467 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:14:24.223107 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:14:24.223070 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1dcc27-nn22n"] Jul 29 19:14:24.223666 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:14:24.223646 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="183c2db0-222b-4f8d-bf3c-141b0f0c0c79" containerName="revoke-keys" Jul 29 19:14:24.223744 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:14:24.223670 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="183c2db0-222b-4f8d-bf3c-141b0f0c0c79" containerName="revoke-keys" Jul 29 19:14:24.223831 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:14:24.223818 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="183c2db0-222b-4f8d-bf3c-141b0f0c0c79" containerName="revoke-keys" Jul 29 19:14:24.226881 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:14:24.226860 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1dcc27-nn22n" Jul 29 19:14:24.229284 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:14:24.229267 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vgb9z\"" Jul 29 19:14:24.235342 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:14:24.235321 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1dcc27-nn22n"] Jul 29 19:14:24.317778 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:14:24.317734 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8ffb35a-3b9c-416c-8cc0-86bae686df2c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-1dcc27-nn22n\" (UID: \"f8ffb35a-3b9c-416c-8cc0-86bae686df2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1dcc27-nn22n" Jul 29 19:14:24.364144 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:14:24.364107 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:14:24.418499 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:14:24.418468 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8ffb35a-3b9c-416c-8cc0-86bae686df2c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-1dcc27-nn22n\" (UID: \"f8ffb35a-3b9c-416c-8cc0-86bae686df2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1dcc27-nn22n" Jul 29 19:14:24.419067 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:14:24.419047 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8ffb35a-3b9c-416c-8cc0-86bae686df2c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-1dcc27-nn22n\" (UID: \"f8ffb35a-3b9c-416c-8cc0-86bae686df2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1dcc27-nn22n" Jul 29 19:14:24.539431 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:14:24.539403 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1dcc27-nn22n" Jul 29 19:14:24.675194 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:14:24.675170 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1dcc27-nn22n"] Jul 29 19:14:24.677133 ip-10-0-140-205 kubenswrapper[2560]: W0729 19:14:24.677100 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8ffb35a_3b9c_416c_8cc0_86bae686df2c.slice/crio-bddf5784c4375ff096c458e595e3df123ac87b5a86ec34e6a877b5bbc0c69fec WatchSource:0}: Error finding container bddf5784c4375ff096c458e595e3df123ac87b5a86ec34e6a877b5bbc0c69fec: Status 404 returned error can't find the container with id bddf5784c4375ff096c458e595e3df123ac87b5a86ec34e6a877b5bbc0c69fec Jul 29 19:14:25.018112 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:14:25.018077 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1dcc27-nn22n" event={"ID":"f8ffb35a-3b9c-416c-8cc0-86bae686df2c","Type":"ContainerStarted","Data":"46ee5e97b80451d80b8ad4946130b1e6da2a169b4b5fa294ec850fe0585e892c"} Jul 29 19:14:25.018300 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:14:25.018133 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1dcc27-nn22n" event={"ID":"f8ffb35a-3b9c-416c-8cc0-86bae686df2c","Type":"ContainerStarted","Data":"bddf5784c4375ff096c458e595e3df123ac87b5a86ec34e6a877b5bbc0c69fec"} Jul 29 19:14:25.034886 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:14:25.034841 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1dcc27-nn22n" podStartSLOduration=1.034826024 podStartE2EDuration="1.034826024s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:14:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:14:25.03455272 +0000 UTC m=+2400.260221609" watchObservedRunningTime="2026-07-29 19:14:25.034826024 +0000 UTC m=+2400.260494895" Jul 29 19:14:25.583175 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:14:25.583031 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-v4n5x_1727fb40-fbed-4984-91d4-a5890d0f4103/ovn-acl-logging/0.log" Jul 29 19:14:25.593753 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:14:25.590852 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-v4n5x_1727fb40-fbed-4984-91d4-a5890d0f4103/ovn-acl-logging/0.log" Jul 29 19:14:26.024437 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:14:26.024406 2560 generic.go:356] "Generic (PLEG): container finished" podID="f8ffb35a-3b9c-416c-8cc0-86bae686df2c" containerID="46ee5e97b80451d80b8ad4946130b1e6da2a169b4b5fa294ec850fe0585e892c" exitCode=0 Jul 29 19:14:26.024598 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:14:26.024445 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1dcc27-nn22n" event={"ID":"f8ffb35a-3b9c-416c-8cc0-86bae686df2c","Type":"ContainerDied","Data":"46ee5e97b80451d80b8ad4946130b1e6da2a169b4b5fa294ec850fe0585e892c"} Jul 29 19:14:27.172696 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:14:27.172672 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1dcc27-nn22n" Jul 29 19:14:27.242556 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:14:27.242528 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8ffb35a-3b9c-416c-8cc0-86bae686df2c-maas-api-service-ca\") pod \"f8ffb35a-3b9c-416c-8cc0-86bae686df2c\" (UID: \"f8ffb35a-3b9c-416c-8cc0-86bae686df2c\") " Jul 29 19:14:27.242863 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:14:27.242841 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8ffb35a-3b9c-416c-8cc0-86bae686df2c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f8ffb35a-3b9c-416c-8cc0-86bae686df2c" (UID: "f8ffb35a-3b9c-416c-8cc0-86bae686df2c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 19:14:27.343403 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:14:27.343315 2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8ffb35a-3b9c-416c-8cc0-86bae686df2c-maas-api-service-ca\") on node \"ip-10-0-140-205.ec2.internal\" DevicePath \"\"" Jul 29 19:14:28.036897 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:14:28.036866 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1dcc27-nn22n" Jul 29 19:14:28.036897 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:14:28.036895 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1dcc27-nn22n" event={"ID":"f8ffb35a-3b9c-416c-8cc0-86bae686df2c","Type":"ContainerDied","Data":"bddf5784c4375ff096c458e595e3df123ac87b5a86ec34e6a877b5bbc0c69fec"} Jul 29 19:14:28.037109 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:14:28.036938 2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bddf5784c4375ff096c458e595e3df123ac87b5a86ec34e6a877b5bbc0c69fec" Jul 29 19:14:33.127530 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:14:33.127493 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:14:33.226205 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:14:33.226175 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:14:33.303536 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:14:33.303499 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1dcc27-nn22n"] Jul 29 19:14:33.307300 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:14:33.307275 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1dcc27-nn22n"] Jul 29 19:14:33.326962 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:14:33.326935 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:14:33.355178 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:14:33.355151 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f8ffb35a-3b9c-416c-8cc0-86bae686df2c" path="/var/lib/kubelet/pods/f8ffb35a-3b9c-416c-8cc0-86bae686df2c/volumes" Jul 29 19:14:33.694078 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:14:33.694052 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:14:39.503560 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:14:39.503507 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:14:39.574551 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:14:39.574518 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:14:39.806519 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:14:39.806427 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:14:40.046285 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:14:40.046234 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:14:40.170333 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:14:40.170260 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-32da20-mfcxr"] Jul 29 19:14:40.170657 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:14:40.170645 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8ffb35a-3b9c-416c-8cc0-86bae686df2c" containerName="revoke-keys" Jul 29 19:14:40.170699 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:14:40.170658 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8ffb35a-3b9c-416c-8cc0-86bae686df2c" containerName="revoke-keys" Jul 29 19:14:40.170733 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:14:40.170720 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="f8ffb35a-3b9c-416c-8cc0-86bae686df2c" containerName="revoke-keys" Jul 29 19:14:40.173919 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:14:40.173903 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-32da20-mfcxr" Jul 29 19:14:40.176195 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:14:40.176175 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vgb9z\"" Jul 29 19:14:40.182217 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:14:40.182196 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-32da20-mfcxr"] Jul 29 19:14:40.257284 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:14:40.257252 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dee3b0db-3920-42b0-b1c6-cc22230b29fc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-32da20-mfcxr\" (UID: \"dee3b0db-3920-42b0-b1c6-cc22230b29fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-32da20-mfcxr" Jul 29 19:14:40.263516 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:14:40.263491 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:14:40.357935 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:14:40.357895 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dee3b0db-3920-42b0-b1c6-cc22230b29fc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-32da20-mfcxr\" (UID: \"dee3b0db-3920-42b0-b1c6-cc22230b29fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-32da20-mfcxr" Jul 29 19:14:40.358505 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:14:40.358488 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dee3b0db-3920-42b0-b1c6-cc22230b29fc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-32da20-mfcxr\" (UID: \"dee3b0db-3920-42b0-b1c6-cc22230b29fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-32da20-mfcxr" Jul 29 19:14:40.484760 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:14:40.484695 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-32da20-mfcxr" Jul 29 19:14:40.620154 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:14:40.620125 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-32da20-mfcxr"] Jul 29 19:14:40.622567 ip-10-0-140-205 kubenswrapper[2560]: W0729 19:14:40.622536 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddee3b0db_3920_42b0_b1c6_cc22230b29fc.slice/crio-021beee5827dac0f5ce1958ec7396c33c27e0da1e22659122afdf01378f3cc7f WatchSource:0}: Error finding container 021beee5827dac0f5ce1958ec7396c33c27e0da1e22659122afdf01378f3cc7f: Status 404 returned error can't find the container with id 021beee5827dac0f5ce1958ec7396c33c27e0da1e22659122afdf01378f3cc7f Jul 29 19:14:41.111111 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:14:41.111027 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-32da20-mfcxr" event={"ID":"dee3b0db-3920-42b0-b1c6-cc22230b29fc","Type":"ContainerStarted","Data":"a7ac7ebeaf485f0987ce48d6366597a17c8af627f83d654db6a05769076db9a7"} Jul 29 19:14:41.111111 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:14:41.111088 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-32da20-mfcxr" event={"ID":"dee3b0db-3920-42b0-b1c6-cc22230b29fc","Type":"ContainerStarted","Data":"021beee5827dac0f5ce1958ec7396c33c27e0da1e22659122afdf01378f3cc7f"} Jul 29 19:14:41.128656 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:14:41.128603 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-32da20-mfcxr" podStartSLOduration=1.128587045 podStartE2EDuration="1.128587045s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:14:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:14:41.125567509 +0000 UTC m=+2416.351236372" watchObservedRunningTime="2026-07-29 19:14:41.128587045 +0000 UTC m=+2416.354255915" Jul 29 19:14:42.117791 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:14:42.117754 2560 generic.go:356] "Generic (PLEG): container finished" podID="dee3b0db-3920-42b0-b1c6-cc22230b29fc" containerID="a7ac7ebeaf485f0987ce48d6366597a17c8af627f83d654db6a05769076db9a7" exitCode=0 Jul 29 19:14:42.118162 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:14:42.117812 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-32da20-mfcxr" event={"ID":"dee3b0db-3920-42b0-b1c6-cc22230b29fc","Type":"ContainerDied","Data":"a7ac7ebeaf485f0987ce48d6366597a17c8af627f83d654db6a05769076db9a7"} Jul 29 19:14:43.251648 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:14:43.251624 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-32da20-mfcxr" Jul 29 19:14:43.380707 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:14:43.380634 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dee3b0db-3920-42b0-b1c6-cc22230b29fc-maas-api-service-ca\") pod \"dee3b0db-3920-42b0-b1c6-cc22230b29fc\" (UID: \"dee3b0db-3920-42b0-b1c6-cc22230b29fc\") " Jul 29 19:14:43.380954 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:14:43.380932 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dee3b0db-3920-42b0-b1c6-cc22230b29fc-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "dee3b0db-3920-42b0-b1c6-cc22230b29fc" (UID: "dee3b0db-3920-42b0-b1c6-cc22230b29fc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 19:14:43.481637 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:14:43.481585 2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dee3b0db-3920-42b0-b1c6-cc22230b29fc-maas-api-service-ca\") on node \"ip-10-0-140-205.ec2.internal\" DevicePath \"\"" Jul 29 19:14:44.129638 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:14:44.129607 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-32da20-mfcxr" event={"ID":"dee3b0db-3920-42b0-b1c6-cc22230b29fc","Type":"ContainerDied","Data":"021beee5827dac0f5ce1958ec7396c33c27e0da1e22659122afdf01378f3cc7f"} Jul 29 19:14:44.129824 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:14:44.129660 2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="021beee5827dac0f5ce1958ec7396c33c27e0da1e22659122afdf01378f3cc7f" Jul 29 19:14:44.129824 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:14:44.129661 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-32da20-mfcxr" Jul 29 19:14:44.657279 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:14:44.657241 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:14:55.642063 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:14:55.641968 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:14:55.741996 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:14:55.741966 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:14:55.836718 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:14:55.836689 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:14:55.888738 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:14:55.888709 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-32da20-mfcxr"] Jul 29 19:14:55.895075 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:14:55.895014 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-32da20-mfcxr"] Jul 29 19:14:55.935192 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:14:55.935166 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:14:56.310686 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:14:56.310657 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:14:57.353844 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:14:57.353808 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dee3b0db-3920-42b0-b1c6-cc22230b29fc" path="/var/lib/kubelet/pods/dee3b0db-3920-42b0-b1c6-cc22230b29fc/volumes" Jul 29 19:15:02.388617 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:15:02.388584 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:15:02.576163 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:15:02.576130 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:15:02.723567 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:15:02.723472 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:15:02.917317 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:15:02.917278 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:15:03.328021 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:15:03.327987 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:15:04.479725 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:15:04.479693 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:15:04.580375 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:15:04.580319 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:15:06.274795 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:15:06.274761 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:15:06.384634 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:15:06.384603 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:15:07.006878 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:15:07.006850 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:15:07.116694 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:15:07.116667 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:15:31.364419 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:15:31.364384 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:15:31.669924 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:15:31.669831 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:15:32.027598 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:15:32.027555 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:15:32.127788 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:15:32.127752 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:15:35.405227 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:15:35.405172 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:15:35.498608 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:15:35.498523 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:15:35.757856 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:15:35.757821 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:15:35.856516 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:15:35.856473 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:15:35.951232 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:15:35.951197 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:15:36.281943 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:15:36.281908 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:15:36.374633 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:15:36.374603 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:16:00.470440 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:16:00.470401 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:16:18.979309 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:16:18.979234 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-d04442/auth-test-model-d04442-kserve-5dc564ddc-25lzw"] Jul 29 19:16:18.979770 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:16:18.979612 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dee3b0db-3920-42b0-b1c6-cc22230b29fc" containerName="revoke-keys" Jul 29 19:16:18.979770 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:16:18.979623 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="dee3b0db-3920-42b0-b1c6-cc22230b29fc" containerName="revoke-keys" Jul 29 19:16:18.979770 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:16:18.979678 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="dee3b0db-3920-42b0-b1c6-cc22230b29fc" containerName="revoke-keys" Jul 29 19:16:18.982628 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:16:18.982612 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-d04442/auth-test-model-d04442-kserve-5dc564ddc-25lzw" Jul 29 19:16:18.985129 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:16:18.985106 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-d04442\"/\"openshift-service-ca.crt\"" Jul 29 19:16:18.986076 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:16:18.986060 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-d04442\"/\"default-dockercfg-bbg88\"" Jul 29 19:16:18.986076 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:16:18.986068 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-d04442\"/\"auth-test-model-d04442-kserve-self-signed-certs\"" Jul 29 19:16:18.986184 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:16:18.986100 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-d04442\"/\"kube-root-ca.crt\"" Jul 29 19:16:18.993845 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:16:18.993815 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-d04442/auth-test-model-d04442-kserve-5dc564ddc-25lzw"] Jul 29 19:16:19.115085 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:16:19.115056 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:16:19.123194 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:16:19.123169 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4f72b487-4a5f-4952-89b1-fea27630d6f6-home\") pod \"auth-test-model-d04442-kserve-5dc564ddc-25lzw\" (UID: \"4f72b487-4a5f-4952-89b1-fea27630d6f6\") " pod="ai-tenant-e2e-shared-a-d04442/auth-test-model-d04442-kserve-5dc564ddc-25lzw" Jul 29 19:16:19.123327 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:16:19.123221 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f72b487-4a5f-4952-89b1-fea27630d6f6-dshm\") pod \"auth-test-model-d04442-kserve-5dc564ddc-25lzw\" (UID: \"4f72b487-4a5f-4952-89b1-fea27630d6f6\") " pod="ai-tenant-e2e-shared-a-d04442/auth-test-model-d04442-kserve-5dc564ddc-25lzw" Jul 29 19:16:19.123327 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:16:19.123279 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2ksb\" (UniqueName: \"kubernetes.io/projected/4f72b487-4a5f-4952-89b1-fea27630d6f6-kube-api-access-j2ksb\") pod \"auth-test-model-d04442-kserve-5dc564ddc-25lzw\" (UID: \"4f72b487-4a5f-4952-89b1-fea27630d6f6\") " pod="ai-tenant-e2e-shared-a-d04442/auth-test-model-d04442-kserve-5dc564ddc-25lzw" Jul 29 19:16:19.123327 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:16:19.123311 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f72b487-4a5f-4952-89b1-fea27630d6f6-tls-certs\") pod \"auth-test-model-d04442-kserve-5dc564ddc-25lzw\" (UID: \"4f72b487-4a5f-4952-89b1-fea27630d6f6\") " pod="ai-tenant-e2e-shared-a-d04442/auth-test-model-d04442-kserve-5dc564ddc-25lzw" Jul 29 19:16:19.123465 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:16:19.123380 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f72b487-4a5f-4952-89b1-fea27630d6f6-tmp-dir\") pod \"auth-test-model-d04442-kserve-5dc564ddc-25lzw\" (UID: \"4f72b487-4a5f-4952-89b1-fea27630d6f6\") " pod="ai-tenant-e2e-shared-a-d04442/auth-test-model-d04442-kserve-5dc564ddc-25lzw" Jul 29 19:16:19.123465 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:16:19.123428 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f72b487-4a5f-4952-89b1-fea27630d6f6-model-cache\") pod \"auth-test-model-d04442-kserve-5dc564ddc-25lzw\" (UID: \"4f72b487-4a5f-4952-89b1-fea27630d6f6\") " pod="ai-tenant-e2e-shared-a-d04442/auth-test-model-d04442-kserve-5dc564ddc-25lzw" Jul 29 19:16:19.224148 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:16:19.224108 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f72b487-4a5f-4952-89b1-fea27630d6f6-dshm\") pod \"auth-test-model-d04442-kserve-5dc564ddc-25lzw\" (UID: \"4f72b487-4a5f-4952-89b1-fea27630d6f6\") " pod="ai-tenant-e2e-shared-a-d04442/auth-test-model-d04442-kserve-5dc564ddc-25lzw" Jul 29 19:16:19.224148 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:16:19.224154 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2ksb\" (UniqueName: \"kubernetes.io/projected/4f72b487-4a5f-4952-89b1-fea27630d6f6-kube-api-access-j2ksb\") pod \"auth-test-model-d04442-kserve-5dc564ddc-25lzw\" (UID: \"4f72b487-4a5f-4952-89b1-fea27630d6f6\") " pod="ai-tenant-e2e-shared-a-d04442/auth-test-model-d04442-kserve-5dc564ddc-25lzw" Jul 29 19:16:19.224443 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:16:19.224181 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f72b487-4a5f-4952-89b1-fea27630d6f6-tls-certs\") pod \"auth-test-model-d04442-kserve-5dc564ddc-25lzw\" (UID: \"4f72b487-4a5f-4952-89b1-fea27630d6f6\") " pod="ai-tenant-e2e-shared-a-d04442/auth-test-model-d04442-kserve-5dc564ddc-25lzw" Jul 29 19:16:19.224443 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:16:19.224200 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f72b487-4a5f-4952-89b1-fea27630d6f6-tmp-dir\") pod \"auth-test-model-d04442-kserve-5dc564ddc-25lzw\" (UID: \"4f72b487-4a5f-4952-89b1-fea27630d6f6\") " pod="ai-tenant-e2e-shared-a-d04442/auth-test-model-d04442-kserve-5dc564ddc-25lzw" Jul 29 19:16:19.224443 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:16:19.224231 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f72b487-4a5f-4952-89b1-fea27630d6f6-model-cache\") pod \"auth-test-model-d04442-kserve-5dc564ddc-25lzw\" (UID: \"4f72b487-4a5f-4952-89b1-fea27630d6f6\") " pod="ai-tenant-e2e-shared-a-d04442/auth-test-model-d04442-kserve-5dc564ddc-25lzw" Jul 29 19:16:19.224443 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:16:19.224264 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4f72b487-4a5f-4952-89b1-fea27630d6f6-home\") pod \"auth-test-model-d04442-kserve-5dc564ddc-25lzw\" (UID: \"4f72b487-4a5f-4952-89b1-fea27630d6f6\") " pod="ai-tenant-e2e-shared-a-d04442/auth-test-model-d04442-kserve-5dc564ddc-25lzw" Jul 29 19:16:19.224689 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:16:19.224661 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f72b487-4a5f-4952-89b1-fea27630d6f6-tmp-dir\") pod \"auth-test-model-d04442-kserve-5dc564ddc-25lzw\" (UID: \"4f72b487-4a5f-4952-89b1-fea27630d6f6\") " pod="ai-tenant-e2e-shared-a-d04442/auth-test-model-d04442-kserve-5dc564ddc-25lzw" Jul 29 19:16:19.224758 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:16:19.224711 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f72b487-4a5f-4952-89b1-fea27630d6f6-model-cache\") pod \"auth-test-model-d04442-kserve-5dc564ddc-25lzw\" (UID: \"4f72b487-4a5f-4952-89b1-fea27630d6f6\") " pod="ai-tenant-e2e-shared-a-d04442/auth-test-model-d04442-kserve-5dc564ddc-25lzw" Jul 29 19:16:19.224758 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:16:19.224717 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4f72b487-4a5f-4952-89b1-fea27630d6f6-home\") pod \"auth-test-model-d04442-kserve-5dc564ddc-25lzw\" (UID: \"4f72b487-4a5f-4952-89b1-fea27630d6f6\") " pod="ai-tenant-e2e-shared-a-d04442/auth-test-model-d04442-kserve-5dc564ddc-25lzw" Jul 29 19:16:19.226565 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:16:19.226544 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f72b487-4a5f-4952-89b1-fea27630d6f6-dshm\") pod \"auth-test-model-d04442-kserve-5dc564ddc-25lzw\" (UID: \"4f72b487-4a5f-4952-89b1-fea27630d6f6\") " pod="ai-tenant-e2e-shared-a-d04442/auth-test-model-d04442-kserve-5dc564ddc-25lzw" Jul 29 19:16:19.226971 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:16:19.226950 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f72b487-4a5f-4952-89b1-fea27630d6f6-tls-certs\") pod \"auth-test-model-d04442-kserve-5dc564ddc-25lzw\" (UID: \"4f72b487-4a5f-4952-89b1-fea27630d6f6\") " pod="ai-tenant-e2e-shared-a-d04442/auth-test-model-d04442-kserve-5dc564ddc-25lzw" Jul 29 19:16:19.232414 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:16:19.232326 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2ksb\" (UniqueName: \"kubernetes.io/projected/4f72b487-4a5f-4952-89b1-fea27630d6f6-kube-api-access-j2ksb\") pod \"auth-test-model-d04442-kserve-5dc564ddc-25lzw\" (UID: \"4f72b487-4a5f-4952-89b1-fea27630d6f6\") " pod="ai-tenant-e2e-shared-a-d04442/auth-test-model-d04442-kserve-5dc564ddc-25lzw" Jul 29 19:16:19.291648 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:16:19.291609 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-d04442/auth-test-model-d04442-kserve-5dc564ddc-25lzw" Jul 29 19:16:19.428716 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:16:19.428694 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-d04442/auth-test-model-d04442-kserve-5dc564ddc-25lzw"] Jul 29 19:16:19.431491 ip-10-0-140-205 kubenswrapper[2560]: W0729 19:16:19.431466 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4f72b487_4a5f_4952_89b1_fea27630d6f6.slice/crio-75045b864ea7e789ee9e9b3b9e08f4a83cfd2ab5e62b7033f2f8709367ab3a7e WatchSource:0}: Error finding container 75045b864ea7e789ee9e9b3b9e08f4a83cfd2ab5e62b7033f2f8709367ab3a7e: Status 404 returned error can't find the container with id 75045b864ea7e789ee9e9b3b9e08f4a83cfd2ab5e62b7033f2f8709367ab3a7e Jul 29 19:16:19.650881 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:16:19.650846 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-d04442/auth-test-model-d04442-kserve-5dc564ddc-25lzw" event={"ID":"4f72b487-4a5f-4952-89b1-fea27630d6f6","Type":"ContainerStarted","Data":"6c6b6ad0f3e13d2ce9560d3bed91f92486ce54c6c5da6979b656e8f047fd9238"} Jul 29 19:16:19.651046 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:16:19.650890 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-d04442/auth-test-model-d04442-kserve-5dc564ddc-25lzw" event={"ID":"4f72b487-4a5f-4952-89b1-fea27630d6f6","Type":"ContainerStarted","Data":"75045b864ea7e789ee9e9b3b9e08f4a83cfd2ab5e62b7033f2f8709367ab3a7e"} Jul 29 19:16:19.671890 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:16:19.671843 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-d04442/auth-test-model-d04442-kserve-5dc564ddc-25lzw" podStartSLOduration=1.671827881 podStartE2EDuration="1.671827881s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:16:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:16:19.670095523 +0000 UTC m=+2514.895764398" watchObservedRunningTime="2026-07-29 19:16:19.671827881 +0000 UTC m=+2514.897496754" Jul 29 19:16:29.291913 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:16:29.291875 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-d04442/auth-test-model-d04442-kserve-5dc564ddc-25lzw" Jul 29 19:16:29.292300 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:16:29.291948 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-d04442/auth-test-model-d04442-kserve-5dc564ddc-25lzw" Jul 29 19:16:29.304393 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:16:29.304373 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-d04442/auth-test-model-d04442-kserve-5dc564ddc-25lzw" Jul 29 19:16:29.718587 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:16:29.718495 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-d04442/auth-test-model-d04442-kserve-5dc564ddc-25lzw" Jul 29 19:16:50.294562 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:16:50.294520 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:16:50.350938 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:16:50.350904 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:16:54.805539 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:16:54.805498 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:18:07.464169 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:18:07.464084 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:18:58.705516 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:18:58.705477 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:19:25.624214 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:19:25.624103 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-v4n5x_1727fb40-fbed-4984-91d4-a5890d0f4103/ovn-acl-logging/0.log" Jul 29 19:19:25.634270 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:19:25.634247 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-v4n5x_1727fb40-fbed-4984-91d4-a5890d0f4103/ovn-acl-logging/0.log" Jul 29 19:19:25.652973 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:19:25.652953 2560 scope.go:117] "RemoveContainer" containerID="0bae1515f2808935a3ec0ad1df767ae90a47732bce0cdb9d6afcb3999c198665" Jul 29 19:19:25.661819 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:19:25.661802 2560 scope.go:117] "RemoveContainer" containerID="c028eef3bf37d38e1d7c1b82b9a023c360381099d0d2486ac0a1f447b5745d59" Jul 29 19:20:25.686439 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:20:25.686402 2560 scope.go:117] "RemoveContainer" containerID="46ee5e97b80451d80b8ad4946130b1e6da2a169b4b5fa294ec850fe0585e892c" Jul 29 19:20:29.364513 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:20:29.364479 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:20:57.273090 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:20:57.273006 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:21:02.969194 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:21:02.969146 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:21:03.036909 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:21:03.036880 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:21:15.609812 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:21:15.609774 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:21:19.320726 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:21:19.320686 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-d04442/auth-test-model-d04442-kserve-5dc564ddc-25lzw"] Jul 29 19:21:19.341870 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:21:19.341838 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:21:20.210931 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:21:20.210902 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:21:25.711977 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:21:25.711938 2560 scope.go:117] "RemoveContainer" containerID="a7ac7ebeaf485f0987ce48d6366597a17c8af627f83d654db6a05769076db9a7" Jul 29 19:21:34.340118 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:21:34.340073 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-d04442/auth-test-model-d04442-kserve-5dc564ddc-25lzw" podUID="4f72b487-4a5f-4952-89b1-fea27630d6f6" containerName="main" containerID="cri-o://6c6b6ad0f3e13d2ce9560d3bed91f92486ce54c6c5da6979b656e8f047fd9238" gracePeriod=45 Jul 29 19:21:34.596602 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:21:34.596543 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-d04442/auth-test-model-d04442-kserve-5dc564ddc-25lzw" Jul 29 19:21:34.763206 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:21:34.763166 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j2ksb\" (UniqueName: \"kubernetes.io/projected/4f72b487-4a5f-4952-89b1-fea27630d6f6-kube-api-access-j2ksb\") pod \"4f72b487-4a5f-4952-89b1-fea27630d6f6\" (UID: \"4f72b487-4a5f-4952-89b1-fea27630d6f6\") " Jul 29 19:21:34.763430 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:21:34.763236 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f72b487-4a5f-4952-89b1-fea27630d6f6-tls-certs\") pod \"4f72b487-4a5f-4952-89b1-fea27630d6f6\" (UID: \"4f72b487-4a5f-4952-89b1-fea27630d6f6\") " Jul 29 19:21:34.763430 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:21:34.763271 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f72b487-4a5f-4952-89b1-fea27630d6f6-tmp-dir\") pod \"4f72b487-4a5f-4952-89b1-fea27630d6f6\" (UID: \"4f72b487-4a5f-4952-89b1-fea27630d6f6\") " Jul 29 19:21:34.763430 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:21:34.763304 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4f72b487-4a5f-4952-89b1-fea27630d6f6-home\") pod \"4f72b487-4a5f-4952-89b1-fea27630d6f6\" (UID: \"4f72b487-4a5f-4952-89b1-fea27630d6f6\") " Jul 29 19:21:34.763430 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:21:34.763323 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f72b487-4a5f-4952-89b1-fea27630d6f6-dshm\") pod \"4f72b487-4a5f-4952-89b1-fea27630d6f6\" (UID: \"4f72b487-4a5f-4952-89b1-fea27630d6f6\") " Jul 29 19:21:34.763430 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:21:34.763342 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f72b487-4a5f-4952-89b1-fea27630d6f6-model-cache\") pod \"4f72b487-4a5f-4952-89b1-fea27630d6f6\" (UID: \"4f72b487-4a5f-4952-89b1-fea27630d6f6\") " Jul 29 19:21:34.763706 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:21:34.763623 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f72b487-4a5f-4952-89b1-fea27630d6f6-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4f72b487-4a5f-4952-89b1-fea27630d6f6" (UID: "4f72b487-4a5f-4952-89b1-fea27630d6f6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 19:21:34.763706 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:21:34.763632 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f72b487-4a5f-4952-89b1-fea27630d6f6-home" (OuterVolumeSpecName: "home") pod "4f72b487-4a5f-4952-89b1-fea27630d6f6" (UID: "4f72b487-4a5f-4952-89b1-fea27630d6f6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 19:21:34.763796 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:21:34.763713 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f72b487-4a5f-4952-89b1-fea27630d6f6-model-cache" (OuterVolumeSpecName: "model-cache") pod "4f72b487-4a5f-4952-89b1-fea27630d6f6" (UID: "4f72b487-4a5f-4952-89b1-fea27630d6f6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 19:21:34.765634 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:21:34.765604 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f72b487-4a5f-4952-89b1-fea27630d6f6-dshm" (OuterVolumeSpecName: "dshm") pod "4f72b487-4a5f-4952-89b1-fea27630d6f6" (UID: "4f72b487-4a5f-4952-89b1-fea27630d6f6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 19:21:34.765763 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:21:34.765682 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f72b487-4a5f-4952-89b1-fea27630d6f6-kube-api-access-j2ksb" (OuterVolumeSpecName: "kube-api-access-j2ksb") pod "4f72b487-4a5f-4952-89b1-fea27630d6f6" (UID: "4f72b487-4a5f-4952-89b1-fea27630d6f6"). InnerVolumeSpecName "kube-api-access-j2ksb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 19:21:34.765816 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:21:34.765768 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f72b487-4a5f-4952-89b1-fea27630d6f6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4f72b487-4a5f-4952-89b1-fea27630d6f6" (UID: "4f72b487-4a5f-4952-89b1-fea27630d6f6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 19:21:34.864491 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:21:34.864397 2560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f72b487-4a5f-4952-89b1-fea27630d6f6-tmp-dir\") on node \"ip-10-0-140-205.ec2.internal\" DevicePath \"\"" Jul 29 19:21:34.864491 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:21:34.864437 2560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4f72b487-4a5f-4952-89b1-fea27630d6f6-home\") on node \"ip-10-0-140-205.ec2.internal\" DevicePath \"\"" Jul 29 19:21:34.864491 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:21:34.864445 2560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f72b487-4a5f-4952-89b1-fea27630d6f6-dshm\") on node \"ip-10-0-140-205.ec2.internal\" DevicePath \"\"" Jul 29 19:21:34.864491 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:21:34.864453 2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f72b487-4a5f-4952-89b1-fea27630d6f6-model-cache\") on node \"ip-10-0-140-205.ec2.internal\" DevicePath \"\"" Jul 29 19:21:34.864491 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:21:34.864463 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j2ksb\" (UniqueName: \"kubernetes.io/projected/4f72b487-4a5f-4952-89b1-fea27630d6f6-kube-api-access-j2ksb\") on node \"ip-10-0-140-205.ec2.internal\" DevicePath \"\"" Jul 29 19:21:34.864491 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:21:34.864470 2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f72b487-4a5f-4952-89b1-fea27630d6f6-tls-certs\") on node \"ip-10-0-140-205.ec2.internal\" DevicePath \"\"" Jul 29 19:21:35.405301 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:21:35.405268 2560 generic.go:356] "Generic (PLEG): container finished" podID="4f72b487-4a5f-4952-89b1-fea27630d6f6" containerID="6c6b6ad0f3e13d2ce9560d3bed91f92486ce54c6c5da6979b656e8f047fd9238" exitCode=0 Jul 29 19:21:35.405698 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:21:35.405305 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-d04442/auth-test-model-d04442-kserve-5dc564ddc-25lzw" Jul 29 19:21:35.405698 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:21:35.405313 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-d04442/auth-test-model-d04442-kserve-5dc564ddc-25lzw" event={"ID":"4f72b487-4a5f-4952-89b1-fea27630d6f6","Type":"ContainerDied","Data":"6c6b6ad0f3e13d2ce9560d3bed91f92486ce54c6c5da6979b656e8f047fd9238"} Jul 29 19:21:35.405698 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:21:35.405375 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-d04442/auth-test-model-d04442-kserve-5dc564ddc-25lzw" event={"ID":"4f72b487-4a5f-4952-89b1-fea27630d6f6","Type":"ContainerDied","Data":"75045b864ea7e789ee9e9b3b9e08f4a83cfd2ab5e62b7033f2f8709367ab3a7e"} Jul 29 19:21:35.405698 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:21:35.405402 2560 scope.go:117] "RemoveContainer" containerID="6c6b6ad0f3e13d2ce9560d3bed91f92486ce54c6c5da6979b656e8f047fd9238" Jul 29 19:21:35.414342 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:21:35.414326 2560 scope.go:117] "RemoveContainer" containerID="6c6b6ad0f3e13d2ce9560d3bed91f92486ce54c6c5da6979b656e8f047fd9238" Jul 29 19:21:35.414624 ip-10-0-140-205 kubenswrapper[2560]: E0729 19:21:35.414602 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c6b6ad0f3e13d2ce9560d3bed91f92486ce54c6c5da6979b656e8f047fd9238\": container with ID starting with 6c6b6ad0f3e13d2ce9560d3bed91f92486ce54c6c5da6979b656e8f047fd9238 not found: ID does not exist" containerID="6c6b6ad0f3e13d2ce9560d3bed91f92486ce54c6c5da6979b656e8f047fd9238" Jul 29 19:21:35.414677 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:21:35.414633 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c6b6ad0f3e13d2ce9560d3bed91f92486ce54c6c5da6979b656e8f047fd9238"} err="failed to get container status \"6c6b6ad0f3e13d2ce9560d3bed91f92486ce54c6c5da6979b656e8f047fd9238\": rpc error: code = NotFound desc = could not find container \"6c6b6ad0f3e13d2ce9560d3bed91f92486ce54c6c5da6979b656e8f047fd9238\": container with ID starting with 6c6b6ad0f3e13d2ce9560d3bed91f92486ce54c6c5da6979b656e8f047fd9238 not found: ID does not exist" Jul 29 19:21:35.422481 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:21:35.422447 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-d04442/auth-test-model-d04442-kserve-5dc564ddc-25lzw"] Jul 29 19:21:35.426873 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:21:35.426853 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-d04442/auth-test-model-d04442-kserve-5dc564ddc-25lzw"] Jul 29 19:21:36.386808 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:21:36.386780 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-d04442/sub-test-model-d04442-kserve-6888657659-xjn9k"] Jul 29 19:21:36.387168 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:21:36.387156 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f72b487-4a5f-4952-89b1-fea27630d6f6" containerName="main" Jul 29 19:21:36.387215 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:21:36.387170 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f72b487-4a5f-4952-89b1-fea27630d6f6" containerName="main" Jul 29 19:21:36.387251 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:21:36.387223 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="4f72b487-4a5f-4952-89b1-fea27630d6f6" containerName="main" Jul 29 19:21:36.391475 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:21:36.391455 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-d04442/sub-test-model-d04442-kserve-6888657659-xjn9k" Jul 29 19:21:36.394014 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:21:36.393997 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-d04442\"/\"openshift-service-ca.crt\"" Jul 29 19:21:36.395057 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:21:36.395040 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-d04442\"/\"kube-root-ca.crt\"" Jul 29 19:21:36.395143 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:21:36.395055 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-d04442\"/\"sub-test-model-d04442-kserve-self-signed-certs\"" Jul 29 19:21:36.395143 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:21:36.395061 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-d04442\"/\"default-dockercfg-bbg88\"" Jul 29 19:21:36.399986 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:21:36.399963 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-d04442/sub-test-model-d04442-kserve-6888657659-xjn9k"] Jul 29 19:21:36.422241 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:21:36.422217 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:21:36.478532 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:21:36.478507 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6be50932-bc16-449f-8459-218aeafac3ef-tmp-dir\") pod \"sub-test-model-d04442-kserve-6888657659-xjn9k\" (UID: \"6be50932-bc16-449f-8459-218aeafac3ef\") " pod="ai-tenant-e2e-shared-a-d04442/sub-test-model-d04442-kserve-6888657659-xjn9k" Jul 29 19:21:36.478620 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:21:36.478542 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6be50932-bc16-449f-8459-218aeafac3ef-model-cache\") pod \"sub-test-model-d04442-kserve-6888657659-xjn9k\" (UID: \"6be50932-bc16-449f-8459-218aeafac3ef\") " pod="ai-tenant-e2e-shared-a-d04442/sub-test-model-d04442-kserve-6888657659-xjn9k" Jul 29 19:21:36.478661 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:21:36.478635 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-th84p\" (UniqueName: \"kubernetes.io/projected/6be50932-bc16-449f-8459-218aeafac3ef-kube-api-access-th84p\") pod \"sub-test-model-d04442-kserve-6888657659-xjn9k\" (UID: \"6be50932-bc16-449f-8459-218aeafac3ef\") " pod="ai-tenant-e2e-shared-a-d04442/sub-test-model-d04442-kserve-6888657659-xjn9k" Jul 29 19:21:36.478697 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:21:36.478674 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6be50932-bc16-449f-8459-218aeafac3ef-dshm\") pod \"sub-test-model-d04442-kserve-6888657659-xjn9k\" (UID: \"6be50932-bc16-449f-8459-218aeafac3ef\") " pod="ai-tenant-e2e-shared-a-d04442/sub-test-model-d04442-kserve-6888657659-xjn9k" Jul 29 19:21:36.478731 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:21:36.478704 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6be50932-bc16-449f-8459-218aeafac3ef-tls-certs\") pod \"sub-test-model-d04442-kserve-6888657659-xjn9k\" (UID: \"6be50932-bc16-449f-8459-218aeafac3ef\") " pod="ai-tenant-e2e-shared-a-d04442/sub-test-model-d04442-kserve-6888657659-xjn9k" Jul 29 19:21:36.478773 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:21:36.478755 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6be50932-bc16-449f-8459-218aeafac3ef-home\") pod \"sub-test-model-d04442-kserve-6888657659-xjn9k\" (UID: \"6be50932-bc16-449f-8459-218aeafac3ef\") " pod="ai-tenant-e2e-shared-a-d04442/sub-test-model-d04442-kserve-6888657659-xjn9k" Jul 29 19:21:36.579965 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:21:36.579922 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-th84p\" (UniqueName: \"kubernetes.io/projected/6be50932-bc16-449f-8459-218aeafac3ef-kube-api-access-th84p\") pod \"sub-test-model-d04442-kserve-6888657659-xjn9k\" (UID: \"6be50932-bc16-449f-8459-218aeafac3ef\") " pod="ai-tenant-e2e-shared-a-d04442/sub-test-model-d04442-kserve-6888657659-xjn9k" Jul 29 19:21:36.579965 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:21:36.579970 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6be50932-bc16-449f-8459-218aeafac3ef-dshm\") pod \"sub-test-model-d04442-kserve-6888657659-xjn9k\" (UID: \"6be50932-bc16-449f-8459-218aeafac3ef\") " pod="ai-tenant-e2e-shared-a-d04442/sub-test-model-d04442-kserve-6888657659-xjn9k" Jul 29 19:21:36.580201 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:21:36.579997 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6be50932-bc16-449f-8459-218aeafac3ef-tls-certs\") pod \"sub-test-model-d04442-kserve-6888657659-xjn9k\" (UID: \"6be50932-bc16-449f-8459-218aeafac3ef\") " pod="ai-tenant-e2e-shared-a-d04442/sub-test-model-d04442-kserve-6888657659-xjn9k" Jul 29 19:21:36.580201 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:21:36.580030 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6be50932-bc16-449f-8459-218aeafac3ef-home\") pod \"sub-test-model-d04442-kserve-6888657659-xjn9k\" (UID: \"6be50932-bc16-449f-8459-218aeafac3ef\") " pod="ai-tenant-e2e-shared-a-d04442/sub-test-model-d04442-kserve-6888657659-xjn9k" Jul 29 19:21:36.580201 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:21:36.580067 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6be50932-bc16-449f-8459-218aeafac3ef-tmp-dir\") pod \"sub-test-model-d04442-kserve-6888657659-xjn9k\" (UID: \"6be50932-bc16-449f-8459-218aeafac3ef\") " pod="ai-tenant-e2e-shared-a-d04442/sub-test-model-d04442-kserve-6888657659-xjn9k" Jul 29 19:21:36.580201 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:21:36.580098 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6be50932-bc16-449f-8459-218aeafac3ef-model-cache\") pod \"sub-test-model-d04442-kserve-6888657659-xjn9k\" (UID: \"6be50932-bc16-449f-8459-218aeafac3ef\") " pod="ai-tenant-e2e-shared-a-d04442/sub-test-model-d04442-kserve-6888657659-xjn9k" Jul 29 19:21:36.580560 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:21:36.580537 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6be50932-bc16-449f-8459-218aeafac3ef-home\") pod \"sub-test-model-d04442-kserve-6888657659-xjn9k\" (UID: \"6be50932-bc16-449f-8459-218aeafac3ef\") " pod="ai-tenant-e2e-shared-a-d04442/sub-test-model-d04442-kserve-6888657659-xjn9k" Jul 29 19:21:36.580637 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:21:36.580559 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6be50932-bc16-449f-8459-218aeafac3ef-tmp-dir\") pod \"sub-test-model-d04442-kserve-6888657659-xjn9k\" (UID: \"6be50932-bc16-449f-8459-218aeafac3ef\") " pod="ai-tenant-e2e-shared-a-d04442/sub-test-model-d04442-kserve-6888657659-xjn9k" Jul 29 19:21:36.580637 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:21:36.580613 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6be50932-bc16-449f-8459-218aeafac3ef-model-cache\") pod \"sub-test-model-d04442-kserve-6888657659-xjn9k\" (UID: \"6be50932-bc16-449f-8459-218aeafac3ef\") " pod="ai-tenant-e2e-shared-a-d04442/sub-test-model-d04442-kserve-6888657659-xjn9k" Jul 29 19:21:36.582624 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:21:36.582601 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6be50932-bc16-449f-8459-218aeafac3ef-dshm\") pod \"sub-test-model-d04442-kserve-6888657659-xjn9k\" (UID: \"6be50932-bc16-449f-8459-218aeafac3ef\") " pod="ai-tenant-e2e-shared-a-d04442/sub-test-model-d04442-kserve-6888657659-xjn9k" Jul 29 19:21:36.582824 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:21:36.582806 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6be50932-bc16-449f-8459-218aeafac3ef-tls-certs\") pod \"sub-test-model-d04442-kserve-6888657659-xjn9k\" (UID: \"6be50932-bc16-449f-8459-218aeafac3ef\") " pod="ai-tenant-e2e-shared-a-d04442/sub-test-model-d04442-kserve-6888657659-xjn9k" Jul 29 19:21:36.588739 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:21:36.588720 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-th84p\" (UniqueName: \"kubernetes.io/projected/6be50932-bc16-449f-8459-218aeafac3ef-kube-api-access-th84p\") pod \"sub-test-model-d04442-kserve-6888657659-xjn9k\" (UID: \"6be50932-bc16-449f-8459-218aeafac3ef\") " pod="ai-tenant-e2e-shared-a-d04442/sub-test-model-d04442-kserve-6888657659-xjn9k" Jul 29 19:21:36.701317 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:21:36.701222 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-d04442/sub-test-model-d04442-kserve-6888657659-xjn9k" Jul 29 19:21:36.835108 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:21:36.835076 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-d04442/sub-test-model-d04442-kserve-6888657659-xjn9k"] Jul 29 19:21:36.837400 ip-10-0-140-205 kubenswrapper[2560]: W0729 19:21:36.837370 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6be50932_bc16_449f_8459_218aeafac3ef.slice/crio-f1ea26186cb030ba54ac9b4021ffbbf6b8fdafa4f7804e3723a96b23ae1d3df1 WatchSource:0}: Error finding container f1ea26186cb030ba54ac9b4021ffbbf6b8fdafa4f7804e3723a96b23ae1d3df1: Status 404 returned error can't find the container with id f1ea26186cb030ba54ac9b4021ffbbf6b8fdafa4f7804e3723a96b23ae1d3df1 Jul 29 19:21:37.354532 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:21:37.354498 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4f72b487-4a5f-4952-89b1-fea27630d6f6" path="/var/lib/kubelet/pods/4f72b487-4a5f-4952-89b1-fea27630d6f6/volumes" Jul 29 19:21:37.418479 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:21:37.418439 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-d04442/sub-test-model-d04442-kserve-6888657659-xjn9k" event={"ID":"6be50932-bc16-449f-8459-218aeafac3ef","Type":"ContainerStarted","Data":"b5108783e77c6b48749a1ff019920b36ba2a2eed83fca76eed2c66a62c1776dd"} Jul 29 19:21:37.418642 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:21:37.418484 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-d04442/sub-test-model-d04442-kserve-6888657659-xjn9k" event={"ID":"6be50932-bc16-449f-8459-218aeafac3ef","Type":"ContainerStarted","Data":"f1ea26186cb030ba54ac9b4021ffbbf6b8fdafa4f7804e3723a96b23ae1d3df1"} Jul 29 19:21:37.437470 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:21:37.437423 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-d04442/sub-test-model-d04442-kserve-6888657659-xjn9k" podStartSLOduration=1.437408901 podStartE2EDuration="1.437408901s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:21:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:21:37.434985619 +0000 UTC m=+2832.660654491" watchObservedRunningTime="2026-07-29 19:21:37.437408901 +0000 UTC m=+2832.663077773" Jul 29 19:21:46.702363 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:21:46.702313 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-d04442/sub-test-model-d04442-kserve-6888657659-xjn9k" Jul 29 19:21:46.702746 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:21:46.702392 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-d04442/sub-test-model-d04442-kserve-6888657659-xjn9k" Jul 29 19:21:46.714547 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:21:46.714517 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-d04442/sub-test-model-d04442-kserve-6888657659-xjn9k" Jul 29 19:21:47.482687 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:21:47.482653 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-d04442/sub-test-model-d04442-kserve-6888657659-xjn9k" Jul 29 19:22:00.704818 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:22:00.704786 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-422d30/sub-test-model-422d30-kserve-5c49746b97-kn695"] Jul 29 19:22:00.708334 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:22:00.708318 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-422d30/sub-test-model-422d30-kserve-5c49746b97-kn695" Jul 29 19:22:00.710749 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:22:00.710728 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-422d30\"/\"default-dockercfg-8b8fl\"" Jul 29 19:22:00.710879 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:22:00.710830 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-422d30\"/\"openshift-service-ca.crt\"" Jul 29 19:22:00.711683 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:22:00.711667 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-422d30\"/\"sub-test-model-422d30-kserve-self-signed-certs\"" Jul 29 19:22:00.711780 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:22:00.711712 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-422d30\"/\"kube-root-ca.crt\"" Jul 29 19:22:00.718987 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:22:00.718965 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-422d30/sub-test-model-422d30-kserve-5c49746b97-kn695"] Jul 29 19:22:00.737445 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:22:00.737412 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:22:00.792084 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:22:00.792060 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/65477033-c1cc-4dd1-9fc0-36e9d5dd0f11-home\") pod \"sub-test-model-422d30-kserve-5c49746b97-kn695\" (UID: \"65477033-c1cc-4dd1-9fc0-36e9d5dd0f11\") " pod="ai-tenant-e2e-shared-b-422d30/sub-test-model-422d30-kserve-5c49746b97-kn695" Jul 29 19:22:00.792195 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:22:00.792088 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/65477033-c1cc-4dd1-9fc0-36e9d5dd0f11-dshm\") pod \"sub-test-model-422d30-kserve-5c49746b97-kn695\" (UID: \"65477033-c1cc-4dd1-9fc0-36e9d5dd0f11\") " pod="ai-tenant-e2e-shared-b-422d30/sub-test-model-422d30-kserve-5c49746b97-kn695" Jul 29 19:22:00.792195 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:22:00.792158 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/65477033-c1cc-4dd1-9fc0-36e9d5dd0f11-tmp-dir\") pod \"sub-test-model-422d30-kserve-5c49746b97-kn695\" (UID: \"65477033-c1cc-4dd1-9fc0-36e9d5dd0f11\") " pod="ai-tenant-e2e-shared-b-422d30/sub-test-model-422d30-kserve-5c49746b97-kn695" Jul 29 19:22:00.792195 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:22:00.792177 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/65477033-c1cc-4dd1-9fc0-36e9d5dd0f11-model-cache\") pod \"sub-test-model-422d30-kserve-5c49746b97-kn695\" (UID: \"65477033-c1cc-4dd1-9fc0-36e9d5dd0f11\") " pod="ai-tenant-e2e-shared-b-422d30/sub-test-model-422d30-kserve-5c49746b97-kn695" Jul 29 19:22:00.792313 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:22:00.792211 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mnzl5\" (UniqueName: \"kubernetes.io/projected/65477033-c1cc-4dd1-9fc0-36e9d5dd0f11-kube-api-access-mnzl5\") pod \"sub-test-model-422d30-kserve-5c49746b97-kn695\" (UID: \"65477033-c1cc-4dd1-9fc0-36e9d5dd0f11\") " pod="ai-tenant-e2e-shared-b-422d30/sub-test-model-422d30-kserve-5c49746b97-kn695" Jul 29 19:22:00.792313 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:22:00.792235 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/65477033-c1cc-4dd1-9fc0-36e9d5dd0f11-tls-certs\") pod \"sub-test-model-422d30-kserve-5c49746b97-kn695\" (UID: \"65477033-c1cc-4dd1-9fc0-36e9d5dd0f11\") " pod="ai-tenant-e2e-shared-b-422d30/sub-test-model-422d30-kserve-5c49746b97-kn695" Jul 29 19:22:00.893733 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:22:00.893705 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/65477033-c1cc-4dd1-9fc0-36e9d5dd0f11-tmp-dir\") pod \"sub-test-model-422d30-kserve-5c49746b97-kn695\" (UID: \"65477033-c1cc-4dd1-9fc0-36e9d5dd0f11\") " pod="ai-tenant-e2e-shared-b-422d30/sub-test-model-422d30-kserve-5c49746b97-kn695" Jul 29 19:22:00.893874 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:22:00.893737 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/65477033-c1cc-4dd1-9fc0-36e9d5dd0f11-model-cache\") pod \"sub-test-model-422d30-kserve-5c49746b97-kn695\" (UID: \"65477033-c1cc-4dd1-9fc0-36e9d5dd0f11\") " pod="ai-tenant-e2e-shared-b-422d30/sub-test-model-422d30-kserve-5c49746b97-kn695" Jul 29 19:22:00.893874 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:22:00.893763 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mnzl5\" (UniqueName: \"kubernetes.io/projected/65477033-c1cc-4dd1-9fc0-36e9d5dd0f11-kube-api-access-mnzl5\") pod \"sub-test-model-422d30-kserve-5c49746b97-kn695\" (UID: \"65477033-c1cc-4dd1-9fc0-36e9d5dd0f11\") " pod="ai-tenant-e2e-shared-b-422d30/sub-test-model-422d30-kserve-5c49746b97-kn695" Jul 29 19:22:00.893987 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:22:00.893875 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/65477033-c1cc-4dd1-9fc0-36e9d5dd0f11-tls-certs\") pod \"sub-test-model-422d30-kserve-5c49746b97-kn695\" (UID: \"65477033-c1cc-4dd1-9fc0-36e9d5dd0f11\") " pod="ai-tenant-e2e-shared-b-422d30/sub-test-model-422d30-kserve-5c49746b97-kn695" Jul 29 19:22:00.893987 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:22:00.893938 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/65477033-c1cc-4dd1-9fc0-36e9d5dd0f11-home\") pod \"sub-test-model-422d30-kserve-5c49746b97-kn695\" (UID: \"65477033-c1cc-4dd1-9fc0-36e9d5dd0f11\") " pod="ai-tenant-e2e-shared-b-422d30/sub-test-model-422d30-kserve-5c49746b97-kn695" Jul 29 19:22:00.893987 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:22:00.893955 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/65477033-c1cc-4dd1-9fc0-36e9d5dd0f11-dshm\") pod \"sub-test-model-422d30-kserve-5c49746b97-kn695\" (UID: \"65477033-c1cc-4dd1-9fc0-36e9d5dd0f11\") " pod="ai-tenant-e2e-shared-b-422d30/sub-test-model-422d30-kserve-5c49746b97-kn695" Jul 29 19:22:00.894160 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:22:00.894136 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/65477033-c1cc-4dd1-9fc0-36e9d5dd0f11-tmp-dir\") pod \"sub-test-model-422d30-kserve-5c49746b97-kn695\" (UID: \"65477033-c1cc-4dd1-9fc0-36e9d5dd0f11\") " pod="ai-tenant-e2e-shared-b-422d30/sub-test-model-422d30-kserve-5c49746b97-kn695" Jul 29 19:22:00.894235 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:22:00.894217 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/65477033-c1cc-4dd1-9fc0-36e9d5dd0f11-model-cache\") pod \"sub-test-model-422d30-kserve-5c49746b97-kn695\" (UID: \"65477033-c1cc-4dd1-9fc0-36e9d5dd0f11\") " pod="ai-tenant-e2e-shared-b-422d30/sub-test-model-422d30-kserve-5c49746b97-kn695" Jul 29 19:22:00.894294 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:22:00.894236 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/65477033-c1cc-4dd1-9fc0-36e9d5dd0f11-home\") pod \"sub-test-model-422d30-kserve-5c49746b97-kn695\" (UID: \"65477033-c1cc-4dd1-9fc0-36e9d5dd0f11\") " pod="ai-tenant-e2e-shared-b-422d30/sub-test-model-422d30-kserve-5c49746b97-kn695" Jul 29 19:22:00.896336 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:22:00.896314 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/65477033-c1cc-4dd1-9fc0-36e9d5dd0f11-dshm\") pod \"sub-test-model-422d30-kserve-5c49746b97-kn695\" (UID: \"65477033-c1cc-4dd1-9fc0-36e9d5dd0f11\") " pod="ai-tenant-e2e-shared-b-422d30/sub-test-model-422d30-kserve-5c49746b97-kn695" Jul 29 19:22:00.896599 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:22:00.896585 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/65477033-c1cc-4dd1-9fc0-36e9d5dd0f11-tls-certs\") pod \"sub-test-model-422d30-kserve-5c49746b97-kn695\" (UID: \"65477033-c1cc-4dd1-9fc0-36e9d5dd0f11\") " pod="ai-tenant-e2e-shared-b-422d30/sub-test-model-422d30-kserve-5c49746b97-kn695" Jul 29 19:22:00.903269 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:22:00.903246 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mnzl5\" (UniqueName: \"kubernetes.io/projected/65477033-c1cc-4dd1-9fc0-36e9d5dd0f11-kube-api-access-mnzl5\") pod \"sub-test-model-422d30-kserve-5c49746b97-kn695\" (UID: \"65477033-c1cc-4dd1-9fc0-36e9d5dd0f11\") " pod="ai-tenant-e2e-shared-b-422d30/sub-test-model-422d30-kserve-5c49746b97-kn695" Jul 29 19:22:01.018175 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:22:01.018151 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-422d30/sub-test-model-422d30-kserve-5c49746b97-kn695" Jul 29 19:22:01.157283 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:22:01.157254 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-422d30/sub-test-model-422d30-kserve-5c49746b97-kn695"] Jul 29 19:22:01.158911 ip-10-0-140-205 kubenswrapper[2560]: W0729 19:22:01.158884 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod65477033_c1cc_4dd1_9fc0_36e9d5dd0f11.slice/crio-cecf467988b5a69c635ffda3da87ddeec2f4864c2123587cb758394a0846adbf WatchSource:0}: Error finding container cecf467988b5a69c635ffda3da87ddeec2f4864c2123587cb758394a0846adbf: Status 404 returned error can't find the container with id cecf467988b5a69c635ffda3da87ddeec2f4864c2123587cb758394a0846adbf Jul 29 19:22:01.549242 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:22:01.549209 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-422d30/sub-test-model-422d30-kserve-5c49746b97-kn695" event={"ID":"65477033-c1cc-4dd1-9fc0-36e9d5dd0f11","Type":"ContainerStarted","Data":"19f8381ba99c4dbd43c626322bab95c71ed02781b9e60d1d362d625a7df70f61"} Jul 29 19:22:01.549242 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:22:01.549247 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-422d30/sub-test-model-422d30-kserve-5c49746b97-kn695" event={"ID":"65477033-c1cc-4dd1-9fc0-36e9d5dd0f11","Type":"ContainerStarted","Data":"cecf467988b5a69c635ffda3da87ddeec2f4864c2123587cb758394a0846adbf"} Jul 29 19:22:01.570445 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:22:01.570396 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-422d30/sub-test-model-422d30-kserve-5c49746b97-kn695" podStartSLOduration=1.570381907 podStartE2EDuration="1.570381907s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:22:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:22:01.567800244 +0000 UTC m=+2856.793469120" watchObservedRunningTime="2026-07-29 19:22:01.570381907 +0000 UTC m=+2856.796050779" Jul 29 19:22:11.018992 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:22:11.018951 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-422d30/sub-test-model-422d30-kserve-5c49746b97-kn695" Jul 29 19:22:11.018992 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:22:11.018998 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-422d30/sub-test-model-422d30-kserve-5c49746b97-kn695" Jul 29 19:22:11.031446 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:22:11.031424 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-422d30/sub-test-model-422d30-kserve-5c49746b97-kn695" Jul 29 19:22:11.614420 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:22:11.614392 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-422d30/sub-test-model-422d30-kserve-5c49746b97-kn695" Jul 29 19:22:57.886300 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:22:57.886261 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-d04442/sub-test-model-d04442-kserve-6888657659-xjn9k"] Jul 29 19:22:57.906299 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:22:57.906271 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:22:58.970454 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:22:58.970414 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-422d30/sub-test-model-422d30-kserve-5c49746b97-kn695"] Jul 29 19:22:58.985323 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:22:58.985283 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:23:10.289688 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:23:10.289642 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:23:12.905607 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:23:12.905563 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-d04442/sub-test-model-d04442-kserve-6888657659-xjn9k" podUID="6be50932-bc16-449f-8459-218aeafac3ef" containerName="main" containerID="cri-o://b5108783e77c6b48749a1ff019920b36ba2a2eed83fca76eed2c66a62c1776dd" gracePeriod=45 Jul 29 19:23:13.150779 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:23:13.150757 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-d04442/sub-test-model-d04442-kserve-6888657659-xjn9k" Jul 29 19:23:13.332362 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:23:13.332318 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6be50932-bc16-449f-8459-218aeafac3ef-dshm\") pod \"6be50932-bc16-449f-8459-218aeafac3ef\" (UID: \"6be50932-bc16-449f-8459-218aeafac3ef\") " Jul 29 19:23:13.332529 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:23:13.332372 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6be50932-bc16-449f-8459-218aeafac3ef-tmp-dir\") pod \"6be50932-bc16-449f-8459-218aeafac3ef\" (UID: \"6be50932-bc16-449f-8459-218aeafac3ef\") " Jul 29 19:23:13.332529 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:23:13.332388 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6be50932-bc16-449f-8459-218aeafac3ef-model-cache\") pod \"6be50932-bc16-449f-8459-218aeafac3ef\" (UID: \"6be50932-bc16-449f-8459-218aeafac3ef\") " Jul 29 19:23:13.332529 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:23:13.332412 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6be50932-bc16-449f-8459-218aeafac3ef-home\") pod \"6be50932-bc16-449f-8459-218aeafac3ef\" (UID: \"6be50932-bc16-449f-8459-218aeafac3ef\") " Jul 29 19:23:13.332529 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:23:13.332482 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-th84p\" (UniqueName: \"kubernetes.io/projected/6be50932-bc16-449f-8459-218aeafac3ef-kube-api-access-th84p\") pod \"6be50932-bc16-449f-8459-218aeafac3ef\" (UID: \"6be50932-bc16-449f-8459-218aeafac3ef\") " Jul 29 19:23:13.332529 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:23:13.332514 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6be50932-bc16-449f-8459-218aeafac3ef-tls-certs\") pod \"6be50932-bc16-449f-8459-218aeafac3ef\" (UID: \"6be50932-bc16-449f-8459-218aeafac3ef\") " Jul 29 19:23:13.332781 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:23:13.332621 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6be50932-bc16-449f-8459-218aeafac3ef-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6be50932-bc16-449f-8459-218aeafac3ef" (UID: "6be50932-bc16-449f-8459-218aeafac3ef"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 19:23:13.332781 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:23:13.332639 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6be50932-bc16-449f-8459-218aeafac3ef-model-cache" (OuterVolumeSpecName: "model-cache") pod "6be50932-bc16-449f-8459-218aeafac3ef" (UID: "6be50932-bc16-449f-8459-218aeafac3ef"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 19:23:13.332781 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:23:13.332683 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6be50932-bc16-449f-8459-218aeafac3ef-home" (OuterVolumeSpecName: "home") pod "6be50932-bc16-449f-8459-218aeafac3ef" (UID: "6be50932-bc16-449f-8459-218aeafac3ef"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 19:23:13.332781 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:23:13.332778 2560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6be50932-bc16-449f-8459-218aeafac3ef-tmp-dir\") on node \"ip-10-0-140-205.ec2.internal\" DevicePath \"\"" Jul 29 19:23:13.332915 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:23:13.332792 2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6be50932-bc16-449f-8459-218aeafac3ef-model-cache\") on node \"ip-10-0-140-205.ec2.internal\" DevicePath \"\"" Jul 29 19:23:13.332915 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:23:13.332800 2560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6be50932-bc16-449f-8459-218aeafac3ef-home\") on node \"ip-10-0-140-205.ec2.internal\" DevicePath \"\"" Jul 29 19:23:13.334600 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:23:13.334572 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6be50932-bc16-449f-8459-218aeafac3ef-dshm" (OuterVolumeSpecName: "dshm") pod "6be50932-bc16-449f-8459-218aeafac3ef" (UID: "6be50932-bc16-449f-8459-218aeafac3ef"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 19:23:13.334707 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:23:13.334622 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6be50932-bc16-449f-8459-218aeafac3ef-kube-api-access-th84p" (OuterVolumeSpecName: "kube-api-access-th84p") pod "6be50932-bc16-449f-8459-218aeafac3ef" (UID: "6be50932-bc16-449f-8459-218aeafac3ef"). InnerVolumeSpecName "kube-api-access-th84p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 19:23:13.334707 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:23:13.334681 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6be50932-bc16-449f-8459-218aeafac3ef-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6be50932-bc16-449f-8459-218aeafac3ef" (UID: "6be50932-bc16-449f-8459-218aeafac3ef"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 19:23:13.433930 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:23:13.433905 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-th84p\" (UniqueName: \"kubernetes.io/projected/6be50932-bc16-449f-8459-218aeafac3ef-kube-api-access-th84p\") on node \"ip-10-0-140-205.ec2.internal\" DevicePath \"\"" Jul 29 19:23:13.433930 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:23:13.433928 2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6be50932-bc16-449f-8459-218aeafac3ef-tls-certs\") on node \"ip-10-0-140-205.ec2.internal\" DevicePath \"\"" Jul 29 19:23:13.434057 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:23:13.433937 2560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6be50932-bc16-449f-8459-218aeafac3ef-dshm\") on node \"ip-10-0-140-205.ec2.internal\" DevicePath \"\"" Jul 29 19:23:13.944044 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:23:13.944011 2560 generic.go:356] "Generic (PLEG): container finished" podID="6be50932-bc16-449f-8459-218aeafac3ef" containerID="b5108783e77c6b48749a1ff019920b36ba2a2eed83fca76eed2c66a62c1776dd" exitCode=0 Jul 29 19:23:13.944518 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:23:13.944057 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-d04442/sub-test-model-d04442-kserve-6888657659-xjn9k" Jul 29 19:23:13.944518 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:23:13.944095 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-d04442/sub-test-model-d04442-kserve-6888657659-xjn9k" event={"ID":"6be50932-bc16-449f-8459-218aeafac3ef","Type":"ContainerDied","Data":"b5108783e77c6b48749a1ff019920b36ba2a2eed83fca76eed2c66a62c1776dd"} Jul 29 19:23:13.944518 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:23:13.944138 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-d04442/sub-test-model-d04442-kserve-6888657659-xjn9k" event={"ID":"6be50932-bc16-449f-8459-218aeafac3ef","Type":"ContainerDied","Data":"f1ea26186cb030ba54ac9b4021ffbbf6b8fdafa4f7804e3723a96b23ae1d3df1"} Jul 29 19:23:13.944518 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:23:13.944157 2560 scope.go:117] "RemoveContainer" containerID="b5108783e77c6b48749a1ff019920b36ba2a2eed83fca76eed2c66a62c1776dd" Jul 29 19:23:13.953338 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:23:13.953320 2560 scope.go:117] "RemoveContainer" containerID="b5108783e77c6b48749a1ff019920b36ba2a2eed83fca76eed2c66a62c1776dd" Jul 29 19:23:13.953629 ip-10-0-140-205 kubenswrapper[2560]: E0729 19:23:13.953610 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5108783e77c6b48749a1ff019920b36ba2a2eed83fca76eed2c66a62c1776dd\": container with ID starting with b5108783e77c6b48749a1ff019920b36ba2a2eed83fca76eed2c66a62c1776dd not found: ID does not exist" containerID="b5108783e77c6b48749a1ff019920b36ba2a2eed83fca76eed2c66a62c1776dd" Jul 29 19:23:13.953675 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:23:13.953636 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5108783e77c6b48749a1ff019920b36ba2a2eed83fca76eed2c66a62c1776dd"} err="failed to get container status \"b5108783e77c6b48749a1ff019920b36ba2a2eed83fca76eed2c66a62c1776dd\": rpc error: code = NotFound desc = could not find container \"b5108783e77c6b48749a1ff019920b36ba2a2eed83fca76eed2c66a62c1776dd\": container with ID starting with b5108783e77c6b48749a1ff019920b36ba2a2eed83fca76eed2c66a62c1776dd not found: ID does not exist" Jul 29 19:23:13.960808 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:23:13.960789 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-d04442/sub-test-model-d04442-kserve-6888657659-xjn9k"] Jul 29 19:23:13.964550 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:23:13.964531 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-d04442/sub-test-model-d04442-kserve-6888657659-xjn9k"] Jul 29 19:23:13.990044 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:23:13.990017 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-422d30/sub-test-model-422d30-kserve-5c49746b97-kn695" podUID="65477033-c1cc-4dd1-9fc0-36e9d5dd0f11" containerName="main" containerID="cri-o://19f8381ba99c4dbd43c626322bab95c71ed02781b9e60d1d362d625a7df70f61" gracePeriod=45 Jul 29 19:23:14.234416 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:23:14.234394 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-422d30/sub-test-model-422d30-kserve-5c49746b97-kn695" Jul 29 19:23:14.342204 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:23:14.342162 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/65477033-c1cc-4dd1-9fc0-36e9d5dd0f11-tmp-dir\") pod \"65477033-c1cc-4dd1-9fc0-36e9d5dd0f11\" (UID: \"65477033-c1cc-4dd1-9fc0-36e9d5dd0f11\") " Jul 29 19:23:14.342396 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:23:14.342230 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/65477033-c1cc-4dd1-9fc0-36e9d5dd0f11-tls-certs\") pod \"65477033-c1cc-4dd1-9fc0-36e9d5dd0f11\" (UID: \"65477033-c1cc-4dd1-9fc0-36e9d5dd0f11\") " Jul 29 19:23:14.342396 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:23:14.342290 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/65477033-c1cc-4dd1-9fc0-36e9d5dd0f11-model-cache\") pod \"65477033-c1cc-4dd1-9fc0-36e9d5dd0f11\" (UID: \"65477033-c1cc-4dd1-9fc0-36e9d5dd0f11\") " Jul 29 19:23:14.342396 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:23:14.342324 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/65477033-c1cc-4dd1-9fc0-36e9d5dd0f11-home\") pod \"65477033-c1cc-4dd1-9fc0-36e9d5dd0f11\" (UID: \"65477033-c1cc-4dd1-9fc0-36e9d5dd0f11\") " Jul 29 19:23:14.342396 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:23:14.342365 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mnzl5\" (UniqueName: \"kubernetes.io/projected/65477033-c1cc-4dd1-9fc0-36e9d5dd0f11-kube-api-access-mnzl5\") pod \"65477033-c1cc-4dd1-9fc0-36e9d5dd0f11\" (UID: \"65477033-c1cc-4dd1-9fc0-36e9d5dd0f11\") " Jul 29 19:23:14.342588 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:23:14.342401 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/65477033-c1cc-4dd1-9fc0-36e9d5dd0f11-dshm\") pod \"65477033-c1cc-4dd1-9fc0-36e9d5dd0f11\" (UID: \"65477033-c1cc-4dd1-9fc0-36e9d5dd0f11\") " Jul 29 19:23:14.342588 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:23:14.342449 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65477033-c1cc-4dd1-9fc0-36e9d5dd0f11-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "65477033-c1cc-4dd1-9fc0-36e9d5dd0f11" (UID: "65477033-c1cc-4dd1-9fc0-36e9d5dd0f11"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 19:23:14.342588 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:23:14.342556 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65477033-c1cc-4dd1-9fc0-36e9d5dd0f11-model-cache" (OuterVolumeSpecName: "model-cache") pod "65477033-c1cc-4dd1-9fc0-36e9d5dd0f11" (UID: "65477033-c1cc-4dd1-9fc0-36e9d5dd0f11"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 19:23:14.342757 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:23:14.342603 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65477033-c1cc-4dd1-9fc0-36e9d5dd0f11-home" (OuterVolumeSpecName: "home") pod "65477033-c1cc-4dd1-9fc0-36e9d5dd0f11" (UID: "65477033-c1cc-4dd1-9fc0-36e9d5dd0f11"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 19:23:14.342798 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:23:14.342768 2560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/65477033-c1cc-4dd1-9fc0-36e9d5dd0f11-tmp-dir\") on node \"ip-10-0-140-205.ec2.internal\" DevicePath \"\"" Jul 29 19:23:14.342838 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:23:14.342795 2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/65477033-c1cc-4dd1-9fc0-36e9d5dd0f11-model-cache\") on node \"ip-10-0-140-205.ec2.internal\" DevicePath \"\"" Jul 29 19:23:14.342838 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:23:14.342807 2560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/65477033-c1cc-4dd1-9fc0-36e9d5dd0f11-home\") on node \"ip-10-0-140-205.ec2.internal\" DevicePath \"\"" Jul 29 19:23:14.344658 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:23:14.344635 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65477033-c1cc-4dd1-9fc0-36e9d5dd0f11-dshm" (OuterVolumeSpecName: "dshm") pod "65477033-c1cc-4dd1-9fc0-36e9d5dd0f11" (UID: "65477033-c1cc-4dd1-9fc0-36e9d5dd0f11"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 19:23:14.344758 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:23:14.344652 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65477033-c1cc-4dd1-9fc0-36e9d5dd0f11-kube-api-access-mnzl5" (OuterVolumeSpecName: "kube-api-access-mnzl5") pod "65477033-c1cc-4dd1-9fc0-36e9d5dd0f11" (UID: "65477033-c1cc-4dd1-9fc0-36e9d5dd0f11"). InnerVolumeSpecName "kube-api-access-mnzl5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 19:23:14.344758 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:23:14.344677 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/65477033-c1cc-4dd1-9fc0-36e9d5dd0f11-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "65477033-c1cc-4dd1-9fc0-36e9d5dd0f11" (UID: "65477033-c1cc-4dd1-9fc0-36e9d5dd0f11"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 19:23:14.443959 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:23:14.443923 2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/65477033-c1cc-4dd1-9fc0-36e9d5dd0f11-tls-certs\") on node \"ip-10-0-140-205.ec2.internal\" DevicePath \"\"" Jul 29 19:23:14.443959 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:23:14.443952 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mnzl5\" (UniqueName: \"kubernetes.io/projected/65477033-c1cc-4dd1-9fc0-36e9d5dd0f11-kube-api-access-mnzl5\") on node \"ip-10-0-140-205.ec2.internal\" DevicePath \"\"" Jul 29 19:23:14.443959 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:23:14.443961 2560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/65477033-c1cc-4dd1-9fc0-36e9d5dd0f11-dshm\") on node \"ip-10-0-140-205.ec2.internal\" DevicePath \"\"" Jul 29 19:23:14.951962 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:23:14.951928 2560 generic.go:356] "Generic (PLEG): container finished" podID="65477033-c1cc-4dd1-9fc0-36e9d5dd0f11" containerID="19f8381ba99c4dbd43c626322bab95c71ed02781b9e60d1d362d625a7df70f61" exitCode=0 Jul 29 19:23:14.952430 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:23:14.951969 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-422d30/sub-test-model-422d30-kserve-5c49746b97-kn695" event={"ID":"65477033-c1cc-4dd1-9fc0-36e9d5dd0f11","Type":"ContainerDied","Data":"19f8381ba99c4dbd43c626322bab95c71ed02781b9e60d1d362d625a7df70f61"} Jul 29 19:23:14.952430 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:23:14.952003 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-422d30/sub-test-model-422d30-kserve-5c49746b97-kn695" event={"ID":"65477033-c1cc-4dd1-9fc0-36e9d5dd0f11","Type":"ContainerDied","Data":"cecf467988b5a69c635ffda3da87ddeec2f4864c2123587cb758394a0846adbf"} Jul 29 19:23:14.952430 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:23:14.952025 2560 scope.go:117] "RemoveContainer" containerID="19f8381ba99c4dbd43c626322bab95c71ed02781b9e60d1d362d625a7df70f61" Jul 29 19:23:14.952430 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:23:14.951973 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-422d30/sub-test-model-422d30-kserve-5c49746b97-kn695" Jul 29 19:23:14.961783 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:23:14.961765 2560 scope.go:117] "RemoveContainer" containerID="19f8381ba99c4dbd43c626322bab95c71ed02781b9e60d1d362d625a7df70f61" Jul 29 19:23:14.962048 ip-10-0-140-205 kubenswrapper[2560]: E0729 19:23:14.962028 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"19f8381ba99c4dbd43c626322bab95c71ed02781b9e60d1d362d625a7df70f61\": container with ID starting with 19f8381ba99c4dbd43c626322bab95c71ed02781b9e60d1d362d625a7df70f61 not found: ID does not exist" containerID="19f8381ba99c4dbd43c626322bab95c71ed02781b9e60d1d362d625a7df70f61" Jul 29 19:23:14.962099 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:23:14.962056 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19f8381ba99c4dbd43c626322bab95c71ed02781b9e60d1d362d625a7df70f61"} err="failed to get container status \"19f8381ba99c4dbd43c626322bab95c71ed02781b9e60d1d362d625a7df70f61\": rpc error: code = NotFound desc = could not find container \"19f8381ba99c4dbd43c626322bab95c71ed02781b9e60d1d362d625a7df70f61\": container with ID starting with 19f8381ba99c4dbd43c626322bab95c71ed02781b9e60d1d362d625a7df70f61 not found: ID does not exist" Jul 29 19:23:14.973172 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:23:14.973149 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-422d30/sub-test-model-422d30-kserve-5c49746b97-kn695"] Jul 29 19:23:14.982190 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:23:14.977773 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-422d30/sub-test-model-422d30-kserve-5c49746b97-kn695"] Jul 29 19:23:15.354384 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:23:15.354334 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="65477033-c1cc-4dd1-9fc0-36e9d5dd0f11" path="/var/lib/kubelet/pods/65477033-c1cc-4dd1-9fc0-36e9d5dd0f11/volumes" Jul 29 19:23:15.354738 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:23:15.354724 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6be50932-bc16-449f-8459-218aeafac3ef" path="/var/lib/kubelet/pods/6be50932-bc16-449f-8459-218aeafac3ef/volumes" Jul 29 19:23:34.268639 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:23:34.268596 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-422d30/rate-test-model-422d30-kserve-85ff8cfd79-fk86c"] Jul 29 19:23:34.269186 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:23:34.269166 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6be50932-bc16-449f-8459-218aeafac3ef" containerName="main" Jul 29 19:23:34.269257 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:23:34.269189 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="6be50932-bc16-449f-8459-218aeafac3ef" containerName="main" Jul 29 19:23:34.269257 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:23:34.269226 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="65477033-c1cc-4dd1-9fc0-36e9d5dd0f11" containerName="main" Jul 29 19:23:34.269257 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:23:34.269235 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="65477033-c1cc-4dd1-9fc0-36e9d5dd0f11" containerName="main" Jul 29 19:23:34.269456 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:23:34.269319 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="6be50932-bc16-449f-8459-218aeafac3ef" containerName="main" Jul 29 19:23:34.269456 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:23:34.269336 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="65477033-c1cc-4dd1-9fc0-36e9d5dd0f11" containerName="main" Jul 29 19:23:34.272865 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:23:34.272840 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-422d30/rate-test-model-422d30-kserve-85ff8cfd79-fk86c" Jul 29 19:23:34.275500 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:23:34.275478 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-422d30\"/\"openshift-service-ca.crt\"" Jul 29 19:23:34.275601 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:23:34.275509 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-422d30\"/\"kube-root-ca.crt\"" Jul 29 19:23:34.275658 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:23:34.275621 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-422d30\"/\"rate-test-model-422d30-kserve-self-signed-certs\"" Jul 29 19:23:34.275705 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:23:34.275619 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-422d30\"/\"default-dockercfg-8b8fl\"" Jul 29 19:23:34.280718 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:23:34.280695 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-422d30/rate-test-model-422d30-kserve-85ff8cfd79-fk86c"] Jul 29 19:23:34.301330 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:23:34.301313 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:23:34.305316 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:23:34.305299 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/497efccf-9ff9-47e7-8bc9-c51a085b9dc0-dshm\") pod \"rate-test-model-422d30-kserve-85ff8cfd79-fk86c\" (UID: \"497efccf-9ff9-47e7-8bc9-c51a085b9dc0\") " pod="ai-tenant-e2e-shared-b-422d30/rate-test-model-422d30-kserve-85ff8cfd79-fk86c" Jul 29 19:23:34.305393 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:23:34.305340 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26qrr\" (UniqueName: \"kubernetes.io/projected/497efccf-9ff9-47e7-8bc9-c51a085b9dc0-kube-api-access-26qrr\") pod \"rate-test-model-422d30-kserve-85ff8cfd79-fk86c\" (UID: \"497efccf-9ff9-47e7-8bc9-c51a085b9dc0\") " pod="ai-tenant-e2e-shared-b-422d30/rate-test-model-422d30-kserve-85ff8cfd79-fk86c" Jul 29 19:23:34.305434 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:23:34.305413 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/497efccf-9ff9-47e7-8bc9-c51a085b9dc0-tls-certs\") pod \"rate-test-model-422d30-kserve-85ff8cfd79-fk86c\" (UID: \"497efccf-9ff9-47e7-8bc9-c51a085b9dc0\") " pod="ai-tenant-e2e-shared-b-422d30/rate-test-model-422d30-kserve-85ff8cfd79-fk86c" Jul 29 19:23:34.305470 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:23:34.305438 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/497efccf-9ff9-47e7-8bc9-c51a085b9dc0-model-cache\") pod \"rate-test-model-422d30-kserve-85ff8cfd79-fk86c\" (UID: \"497efccf-9ff9-47e7-8bc9-c51a085b9dc0\") " pod="ai-tenant-e2e-shared-b-422d30/rate-test-model-422d30-kserve-85ff8cfd79-fk86c" Jul 29 19:23:34.305508 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:23:34.305477 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/497efccf-9ff9-47e7-8bc9-c51a085b9dc0-tmp-dir\") pod \"rate-test-model-422d30-kserve-85ff8cfd79-fk86c\" (UID: \"497efccf-9ff9-47e7-8bc9-c51a085b9dc0\") " pod="ai-tenant-e2e-shared-b-422d30/rate-test-model-422d30-kserve-85ff8cfd79-fk86c" Jul 29 19:23:34.305542 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:23:34.305530 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/497efccf-9ff9-47e7-8bc9-c51a085b9dc0-home\") pod \"rate-test-model-422d30-kserve-85ff8cfd79-fk86c\" (UID: \"497efccf-9ff9-47e7-8bc9-c51a085b9dc0\") " pod="ai-tenant-e2e-shared-b-422d30/rate-test-model-422d30-kserve-85ff8cfd79-fk86c" Jul 29 19:23:34.406884 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:23:34.406856 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/497efccf-9ff9-47e7-8bc9-c51a085b9dc0-tmp-dir\") pod \"rate-test-model-422d30-kserve-85ff8cfd79-fk86c\" (UID: \"497efccf-9ff9-47e7-8bc9-c51a085b9dc0\") " pod="ai-tenant-e2e-shared-b-422d30/rate-test-model-422d30-kserve-85ff8cfd79-fk86c" Jul 29 19:23:34.407012 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:23:34.406907 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/497efccf-9ff9-47e7-8bc9-c51a085b9dc0-home\") pod \"rate-test-model-422d30-kserve-85ff8cfd79-fk86c\" (UID: \"497efccf-9ff9-47e7-8bc9-c51a085b9dc0\") " pod="ai-tenant-e2e-shared-b-422d30/rate-test-model-422d30-kserve-85ff8cfd79-fk86c" Jul 29 19:23:34.407012 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:23:34.406948 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/497efccf-9ff9-47e7-8bc9-c51a085b9dc0-dshm\") pod \"rate-test-model-422d30-kserve-85ff8cfd79-fk86c\" (UID: \"497efccf-9ff9-47e7-8bc9-c51a085b9dc0\") " pod="ai-tenant-e2e-shared-b-422d30/rate-test-model-422d30-kserve-85ff8cfd79-fk86c" Jul 29 19:23:34.407012 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:23:34.406980 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26qrr\" (UniqueName: \"kubernetes.io/projected/497efccf-9ff9-47e7-8bc9-c51a085b9dc0-kube-api-access-26qrr\") pod \"rate-test-model-422d30-kserve-85ff8cfd79-fk86c\" (UID: \"497efccf-9ff9-47e7-8bc9-c51a085b9dc0\") " pod="ai-tenant-e2e-shared-b-422d30/rate-test-model-422d30-kserve-85ff8cfd79-fk86c" Jul 29 19:23:34.407012 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:23:34.406997 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/497efccf-9ff9-47e7-8bc9-c51a085b9dc0-tls-certs\") pod \"rate-test-model-422d30-kserve-85ff8cfd79-fk86c\" (UID: \"497efccf-9ff9-47e7-8bc9-c51a085b9dc0\") " pod="ai-tenant-e2e-shared-b-422d30/rate-test-model-422d30-kserve-85ff8cfd79-fk86c" Jul 29 19:23:34.407218 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:23:34.407135 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/497efccf-9ff9-47e7-8bc9-c51a085b9dc0-model-cache\") pod \"rate-test-model-422d30-kserve-85ff8cfd79-fk86c\" (UID: \"497efccf-9ff9-47e7-8bc9-c51a085b9dc0\") " pod="ai-tenant-e2e-shared-b-422d30/rate-test-model-422d30-kserve-85ff8cfd79-fk86c" Jul 29 19:23:34.407299 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:23:34.407278 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/497efccf-9ff9-47e7-8bc9-c51a085b9dc0-tmp-dir\") pod \"rate-test-model-422d30-kserve-85ff8cfd79-fk86c\" (UID: \"497efccf-9ff9-47e7-8bc9-c51a085b9dc0\") " pod="ai-tenant-e2e-shared-b-422d30/rate-test-model-422d30-kserve-85ff8cfd79-fk86c" Jul 29 19:23:34.407404 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:23:34.407330 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/497efccf-9ff9-47e7-8bc9-c51a085b9dc0-home\") pod \"rate-test-model-422d30-kserve-85ff8cfd79-fk86c\" (UID: \"497efccf-9ff9-47e7-8bc9-c51a085b9dc0\") " pod="ai-tenant-e2e-shared-b-422d30/rate-test-model-422d30-kserve-85ff8cfd79-fk86c" Jul 29 19:23:34.407523 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:23:34.407490 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/497efccf-9ff9-47e7-8bc9-c51a085b9dc0-model-cache\") pod \"rate-test-model-422d30-kserve-85ff8cfd79-fk86c\" (UID: \"497efccf-9ff9-47e7-8bc9-c51a085b9dc0\") " pod="ai-tenant-e2e-shared-b-422d30/rate-test-model-422d30-kserve-85ff8cfd79-fk86c" Jul 29 19:23:34.409318 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:23:34.409301 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/497efccf-9ff9-47e7-8bc9-c51a085b9dc0-dshm\") pod \"rate-test-model-422d30-kserve-85ff8cfd79-fk86c\" (UID: \"497efccf-9ff9-47e7-8bc9-c51a085b9dc0\") " pod="ai-tenant-e2e-shared-b-422d30/rate-test-model-422d30-kserve-85ff8cfd79-fk86c" Jul 29 19:23:34.409764 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:23:34.409743 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/497efccf-9ff9-47e7-8bc9-c51a085b9dc0-tls-certs\") pod \"rate-test-model-422d30-kserve-85ff8cfd79-fk86c\" (UID: \"497efccf-9ff9-47e7-8bc9-c51a085b9dc0\") " pod="ai-tenant-e2e-shared-b-422d30/rate-test-model-422d30-kserve-85ff8cfd79-fk86c" Jul 29 19:23:34.416405 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:23:34.416383 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-26qrr\" (UniqueName: \"kubernetes.io/projected/497efccf-9ff9-47e7-8bc9-c51a085b9dc0-kube-api-access-26qrr\") pod \"rate-test-model-422d30-kserve-85ff8cfd79-fk86c\" (UID: \"497efccf-9ff9-47e7-8bc9-c51a085b9dc0\") " pod="ai-tenant-e2e-shared-b-422d30/rate-test-model-422d30-kserve-85ff8cfd79-fk86c" Jul 29 19:23:34.582587 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:23:34.582513 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-422d30/rate-test-model-422d30-kserve-85ff8cfd79-fk86c" Jul 29 19:23:34.718222 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:23:34.718197 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-422d30/rate-test-model-422d30-kserve-85ff8cfd79-fk86c"] Jul 29 19:23:34.720922 ip-10-0-140-205 kubenswrapper[2560]: W0729 19:23:34.720894 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod497efccf_9ff9_47e7_8bc9_c51a085b9dc0.slice/crio-bf9b37a9cd59dc346abc197f116c613ca2e44db60b5c2d42c8d1b160900521e9 WatchSource:0}: Error finding container bf9b37a9cd59dc346abc197f116c613ca2e44db60b5c2d42c8d1b160900521e9: Status 404 returned error can't find the container with id bf9b37a9cd59dc346abc197f116c613ca2e44db60b5c2d42c8d1b160900521e9 Jul 29 19:23:35.059115 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:23:35.059083 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-422d30/rate-test-model-422d30-kserve-85ff8cfd79-fk86c" event={"ID":"497efccf-9ff9-47e7-8bc9-c51a085b9dc0","Type":"ContainerStarted","Data":"4e2e342b93b6e6b968ebbe388b1f714558e0c16f2c9835c4a03580d6b255adbd"} Jul 29 19:23:35.059268 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:23:35.059122 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-422d30/rate-test-model-422d30-kserve-85ff8cfd79-fk86c" event={"ID":"497efccf-9ff9-47e7-8bc9-c51a085b9dc0","Type":"ContainerStarted","Data":"bf9b37a9cd59dc346abc197f116c613ca2e44db60b5c2d42c8d1b160900521e9"} Jul 29 19:23:35.076536 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:23:35.076491 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-422d30/rate-test-model-422d30-kserve-85ff8cfd79-fk86c" podStartSLOduration=1.076476185 podStartE2EDuration="1.076476185s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:23:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:23:35.07564147 +0000 UTC m=+2950.301310353" watchObservedRunningTime="2026-07-29 19:23:35.076476185 +0000 UTC m=+2950.302145056" Jul 29 19:23:44.583042 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:23:44.583010 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-422d30/rate-test-model-422d30-kserve-85ff8cfd79-fk86c" Jul 29 19:23:44.583651 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:23:44.583060 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-422d30/rate-test-model-422d30-kserve-85ff8cfd79-fk86c" Jul 29 19:23:44.596632 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:23:44.596604 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-422d30/rate-test-model-422d30-kserve-85ff8cfd79-fk86c" Jul 29 19:23:45.127949 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:23:45.127925 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-422d30/rate-test-model-422d30-kserve-85ff8cfd79-fk86c" Jul 29 19:23:46.197675 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:23:46.197636 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:23:59.476616 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:23:59.476576 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:24:25.660815 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:24:25.660697 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-v4n5x_1727fb40-fbed-4984-91d4-a5890d0f4103/ovn-acl-logging/0.log" Jul 29 19:24:25.671414 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:24:25.671391 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-v4n5x_1727fb40-fbed-4984-91d4-a5890d0f4103/ovn-acl-logging/0.log" Jul 29 19:24:40.805863 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:24:40.805833 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:24:41.667365 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:24:41.667326 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-422d30/rate-test-model-422d30-kserve-85ff8cfd79-fk86c"] Jul 29 19:24:41.687943 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:24:41.687917 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:24:42.727685 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:24:42.727652 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7504-139814632604592"] Jul 29 19:24:42.731249 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:24:42.731231 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7504-139814632604592" Jul 29 19:24:42.733761 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:24:42.733732 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-lj2xq\"" Jul 29 19:24:42.736889 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:24:42.736864 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7504-139814632604592"] Jul 29 19:24:42.824997 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:24:42.824966 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9s8g\" (UniqueName: \"kubernetes.io/projected/c0368d83-03c7-40de-845b-c177ae7a6445-kube-api-access-r9s8g\") pod \"test-curl-7504-139814632604592\" (UID: \"c0368d83-03c7-40de-845b-c177ae7a6445\") " pod="opendatahub/test-curl-7504-139814632604592" Jul 29 19:24:42.925678 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:24:42.925645 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9s8g\" (UniqueName: \"kubernetes.io/projected/c0368d83-03c7-40de-845b-c177ae7a6445-kube-api-access-r9s8g\") pod \"test-curl-7504-139814632604592\" (UID: \"c0368d83-03c7-40de-845b-c177ae7a6445\") " pod="opendatahub/test-curl-7504-139814632604592" Jul 29 19:24:42.934514 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:24:42.934487 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9s8g\" (UniqueName: \"kubernetes.io/projected/c0368d83-03c7-40de-845b-c177ae7a6445-kube-api-access-r9s8g\") pod \"test-curl-7504-139814632604592\" (UID: \"c0368d83-03c7-40de-845b-c177ae7a6445\") " pod="opendatahub/test-curl-7504-139814632604592" Jul 29 19:24:43.041826 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:24:43.041787 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7504-139814632604592" Jul 29 19:24:43.175186 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:24:43.175161 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7504-139814632604592"] Jul 29 19:24:43.177336 ip-10-0-140-205 kubenswrapper[2560]: W0729 19:24:43.177302 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc0368d83_03c7_40de_845b_c177ae7a6445.slice/crio-8ede02898f18bb3d3a510deae4802e8a8707bafc2973905999d92b76ba381cdd WatchSource:0}: Error finding container 8ede02898f18bb3d3a510deae4802e8a8707bafc2973905999d92b76ba381cdd: Status 404 returned error can't find the container with id 8ede02898f18bb3d3a510deae4802e8a8707bafc2973905999d92b76ba381cdd Jul 29 19:24:43.179084 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:24:43.179061 2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 29 19:24:43.427877 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:24:43.427806 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7504-139814632604592" event={"ID":"c0368d83-03c7-40de-845b-c177ae7a6445","Type":"ContainerStarted","Data":"8ede02898f18bb3d3a510deae4802e8a8707bafc2973905999d92b76ba381cdd"} Jul 29 19:24:44.434228 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:24:44.434192 2560 generic.go:356] "Generic (PLEG): container finished" podID="c0368d83-03c7-40de-845b-c177ae7a6445" containerID="9ceea179dbb29fccdbdccaba029664a74c8708a079fbc4b1c2db22898e083999" exitCode=0 Jul 29 19:24:44.434677 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:24:44.434251 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7504-139814632604592" event={"ID":"c0368d83-03c7-40de-845b-c177ae7a6445","Type":"ContainerDied","Data":"9ceea179dbb29fccdbdccaba029664a74c8708a079fbc4b1c2db22898e083999"} Jul 29 19:24:45.570649 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:24:45.570626 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7504-139814632604592" Jul 29 19:24:45.590758 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:24:45.590723 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7504-139814632604592_c0368d83-03c7-40de-845b-c177ae7a6445/test-curl-7504-139814632604592/0.log" Jul 29 19:24:45.615619 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:24:45.615596 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7504-139814632604592"] Jul 29 19:24:45.623360 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:24:45.623326 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7504-139814632604592"] Jul 29 19:24:45.653742 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:24:45.653717 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r9s8g\" (UniqueName: \"kubernetes.io/projected/c0368d83-03c7-40de-845b-c177ae7a6445-kube-api-access-r9s8g\") pod \"c0368d83-03c7-40de-845b-c177ae7a6445\" (UID: \"c0368d83-03c7-40de-845b-c177ae7a6445\") " Jul 29 19:24:45.655968 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:24:45.655940 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0368d83-03c7-40de-845b-c177ae7a6445-kube-api-access-r9s8g" (OuterVolumeSpecName: "kube-api-access-r9s8g") pod "c0368d83-03c7-40de-845b-c177ae7a6445" (UID: "c0368d83-03c7-40de-845b-c177ae7a6445"). InnerVolumeSpecName "kube-api-access-r9s8g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 19:24:45.754950 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:24:45.754916 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r9s8g\" (UniqueName: \"kubernetes.io/projected/c0368d83-03c7-40de-845b-c177ae7a6445-kube-api-access-r9s8g\") on node \"ip-10-0-140-205.ec2.internal\" DevicePath \"\"" Jul 29 19:24:45.758777 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:24:45.758749 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7504-139814632735408"] Jul 29 19:24:45.759179 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:24:45.759161 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c0368d83-03c7-40de-845b-c177ae7a6445" containerName="test-curl-7504-139814632604592" Jul 29 19:24:45.759259 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:24:45.759181 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="c0368d83-03c7-40de-845b-c177ae7a6445" containerName="test-curl-7504-139814632604592" Jul 29 19:24:45.759314 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:24:45.759279 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="c0368d83-03c7-40de-845b-c177ae7a6445" containerName="test-curl-7504-139814632604592" Jul 29 19:24:45.762316 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:24:45.762299 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7504-139814632735408" Jul 29 19:24:45.768819 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:24:45.768794 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7504-139814632735408"] Jul 29 19:24:45.855567 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:24:45.855531 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7cx94\" (UniqueName: \"kubernetes.io/projected/dd77704a-94c4-457b-8181-ad7cd5f50d1c-kube-api-access-7cx94\") pod \"test-curl-7504-139814632735408\" (UID: \"dd77704a-94c4-457b-8181-ad7cd5f50d1c\") " pod="opendatahub/test-curl-7504-139814632735408" Jul 29 19:24:45.956690 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:24:45.956650 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7cx94\" (UniqueName: \"kubernetes.io/projected/dd77704a-94c4-457b-8181-ad7cd5f50d1c-kube-api-access-7cx94\") pod \"test-curl-7504-139814632735408\" (UID: \"dd77704a-94c4-457b-8181-ad7cd5f50d1c\") " pod="opendatahub/test-curl-7504-139814632735408" Jul 29 19:24:45.966733 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:24:45.966712 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7cx94\" (UniqueName: \"kubernetes.io/projected/dd77704a-94c4-457b-8181-ad7cd5f50d1c-kube-api-access-7cx94\") pod \"test-curl-7504-139814632735408\" (UID: \"dd77704a-94c4-457b-8181-ad7cd5f50d1c\") " pod="opendatahub/test-curl-7504-139814632735408" Jul 29 19:24:46.072985 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:24:46.072905 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7504-139814632735408" Jul 29 19:24:46.208054 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:24:46.208026 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7504-139814632735408"] Jul 29 19:24:46.209615 ip-10-0-140-205 kubenswrapper[2560]: W0729 19:24:46.209589 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddd77704a_94c4_457b_8181_ad7cd5f50d1c.slice/crio-e84c3abe1a783192846518f0dd421a2f9f9f27bf8d3bc134eb9af8e4608d18fd WatchSource:0}: Error finding container e84c3abe1a783192846518f0dd421a2f9f9f27bf8d3bc134eb9af8e4608d18fd: Status 404 returned error can't find the container with id e84c3abe1a783192846518f0dd421a2f9f9f27bf8d3bc134eb9af8e4608d18fd Jul 29 19:24:46.445777 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:24:46.445704 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7504-139814632604592" Jul 29 19:24:46.445777 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:24:46.445718 2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8ede02898f18bb3d3a510deae4802e8a8707bafc2973905999d92b76ba381cdd" Jul 29 19:24:46.447124 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:24:46.447095 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7504-139814632735408" event={"ID":"dd77704a-94c4-457b-8181-ad7cd5f50d1c","Type":"ContainerStarted","Data":"e84c3abe1a783192846518f0dd421a2f9f9f27bf8d3bc134eb9af8e4608d18fd"} Jul 29 19:24:46.456088 ip-10-0-140-205 kubenswrapper[2560]: E0729 19:24:46.456067 2560 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7504-139814632604592\" is forbidden: User \"system:node:ip-10-0-140-205.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-205.ec2.internal' and this object" podUID="c0368d83-03c7-40de-845b-c177ae7a6445" pod="opendatahub/test-curl-7504-139814632604592" Jul 29 19:24:47.354599 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:24:47.354565 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c0368d83-03c7-40de-845b-c177ae7a6445" path="/var/lib/kubelet/pods/c0368d83-03c7-40de-845b-c177ae7a6445/volumes" Jul 29 19:24:47.453496 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:24:47.453463 2560 generic.go:356] "Generic (PLEG): container finished" podID="dd77704a-94c4-457b-8181-ad7cd5f50d1c" containerID="99b172c5bc5ff627038a82d4afd76eeab751bc0b66cf7662ac352c2bcc3fddbe" exitCode=0 Jul 29 19:24:47.453645 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:24:47.453550 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7504-139814632735408" event={"ID":"dd77704a-94c4-457b-8181-ad7cd5f50d1c","Type":"ContainerDied","Data":"99b172c5bc5ff627038a82d4afd76eeab751bc0b66cf7662ac352c2bcc3fddbe"} Jul 29 19:24:48.595000 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:24:48.594972 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7504-139814632735408" Jul 29 19:24:48.614541 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:24:48.614513 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7504-139814632735408_dd77704a-94c4-457b-8181-ad7cd5f50d1c/test-curl-7504-139814632735408/0.log" Jul 29 19:24:48.628180 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:24:48.628155 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:24:48.637827 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:24:48.637803 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7504-139814632735408"] Jul 29 19:24:48.641406 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:24:48.641388 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7504-139814632735408"] Jul 29 19:24:48.688590 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:24:48.688566 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7cx94\" (UniqueName: \"kubernetes.io/projected/dd77704a-94c4-457b-8181-ad7cd5f50d1c-kube-api-access-7cx94\") pod \"dd77704a-94c4-457b-8181-ad7cd5f50d1c\" (UID: \"dd77704a-94c4-457b-8181-ad7cd5f50d1c\") " Jul 29 19:24:48.690800 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:24:48.690769 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd77704a-94c4-457b-8181-ad7cd5f50d1c-kube-api-access-7cx94" (OuterVolumeSpecName: "kube-api-access-7cx94") pod "dd77704a-94c4-457b-8181-ad7cd5f50d1c" (UID: "dd77704a-94c4-457b-8181-ad7cd5f50d1c"). InnerVolumeSpecName "kube-api-access-7cx94". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 19:24:48.790075 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:24:48.790048 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7cx94\" (UniqueName: \"kubernetes.io/projected/dd77704a-94c4-457b-8181-ad7cd5f50d1c-kube-api-access-7cx94\") on node \"ip-10-0-140-205.ec2.internal\" DevicePath \"\"" Jul 29 19:24:48.812788 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:24:48.812762 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7504-139814632716304"] Jul 29 19:24:48.813147 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:24:48.813134 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dd77704a-94c4-457b-8181-ad7cd5f50d1c" containerName="test-curl-7504-139814632735408" Jul 29 19:24:48.813189 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:24:48.813148 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="dd77704a-94c4-457b-8181-ad7cd5f50d1c" containerName="test-curl-7504-139814632735408" Jul 29 19:24:48.813234 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:24:48.813226 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="dd77704a-94c4-457b-8181-ad7cd5f50d1c" containerName="test-curl-7504-139814632735408" Jul 29 19:24:48.816204 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:24:48.816187 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7504-139814632716304" Jul 29 19:24:48.823172 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:24:48.823152 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7504-139814632716304"] Jul 29 19:24:48.992305 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:24:48.992267 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4wck\" (UniqueName: \"kubernetes.io/projected/8e9843a8-1b1b-4479-8e57-eef9d700bbd0-kube-api-access-x4wck\") pod \"test-curl-7504-139814632716304\" (UID: \"8e9843a8-1b1b-4479-8e57-eef9d700bbd0\") " pod="opendatahub/test-curl-7504-139814632716304" Jul 29 19:24:49.093140 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:24:49.093063 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4wck\" (UniqueName: \"kubernetes.io/projected/8e9843a8-1b1b-4479-8e57-eef9d700bbd0-kube-api-access-x4wck\") pod \"test-curl-7504-139814632716304\" (UID: \"8e9843a8-1b1b-4479-8e57-eef9d700bbd0\") " pod="opendatahub/test-curl-7504-139814632716304" Jul 29 19:24:49.103606 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:24:49.103575 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4wck\" (UniqueName: \"kubernetes.io/projected/8e9843a8-1b1b-4479-8e57-eef9d700bbd0-kube-api-access-x4wck\") pod \"test-curl-7504-139814632716304\" (UID: \"8e9843a8-1b1b-4479-8e57-eef9d700bbd0\") " pod="opendatahub/test-curl-7504-139814632716304" Jul 29 19:24:49.127737 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:24:49.127706 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7504-139814632716304" Jul 29 19:24:49.260016 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:24:49.259994 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7504-139814632716304"] Jul 29 19:24:49.261729 ip-10-0-140-205 kubenswrapper[2560]: W0729 19:24:49.261695 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8e9843a8_1b1b_4479_8e57_eef9d700bbd0.slice/crio-f843cc26e32e9a6780a0b221a62341a71a7a9a6f7de276d2c1dfcdeba716fea1 WatchSource:0}: Error finding container f843cc26e32e9a6780a0b221a62341a71a7a9a6f7de276d2c1dfcdeba716fea1: Status 404 returned error can't find the container with id f843cc26e32e9a6780a0b221a62341a71a7a9a6f7de276d2c1dfcdeba716fea1 Jul 29 19:24:49.353998 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:24:49.353920 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dd77704a-94c4-457b-8181-ad7cd5f50d1c" path="/var/lib/kubelet/pods/dd77704a-94c4-457b-8181-ad7cd5f50d1c/volumes" Jul 29 19:24:49.466082 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:24:49.466050 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7504-139814632735408" Jul 29 19:24:49.466261 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:24:49.466096 2560 scope.go:117] "RemoveContainer" containerID="99b172c5bc5ff627038a82d4afd76eeab751bc0b66cf7662ac352c2bcc3fddbe" Jul 29 19:24:49.467407 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:24:49.467380 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7504-139814632716304" event={"ID":"8e9843a8-1b1b-4479-8e57-eef9d700bbd0","Type":"ContainerStarted","Data":"f843cc26e32e9a6780a0b221a62341a71a7a9a6f7de276d2c1dfcdeba716fea1"} Jul 29 19:24:50.474789 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:24:50.474751 2560 generic.go:356] "Generic (PLEG): container finished" podID="8e9843a8-1b1b-4479-8e57-eef9d700bbd0" containerID="8d84a74f0f6f643d7c56c632de0d64e918df8b7d3aa89192b95420a99dc995e4" exitCode=0 Jul 29 19:24:50.475216 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:24:50.474807 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7504-139814632716304" event={"ID":"8e9843a8-1b1b-4479-8e57-eef9d700bbd0","Type":"ContainerDied","Data":"8d84a74f0f6f643d7c56c632de0d64e918df8b7d3aa89192b95420a99dc995e4"} Jul 29 19:24:51.618520 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:24:51.618495 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7504-139814632716304" Jul 29 19:24:51.639280 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:24:51.639254 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7504-139814632716304_8e9843a8-1b1b-4479-8e57-eef9d700bbd0/test-curl-7504-139814632716304/0.log" Jul 29 19:24:51.664651 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:24:51.664626 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7504-139814632716304"] Jul 29 19:24:51.669304 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:24:51.669282 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7504-139814632716304"] Jul 29 19:24:51.716081 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:24:51.716062 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x4wck\" (UniqueName: \"kubernetes.io/projected/8e9843a8-1b1b-4479-8e57-eef9d700bbd0-kube-api-access-x4wck\") pod \"8e9843a8-1b1b-4479-8e57-eef9d700bbd0\" (UID: \"8e9843a8-1b1b-4479-8e57-eef9d700bbd0\") " Jul 29 19:24:51.718211 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:24:51.718183 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e9843a8-1b1b-4479-8e57-eef9d700bbd0-kube-api-access-x4wck" (OuterVolumeSpecName: "kube-api-access-x4wck") pod "8e9843a8-1b1b-4479-8e57-eef9d700bbd0" (UID: "8e9843a8-1b1b-4479-8e57-eef9d700bbd0"). InnerVolumeSpecName "kube-api-access-x4wck". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 19:24:51.810412 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:24:51.810378 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7504-139814632716304"] Jul 29 19:24:51.810812 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:24:51.810796 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e9843a8-1b1b-4479-8e57-eef9d700bbd0" containerName="test-curl-7504-139814632716304" Jul 29 19:24:51.810908 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:24:51.810814 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e9843a8-1b1b-4479-8e57-eef9d700bbd0" containerName="test-curl-7504-139814632716304" Jul 29 19:24:51.810908 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:24:51.810888 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="8e9843a8-1b1b-4479-8e57-eef9d700bbd0" containerName="test-curl-7504-139814632716304" Jul 29 19:24:51.813930 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:24:51.813913 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7504-139814632716304" Jul 29 19:24:51.821683 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:24:51.821652 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x4wck\" (UniqueName: \"kubernetes.io/projected/8e9843a8-1b1b-4479-8e57-eef9d700bbd0-kube-api-access-x4wck\") on node \"ip-10-0-140-205.ec2.internal\" DevicePath \"\"" Jul 29 19:24:51.824940 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:24:51.824915 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7504-139814632716304"] Jul 29 19:24:51.922612 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:24:51.922589 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-578tg\" (UniqueName: \"kubernetes.io/projected/11c66d39-e80b-4cb4-9db1-a922304b7f4f-kube-api-access-578tg\") pod \"test-curl-7504-139814632716304\" (UID: \"11c66d39-e80b-4cb4-9db1-a922304b7f4f\") " pod="opendatahub/test-curl-7504-139814632716304" Jul 29 19:24:52.023744 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:24:52.023715 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-578tg\" (UniqueName: \"kubernetes.io/projected/11c66d39-e80b-4cb4-9db1-a922304b7f4f-kube-api-access-578tg\") pod \"test-curl-7504-139814632716304\" (UID: \"11c66d39-e80b-4cb4-9db1-a922304b7f4f\") " pod="opendatahub/test-curl-7504-139814632716304" Jul 29 19:24:52.032439 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:24:52.032412 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-578tg\" (UniqueName: \"kubernetes.io/projected/11c66d39-e80b-4cb4-9db1-a922304b7f4f-kube-api-access-578tg\") pod \"test-curl-7504-139814632716304\" (UID: \"11c66d39-e80b-4cb4-9db1-a922304b7f4f\") " pod="opendatahub/test-curl-7504-139814632716304" Jul 29 19:24:52.131409 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:24:52.131322 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7504-139814632716304" Jul 29 19:24:52.270529 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:24:52.270498 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7504-139814632716304"] Jul 29 19:24:52.272487 ip-10-0-140-205 kubenswrapper[2560]: W0729 19:24:52.272461 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod11c66d39_e80b_4cb4_9db1_a922304b7f4f.slice/crio-29a4f473352cc145e3075a58a0276aa31ec31dc4536c2bc787c67b7ac242f5d2 WatchSource:0}: Error finding container 29a4f473352cc145e3075a58a0276aa31ec31dc4536c2bc787c67b7ac242f5d2: Status 404 returned error can't find the container with id 29a4f473352cc145e3075a58a0276aa31ec31dc4536c2bc787c67b7ac242f5d2 Jul 29 19:24:52.487685 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:24:52.487605 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7504-139814632716304" event={"ID":"11c66d39-e80b-4cb4-9db1-a922304b7f4f","Type":"ContainerStarted","Data":"29a4f473352cc145e3075a58a0276aa31ec31dc4536c2bc787c67b7ac242f5d2"} Jul 29 19:24:52.488781 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:24:52.488762 2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f843cc26e32e9a6780a0b221a62341a71a7a9a6f7de276d2c1dfcdeba716fea1" Jul 29 19:24:52.488836 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:24:52.488796 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7504-139814632716304" Jul 29 19:24:52.500557 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:24:52.500529 2560 status_manager.go:1027] "Pod was deleted and then recreated, skipping status update" pod="opendatahub/test-curl-7504-139814632716304" oldPodUID="8e9843a8-1b1b-4479-8e57-eef9d700bbd0" podUID="11c66d39-e80b-4cb4-9db1-a922304b7f4f" Jul 29 19:24:53.354188 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:24:53.354158 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8e9843a8-1b1b-4479-8e57-eef9d700bbd0" path="/var/lib/kubelet/pods/8e9843a8-1b1b-4479-8e57-eef9d700bbd0/volumes" Jul 29 19:24:53.495547 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:24:53.495513 2560 generic.go:356] "Generic (PLEG): container finished" podID="11c66d39-e80b-4cb4-9db1-a922304b7f4f" containerID="e7e5fd5656f3a9b22a2dae98b3936886e79dc795875477d9aec0db26122f7635" exitCode=0 Jul 29 19:24:53.495694 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:24:53.495602 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7504-139814632716304" event={"ID":"11c66d39-e80b-4cb4-9db1-a922304b7f4f","Type":"ContainerDied","Data":"e7e5fd5656f3a9b22a2dae98b3936886e79dc795875477d9aec0db26122f7635"} Jul 29 19:24:54.637110 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:24:54.637088 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7504-139814632716304" Jul 29 19:24:54.646384 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:24:54.646340 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-578tg\" (UniqueName: \"kubernetes.io/projected/11c66d39-e80b-4cb4-9db1-a922304b7f4f-kube-api-access-578tg\") pod \"11c66d39-e80b-4cb4-9db1-a922304b7f4f\" (UID: \"11c66d39-e80b-4cb4-9db1-a922304b7f4f\") " Jul 29 19:24:54.648401 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:24:54.648382 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11c66d39-e80b-4cb4-9db1-a922304b7f4f-kube-api-access-578tg" (OuterVolumeSpecName: "kube-api-access-578tg") pod "11c66d39-e80b-4cb4-9db1-a922304b7f4f" (UID: "11c66d39-e80b-4cb4-9db1-a922304b7f4f"). InnerVolumeSpecName "kube-api-access-578tg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 19:24:54.658505 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:24:54.658481 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7504-139814632716304_11c66d39-e80b-4cb4-9db1-a922304b7f4f/test-curl-7504-139814632716304/0.log" Jul 29 19:24:54.685563 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:24:54.685537 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7504-139814632716304"] Jul 29 19:24:54.689137 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:24:54.689114 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7504-139814632716304"] Jul 29 19:24:54.747316 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:24:54.747290 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-578tg\" (UniqueName: \"kubernetes.io/projected/11c66d39-e80b-4cb4-9db1-a922304b7f4f-kube-api-access-578tg\") on node \"ip-10-0-140-205.ec2.internal\" DevicePath \"\"" Jul 29 19:24:54.819612 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:24:54.819538 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7504-139814632717312"] Jul 29 19:24:54.819912 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:24:54.819899 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="11c66d39-e80b-4cb4-9db1-a922304b7f4f" containerName="test-curl-7504-139814632716304" Jul 29 19:24:54.819954 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:24:54.819913 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="11c66d39-e80b-4cb4-9db1-a922304b7f4f" containerName="test-curl-7504-139814632716304" Jul 29 19:24:54.820011 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:24:54.820001 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="11c66d39-e80b-4cb4-9db1-a922304b7f4f" containerName="test-curl-7504-139814632716304" Jul 29 19:24:54.822934 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:24:54.822920 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7504-139814632717312" Jul 29 19:24:54.827366 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:24:54.827325 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7504-139814632717312"] Jul 29 19:24:54.848025 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:24:54.848002 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twgsj\" (UniqueName: \"kubernetes.io/projected/22983a40-375d-4db1-b1dc-58071935596a-kube-api-access-twgsj\") pod \"test-curl-7504-139814632717312\" (UID: \"22983a40-375d-4db1-b1dc-58071935596a\") " pod="opendatahub/test-curl-7504-139814632717312" Jul 29 19:24:54.949408 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:24:54.949373 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twgsj\" (UniqueName: \"kubernetes.io/projected/22983a40-375d-4db1-b1dc-58071935596a-kube-api-access-twgsj\") pod \"test-curl-7504-139814632717312\" (UID: \"22983a40-375d-4db1-b1dc-58071935596a\") " pod="opendatahub/test-curl-7504-139814632717312" Jul 29 19:24:54.958595 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:24:54.958568 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-twgsj\" (UniqueName: \"kubernetes.io/projected/22983a40-375d-4db1-b1dc-58071935596a-kube-api-access-twgsj\") pod \"test-curl-7504-139814632717312\" (UID: \"22983a40-375d-4db1-b1dc-58071935596a\") " pod="opendatahub/test-curl-7504-139814632717312" Jul 29 19:24:55.133561 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:24:55.133480 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7504-139814632717312" Jul 29 19:24:55.266018 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:24:55.265988 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7504-139814632717312"] Jul 29 19:24:55.267146 ip-10-0-140-205 kubenswrapper[2560]: W0729 19:24:55.267119 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod22983a40_375d_4db1_b1dc_58071935596a.slice/crio-e3832947cc24ed19fdda18b573b8b2507be93f1b6fe20a4d3eabd2d6fc836b6c WatchSource:0}: Error finding container e3832947cc24ed19fdda18b573b8b2507be93f1b6fe20a4d3eabd2d6fc836b6c: Status 404 returned error can't find the container with id e3832947cc24ed19fdda18b573b8b2507be93f1b6fe20a4d3eabd2d6fc836b6c Jul 29 19:24:55.355253 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:24:55.355226 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="11c66d39-e80b-4cb4-9db1-a922304b7f4f" path="/var/lib/kubelet/pods/11c66d39-e80b-4cb4-9db1-a922304b7f4f/volumes" Jul 29 19:24:55.508672 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:24:55.508644 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7504-139814632716304" Jul 29 19:24:55.508864 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:24:55.508682 2560 scope.go:117] "RemoveContainer" containerID="e7e5fd5656f3a9b22a2dae98b3936886e79dc795875477d9aec0db26122f7635" Jul 29 19:24:55.510014 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:24:55.509916 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7504-139814632717312" event={"ID":"22983a40-375d-4db1-b1dc-58071935596a","Type":"ContainerStarted","Data":"e3832947cc24ed19fdda18b573b8b2507be93f1b6fe20a4d3eabd2d6fc836b6c"} Jul 29 19:24:56.517958 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:24:56.517921 2560 generic.go:356] "Generic (PLEG): container finished" podID="22983a40-375d-4db1-b1dc-58071935596a" containerID="44e96fa83fa73e0d70e2bf512eacea1f07d1d391a3a4107f3df7a8957660a119" exitCode=0 Jul 29 19:24:56.518341 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:24:56.518016 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7504-139814632717312" event={"ID":"22983a40-375d-4db1-b1dc-58071935596a","Type":"ContainerDied","Data":"44e96fa83fa73e0d70e2bf512eacea1f07d1d391a3a4107f3df7a8957660a119"} Jul 29 19:24:56.686465 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:24:56.686424 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-422d30/rate-test-model-422d30-kserve-85ff8cfd79-fk86c" podUID="497efccf-9ff9-47e7-8bc9-c51a085b9dc0" containerName="main" containerID="cri-o://4e2e342b93b6e6b968ebbe388b1f714558e0c16f2c9835c4a03580d6b255adbd" gracePeriod=45 Jul 29 19:24:56.941971 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:24:56.941949 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-422d30/rate-test-model-422d30-kserve-85ff8cfd79-fk86c" Jul 29 19:24:56.962403 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:24:56.962370 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/497efccf-9ff9-47e7-8bc9-c51a085b9dc0-tls-certs\") pod \"497efccf-9ff9-47e7-8bc9-c51a085b9dc0\" (UID: \"497efccf-9ff9-47e7-8bc9-c51a085b9dc0\") " Jul 29 19:24:56.962559 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:24:56.962416 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/497efccf-9ff9-47e7-8bc9-c51a085b9dc0-tmp-dir\") pod \"497efccf-9ff9-47e7-8bc9-c51a085b9dc0\" (UID: \"497efccf-9ff9-47e7-8bc9-c51a085b9dc0\") " Jul 29 19:24:56.962559 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:24:56.962475 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-26qrr\" (UniqueName: \"kubernetes.io/projected/497efccf-9ff9-47e7-8bc9-c51a085b9dc0-kube-api-access-26qrr\") pod \"497efccf-9ff9-47e7-8bc9-c51a085b9dc0\" (UID: \"497efccf-9ff9-47e7-8bc9-c51a085b9dc0\") " Jul 29 19:24:56.962559 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:24:56.962523 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/497efccf-9ff9-47e7-8bc9-c51a085b9dc0-model-cache\") pod \"497efccf-9ff9-47e7-8bc9-c51a085b9dc0\" (UID: \"497efccf-9ff9-47e7-8bc9-c51a085b9dc0\") " Jul 29 19:24:56.962789 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:24:56.962577 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/497efccf-9ff9-47e7-8bc9-c51a085b9dc0-home\") pod \"497efccf-9ff9-47e7-8bc9-c51a085b9dc0\" (UID: \"497efccf-9ff9-47e7-8bc9-c51a085b9dc0\") " Jul 29 19:24:56.962789 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:24:56.962646 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/497efccf-9ff9-47e7-8bc9-c51a085b9dc0-dshm\") pod \"497efccf-9ff9-47e7-8bc9-c51a085b9dc0\" (UID: \"497efccf-9ff9-47e7-8bc9-c51a085b9dc0\") " Jul 29 19:24:56.962897 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:24:56.962781 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/497efccf-9ff9-47e7-8bc9-c51a085b9dc0-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "497efccf-9ff9-47e7-8bc9-c51a085b9dc0" (UID: "497efccf-9ff9-47e7-8bc9-c51a085b9dc0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 19:24:56.962966 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:24:56.962944 2560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/497efccf-9ff9-47e7-8bc9-c51a085b9dc0-tmp-dir\") on node \"ip-10-0-140-205.ec2.internal\" DevicePath \"\"" Jul 29 19:24:56.963030 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:24:56.962961 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/497efccf-9ff9-47e7-8bc9-c51a085b9dc0-model-cache" (OuterVolumeSpecName: "model-cache") pod "497efccf-9ff9-47e7-8bc9-c51a085b9dc0" (UID: "497efccf-9ff9-47e7-8bc9-c51a085b9dc0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 19:24:56.963030 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:24:56.962979 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/497efccf-9ff9-47e7-8bc9-c51a085b9dc0-home" (OuterVolumeSpecName: "home") pod "497efccf-9ff9-47e7-8bc9-c51a085b9dc0" (UID: "497efccf-9ff9-47e7-8bc9-c51a085b9dc0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 19:24:56.964727 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:24:56.964701 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/497efccf-9ff9-47e7-8bc9-c51a085b9dc0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "497efccf-9ff9-47e7-8bc9-c51a085b9dc0" (UID: "497efccf-9ff9-47e7-8bc9-c51a085b9dc0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 19:24:56.965057 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:24:56.965034 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/497efccf-9ff9-47e7-8bc9-c51a085b9dc0-kube-api-access-26qrr" (OuterVolumeSpecName: "kube-api-access-26qrr") pod "497efccf-9ff9-47e7-8bc9-c51a085b9dc0" (UID: "497efccf-9ff9-47e7-8bc9-c51a085b9dc0"). InnerVolumeSpecName "kube-api-access-26qrr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 19:24:56.965185 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:24:56.965168 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/497efccf-9ff9-47e7-8bc9-c51a085b9dc0-dshm" (OuterVolumeSpecName: "dshm") pod "497efccf-9ff9-47e7-8bc9-c51a085b9dc0" (UID: "497efccf-9ff9-47e7-8bc9-c51a085b9dc0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 19:24:57.063626 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:24:57.063553 2560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/497efccf-9ff9-47e7-8bc9-c51a085b9dc0-dshm\") on node \"ip-10-0-140-205.ec2.internal\" DevicePath \"\"" Jul 29 19:24:57.063626 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:24:57.063577 2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/497efccf-9ff9-47e7-8bc9-c51a085b9dc0-tls-certs\") on node \"ip-10-0-140-205.ec2.internal\" DevicePath \"\"" Jul 29 19:24:57.063626 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:24:57.063589 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-26qrr\" (UniqueName: \"kubernetes.io/projected/497efccf-9ff9-47e7-8bc9-c51a085b9dc0-kube-api-access-26qrr\") on node \"ip-10-0-140-205.ec2.internal\" DevicePath \"\"" Jul 29 19:24:57.063626 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:24:57.063599 2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/497efccf-9ff9-47e7-8bc9-c51a085b9dc0-model-cache\") on node \"ip-10-0-140-205.ec2.internal\" DevicePath \"\"" Jul 29 19:24:57.063626 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:24:57.063607 2560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/497efccf-9ff9-47e7-8bc9-c51a085b9dc0-home\") on node \"ip-10-0-140-205.ec2.internal\" DevicePath \"\"" Jul 29 19:24:57.524539 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:24:57.524509 2560 generic.go:356] "Generic (PLEG): container finished" podID="497efccf-9ff9-47e7-8bc9-c51a085b9dc0" containerID="4e2e342b93b6e6b968ebbe388b1f714558e0c16f2c9835c4a03580d6b255adbd" exitCode=0 Jul 29 19:24:57.524930 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:24:57.524551 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-422d30/rate-test-model-422d30-kserve-85ff8cfd79-fk86c" Jul 29 19:24:57.524930 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:24:57.524602 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-422d30/rate-test-model-422d30-kserve-85ff8cfd79-fk86c" event={"ID":"497efccf-9ff9-47e7-8bc9-c51a085b9dc0","Type":"ContainerDied","Data":"4e2e342b93b6e6b968ebbe388b1f714558e0c16f2c9835c4a03580d6b255adbd"} Jul 29 19:24:57.524930 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:24:57.524642 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-422d30/rate-test-model-422d30-kserve-85ff8cfd79-fk86c" event={"ID":"497efccf-9ff9-47e7-8bc9-c51a085b9dc0","Type":"ContainerDied","Data":"bf9b37a9cd59dc346abc197f116c613ca2e44db60b5c2d42c8d1b160900521e9"} Jul 29 19:24:57.524930 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:24:57.524659 2560 scope.go:117] "RemoveContainer" containerID="4e2e342b93b6e6b968ebbe388b1f714558e0c16f2c9835c4a03580d6b255adbd" Jul 29 19:24:57.534039 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:24:57.534020 2560 scope.go:117] "RemoveContainer" containerID="4e2e342b93b6e6b968ebbe388b1f714558e0c16f2c9835c4a03580d6b255adbd" Jul 29 19:24:57.534309 ip-10-0-140-205 kubenswrapper[2560]: E0729 19:24:57.534291 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e2e342b93b6e6b968ebbe388b1f714558e0c16f2c9835c4a03580d6b255adbd\": container with ID starting with 4e2e342b93b6e6b968ebbe388b1f714558e0c16f2c9835c4a03580d6b255adbd not found: ID does not exist" containerID="4e2e342b93b6e6b968ebbe388b1f714558e0c16f2c9835c4a03580d6b255adbd" Jul 29 19:24:57.534419 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:24:57.534320 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e2e342b93b6e6b968ebbe388b1f714558e0c16f2c9835c4a03580d6b255adbd"} err="failed to get container status \"4e2e342b93b6e6b968ebbe388b1f714558e0c16f2c9835c4a03580d6b255adbd\": rpc error: code = NotFound desc = could not find container \"4e2e342b93b6e6b968ebbe388b1f714558e0c16f2c9835c4a03580d6b255adbd\": container with ID starting with 4e2e342b93b6e6b968ebbe388b1f714558e0c16f2c9835c4a03580d6b255adbd not found: ID does not exist" Jul 29 19:24:57.571379 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:24:57.571318 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-422d30/rate-test-model-422d30-kserve-85ff8cfd79-fk86c"] Jul 29 19:24:57.574737 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:24:57.574712 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-422d30/rate-test-model-422d30-kserve-85ff8cfd79-fk86c"] Jul 29 19:24:57.649094 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:24:57.649073 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7504-139814632717312" Jul 29 19:24:57.666841 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:24:57.666816 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-twgsj\" (UniqueName: \"kubernetes.io/projected/22983a40-375d-4db1-b1dc-58071935596a-kube-api-access-twgsj\") pod \"22983a40-375d-4db1-b1dc-58071935596a\" (UID: \"22983a40-375d-4db1-b1dc-58071935596a\") " Jul 29 19:24:57.668879 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:24:57.668860 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22983a40-375d-4db1-b1dc-58071935596a-kube-api-access-twgsj" (OuterVolumeSpecName: "kube-api-access-twgsj") pod "22983a40-375d-4db1-b1dc-58071935596a" (UID: "22983a40-375d-4db1-b1dc-58071935596a"). InnerVolumeSpecName "kube-api-access-twgsj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 19:24:57.676968 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:24:57.676948 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7504-139814632717312_22983a40-375d-4db1-b1dc-58071935596a/test-curl-7504-139814632717312/0.log" Jul 29 19:24:57.702038 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:24:57.702015 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7504-139814632717312"] Jul 29 19:24:57.706192 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:24:57.706173 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7504-139814632717312"] Jul 29 19:24:57.767488 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:24:57.767462 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-twgsj\" (UniqueName: \"kubernetes.io/projected/22983a40-375d-4db1-b1dc-58071935596a-kube-api-access-twgsj\") on node \"ip-10-0-140-205.ec2.internal\" DevicePath \"\"" Jul 29 19:24:57.845071 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:24:57.845033 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7504-139814632717312"] Jul 29 19:24:57.845496 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:24:57.845479 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="22983a40-375d-4db1-b1dc-58071935596a" containerName="test-curl-7504-139814632717312" Jul 29 19:24:57.845602 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:24:57.845498 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="22983a40-375d-4db1-b1dc-58071935596a" containerName="test-curl-7504-139814632717312" Jul 29 19:24:57.845602 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:24:57.845528 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="497efccf-9ff9-47e7-8bc9-c51a085b9dc0" containerName="main" Jul 29 19:24:57.845602 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:24:57.845538 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="497efccf-9ff9-47e7-8bc9-c51a085b9dc0" containerName="main" Jul 29 19:24:57.845761 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:24:57.845644 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="497efccf-9ff9-47e7-8bc9-c51a085b9dc0" containerName="main" Jul 29 19:24:57.845761 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:24:57.845659 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="22983a40-375d-4db1-b1dc-58071935596a" containerName="test-curl-7504-139814632717312" Jul 29 19:24:57.848753 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:24:57.848735 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7504-139814632717312" Jul 29 19:24:57.854276 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:24:57.854258 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7504-139814632717312"] Jul 29 19:24:57.868021 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:24:57.868002 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wr2cq\" (UniqueName: \"kubernetes.io/projected/24fda91c-4dbd-47cf-aad1-169340e95441-kube-api-access-wr2cq\") pod \"test-curl-7504-139814632717312\" (UID: \"24fda91c-4dbd-47cf-aad1-169340e95441\") " pod="opendatahub/test-curl-7504-139814632717312" Jul 29 19:24:57.969434 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:24:57.969367 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wr2cq\" (UniqueName: \"kubernetes.io/projected/24fda91c-4dbd-47cf-aad1-169340e95441-kube-api-access-wr2cq\") pod \"test-curl-7504-139814632717312\" (UID: \"24fda91c-4dbd-47cf-aad1-169340e95441\") " pod="opendatahub/test-curl-7504-139814632717312" Jul 29 19:24:57.977902 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:24:57.977876 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wr2cq\" (UniqueName: \"kubernetes.io/projected/24fda91c-4dbd-47cf-aad1-169340e95441-kube-api-access-wr2cq\") pod \"test-curl-7504-139814632717312\" (UID: \"24fda91c-4dbd-47cf-aad1-169340e95441\") " pod="opendatahub/test-curl-7504-139814632717312" Jul 29 19:24:58.159023 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:24:58.158985 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7504-139814632717312" Jul 29 19:24:58.292085 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:24:58.292055 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7504-139814632717312"] Jul 29 19:24:58.293762 ip-10-0-140-205 kubenswrapper[2560]: W0729 19:24:58.293732 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod24fda91c_4dbd_47cf_aad1_169340e95441.slice/crio-49a4c2b3047ad81978fc52ef2e317438502a909a534abcccf1a03b7660221518 WatchSource:0}: Error finding container 49a4c2b3047ad81978fc52ef2e317438502a909a534abcccf1a03b7660221518: Status 404 returned error can't find the container with id 49a4c2b3047ad81978fc52ef2e317438502a909a534abcccf1a03b7660221518 Jul 29 19:24:58.532513 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:24:58.532485 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7504-139814632717312" Jul 29 19:24:58.532952 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:24:58.532524 2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e3832947cc24ed19fdda18b573b8b2507be93f1b6fe20a4d3eabd2d6fc836b6c" Jul 29 19:24:58.533688 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:24:58.533661 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7504-139814632717312" event={"ID":"24fda91c-4dbd-47cf-aad1-169340e95441","Type":"ContainerStarted","Data":"49a4c2b3047ad81978fc52ef2e317438502a909a534abcccf1a03b7660221518"} Jul 29 19:24:58.544981 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:24:58.544956 2560 status_manager.go:1027] "Pod was deleted and then recreated, skipping status update" pod="opendatahub/test-curl-7504-139814632717312" oldPodUID="22983a40-375d-4db1-b1dc-58071935596a" podUID="24fda91c-4dbd-47cf-aad1-169340e95441" Jul 29 19:24:59.354728 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:24:59.354699 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="22983a40-375d-4db1-b1dc-58071935596a" path="/var/lib/kubelet/pods/22983a40-375d-4db1-b1dc-58071935596a/volumes" Jul 29 19:24:59.355119 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:24:59.355106 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="497efccf-9ff9-47e7-8bc9-c51a085b9dc0" path="/var/lib/kubelet/pods/497efccf-9ff9-47e7-8bc9-c51a085b9dc0/volumes" Jul 29 19:24:59.541265 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:24:59.541233 2560 generic.go:356] "Generic (PLEG): container finished" podID="24fda91c-4dbd-47cf-aad1-169340e95441" containerID="a3e680e8980dd24a63a218dacafbc74c7dcf3736bd29a9b24ec4282930e80440" exitCode=0 Jul 29 19:24:59.541631 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:24:59.541296 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7504-139814632717312" event={"ID":"24fda91c-4dbd-47cf-aad1-169340e95441","Type":"ContainerDied","Data":"a3e680e8980dd24a63a218dacafbc74c7dcf3736bd29a9b24ec4282930e80440"} Jul 29 19:25:00.684936 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:25:00.684910 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7504-139814632717312" Jul 29 19:25:00.688565 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:25:00.688545 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wr2cq\" (UniqueName: \"kubernetes.io/projected/24fda91c-4dbd-47cf-aad1-169340e95441-kube-api-access-wr2cq\") pod \"24fda91c-4dbd-47cf-aad1-169340e95441\" (UID: \"24fda91c-4dbd-47cf-aad1-169340e95441\") " Jul 29 19:25:00.690626 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:25:00.690603 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24fda91c-4dbd-47cf-aad1-169340e95441-kube-api-access-wr2cq" (OuterVolumeSpecName: "kube-api-access-wr2cq") pod "24fda91c-4dbd-47cf-aad1-169340e95441" (UID: "24fda91c-4dbd-47cf-aad1-169340e95441"). InnerVolumeSpecName "kube-api-access-wr2cq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 19:25:00.705552 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:25:00.705529 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7504-139814632717312_24fda91c-4dbd-47cf-aad1-169340e95441/test-curl-7504-139814632717312/0.log" Jul 29 19:25:00.738144 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:25:00.738114 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7504-139814632717312"] Jul 29 19:25:00.743981 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:25:00.743955 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7504-139814632717312"] Jul 29 19:25:00.790224 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:25:00.790197 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wr2cq\" (UniqueName: \"kubernetes.io/projected/24fda91c-4dbd-47cf-aad1-169340e95441-kube-api-access-wr2cq\") on node \"ip-10-0-140-205.ec2.internal\" DevicePath \"\"" Jul 29 19:25:01.355326 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:25:01.355289 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="24fda91c-4dbd-47cf-aad1-169340e95441" path="/var/lib/kubelet/pods/24fda91c-4dbd-47cf-aad1-169340e95441/volumes" Jul 29 19:25:01.554256 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:25:01.554226 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7504-139814632717312" Jul 29 19:25:01.554493 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:25:01.554261 2560 scope.go:117] "RemoveContainer" containerID="a3e680e8980dd24a63a218dacafbc74c7dcf3736bd29a9b24ec4282930e80440" Jul 29 19:25:02.447049 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:25:02.447010 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:25:02.559429 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:25:02.559397 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:25:04.283838 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:25:04.283807 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:25:04.382005 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:25:04.381972 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:25:04.526991 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:25:04.526961 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:25:04.636324 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:25:04.636234 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:25:04.800245 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:25:04.800207 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:25:05.160590 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:25:05.160547 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:25:05.263640 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:25:05.263608 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:25:29.325949 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:25:29.325879 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:25:30.107900 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:25:30.107861 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:25:30.209343 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:25:30.209307 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:25:30.366321 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:25:30.366247 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-f0bee1-openshift-default-54d6899946-pjlwb"] Jul 29 19:25:30.366699 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:25:30.366685 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="24fda91c-4dbd-47cf-aad1-169340e95441" containerName="test-curl-7504-139814632717312" Jul 29 19:25:30.366744 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:25:30.366701 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="24fda91c-4dbd-47cf-aad1-169340e95441" containerName="test-curl-7504-139814632717312" Jul 29 19:25:30.366800 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:25:30.366789 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="24fda91c-4dbd-47cf-aad1-169340e95441" containerName="test-curl-7504-139814632717312" Jul 29 19:25:30.372011 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:25:30.371990 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-f0bee1-openshift-default-54d6899946-pjlwb" Jul 29 19:25:30.374387 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:25:30.374365 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 29 19:25:30.374520 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:25:30.374467 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-f0bee1-openshift-default-dockercfg-fbhcr\"" Jul 29 19:25:30.386155 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:25:30.386130 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-f0bee1-openshift-default-54d6899946-pjlwb"] Jul 29 19:25:30.406139 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:25:30.406114 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:25:30.446935 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:25:30.446909 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7210e159-b3ca-4448-9604-81c688188d2c-istio-envoy\") pod \"e2e-ipp-b-f0bee1-openshift-default-54d6899946-pjlwb\" (UID: \"7210e159-b3ca-4448-9604-81c688188d2c\") " pod="openshift-ingress/e2e-ipp-b-f0bee1-openshift-default-54d6899946-pjlwb" Jul 29 19:25:30.447037 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:25:30.446937 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7210e159-b3ca-4448-9604-81c688188d2c-istio-podinfo\") pod \"e2e-ipp-b-f0bee1-openshift-default-54d6899946-pjlwb\" (UID: \"7210e159-b3ca-4448-9604-81c688188d2c\") " pod="openshift-ingress/e2e-ipp-b-f0bee1-openshift-default-54d6899946-pjlwb" Jul 29 19:25:30.447037 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:25:30.446956 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7210e159-b3ca-4448-9604-81c688188d2c-istiod-ca-cert\") pod \"e2e-ipp-b-f0bee1-openshift-default-54d6899946-pjlwb\" (UID: \"7210e159-b3ca-4448-9604-81c688188d2c\") " pod="openshift-ingress/e2e-ipp-b-f0bee1-openshift-default-54d6899946-pjlwb" Jul 29 19:25:30.447037 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:25:30.446977 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7210e159-b3ca-4448-9604-81c688188d2c-istio-token\") pod \"e2e-ipp-b-f0bee1-openshift-default-54d6899946-pjlwb\" (UID: \"7210e159-b3ca-4448-9604-81c688188d2c\") " pod="openshift-ingress/e2e-ipp-b-f0bee1-openshift-default-54d6899946-pjlwb" Jul 29 19:25:30.447148 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:25:30.447056 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjqdp\" (UniqueName: \"kubernetes.io/projected/7210e159-b3ca-4448-9604-81c688188d2c-kube-api-access-qjqdp\") pod \"e2e-ipp-b-f0bee1-openshift-default-54d6899946-pjlwb\" (UID: \"7210e159-b3ca-4448-9604-81c688188d2c\") " pod="openshift-ingress/e2e-ipp-b-f0bee1-openshift-default-54d6899946-pjlwb" Jul 29 19:25:30.447148 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:25:30.447143 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7210e159-b3ca-4448-9604-81c688188d2c-workload-certs\") pod \"e2e-ipp-b-f0bee1-openshift-default-54d6899946-pjlwb\" (UID: \"7210e159-b3ca-4448-9604-81c688188d2c\") " pod="openshift-ingress/e2e-ipp-b-f0bee1-openshift-default-54d6899946-pjlwb" Jul 29 19:25:30.447213 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:25:30.447183 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7210e159-b3ca-4448-9604-81c688188d2c-credential-socket\") pod \"e2e-ipp-b-f0bee1-openshift-default-54d6899946-pjlwb\" (UID: \"7210e159-b3ca-4448-9604-81c688188d2c\") " pod="openshift-ingress/e2e-ipp-b-f0bee1-openshift-default-54d6899946-pjlwb" Jul 29 19:25:30.447213 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:25:30.447202 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7210e159-b3ca-4448-9604-81c688188d2c-workload-socket\") pod \"e2e-ipp-b-f0bee1-openshift-default-54d6899946-pjlwb\" (UID: \"7210e159-b3ca-4448-9604-81c688188d2c\") " pod="openshift-ingress/e2e-ipp-b-f0bee1-openshift-default-54d6899946-pjlwb" Jul 29 19:25:30.447278 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:25:30.447231 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7210e159-b3ca-4448-9604-81c688188d2c-istio-data\") pod \"e2e-ipp-b-f0bee1-openshift-default-54d6899946-pjlwb\" (UID: \"7210e159-b3ca-4448-9604-81c688188d2c\") " pod="openshift-ingress/e2e-ipp-b-f0bee1-openshift-default-54d6899946-pjlwb" Jul 29 19:25:30.548226 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:25:30.548197 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7210e159-b3ca-4448-9604-81c688188d2c-workload-certs\") pod \"e2e-ipp-b-f0bee1-openshift-default-54d6899946-pjlwb\" (UID: \"7210e159-b3ca-4448-9604-81c688188d2c\") " pod="openshift-ingress/e2e-ipp-b-f0bee1-openshift-default-54d6899946-pjlwb" Jul 29 19:25:30.548372 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:25:30.548238 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7210e159-b3ca-4448-9604-81c688188d2c-credential-socket\") pod \"e2e-ipp-b-f0bee1-openshift-default-54d6899946-pjlwb\" (UID: \"7210e159-b3ca-4448-9604-81c688188d2c\") " pod="openshift-ingress/e2e-ipp-b-f0bee1-openshift-default-54d6899946-pjlwb" Jul 29 19:25:30.548442 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:25:30.548371 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7210e159-b3ca-4448-9604-81c688188d2c-workload-socket\") pod \"e2e-ipp-b-f0bee1-openshift-default-54d6899946-pjlwb\" (UID: \"7210e159-b3ca-4448-9604-81c688188d2c\") " pod="openshift-ingress/e2e-ipp-b-f0bee1-openshift-default-54d6899946-pjlwb" Jul 29 19:25:30.548442 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:25:30.548406 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7210e159-b3ca-4448-9604-81c688188d2c-istio-data\") pod \"e2e-ipp-b-f0bee1-openshift-default-54d6899946-pjlwb\" (UID: \"7210e159-b3ca-4448-9604-81c688188d2c\") " pod="openshift-ingress/e2e-ipp-b-f0bee1-openshift-default-54d6899946-pjlwb" Jul 29 19:25:30.548442 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:25:30.548432 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7210e159-b3ca-4448-9604-81c688188d2c-istio-envoy\") pod \"e2e-ipp-b-f0bee1-openshift-default-54d6899946-pjlwb\" (UID: \"7210e159-b3ca-4448-9604-81c688188d2c\") " pod="openshift-ingress/e2e-ipp-b-f0bee1-openshift-default-54d6899946-pjlwb" Jul 29 19:25:30.548600 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:25:30.548457 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7210e159-b3ca-4448-9604-81c688188d2c-istio-podinfo\") pod \"e2e-ipp-b-f0bee1-openshift-default-54d6899946-pjlwb\" (UID: \"7210e159-b3ca-4448-9604-81c688188d2c\") " pod="openshift-ingress/e2e-ipp-b-f0bee1-openshift-default-54d6899946-pjlwb" Jul 29 19:25:30.548600 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:25:30.548486 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7210e159-b3ca-4448-9604-81c688188d2c-istiod-ca-cert\") pod \"e2e-ipp-b-f0bee1-openshift-default-54d6899946-pjlwb\" (UID: \"7210e159-b3ca-4448-9604-81c688188d2c\") " pod="openshift-ingress/e2e-ipp-b-f0bee1-openshift-default-54d6899946-pjlwb" Jul 29 19:25:30.548709 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:25:30.548653 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7210e159-b3ca-4448-9604-81c688188d2c-credential-socket\") pod \"e2e-ipp-b-f0bee1-openshift-default-54d6899946-pjlwb\" (UID: \"7210e159-b3ca-4448-9604-81c688188d2c\") " pod="openshift-ingress/e2e-ipp-b-f0bee1-openshift-default-54d6899946-pjlwb" Jul 29 19:25:30.548768 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:25:30.548702 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7210e159-b3ca-4448-9604-81c688188d2c-istio-token\") pod \"e2e-ipp-b-f0bee1-openshift-default-54d6899946-pjlwb\" (UID: \"7210e159-b3ca-4448-9604-81c688188d2c\") " pod="openshift-ingress/e2e-ipp-b-f0bee1-openshift-default-54d6899946-pjlwb" Jul 29 19:25:30.548768 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:25:30.548731 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7210e159-b3ca-4448-9604-81c688188d2c-workload-certs\") pod \"e2e-ipp-b-f0bee1-openshift-default-54d6899946-pjlwb\" (UID: \"7210e159-b3ca-4448-9604-81c688188d2c\") " pod="openshift-ingress/e2e-ipp-b-f0bee1-openshift-default-54d6899946-pjlwb" Jul 29 19:25:30.548768 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:25:30.548752 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7210e159-b3ca-4448-9604-81c688188d2c-istio-data\") pod \"e2e-ipp-b-f0bee1-openshift-default-54d6899946-pjlwb\" (UID: \"7210e159-b3ca-4448-9604-81c688188d2c\") " pod="openshift-ingress/e2e-ipp-b-f0bee1-openshift-default-54d6899946-pjlwb" Jul 29 19:25:30.548927 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:25:30.548837 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjqdp\" (UniqueName: \"kubernetes.io/projected/7210e159-b3ca-4448-9604-81c688188d2c-kube-api-access-qjqdp\") pod \"e2e-ipp-b-f0bee1-openshift-default-54d6899946-pjlwb\" (UID: \"7210e159-b3ca-4448-9604-81c688188d2c\") " pod="openshift-ingress/e2e-ipp-b-f0bee1-openshift-default-54d6899946-pjlwb" Jul 29 19:25:30.548927 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:25:30.548874 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7210e159-b3ca-4448-9604-81c688188d2c-workload-socket\") pod \"e2e-ipp-b-f0bee1-openshift-default-54d6899946-pjlwb\" (UID: \"7210e159-b3ca-4448-9604-81c688188d2c\") " pod="openshift-ingress/e2e-ipp-b-f0bee1-openshift-default-54d6899946-pjlwb" Jul 29 19:25:30.549248 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:25:30.549227 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7210e159-b3ca-4448-9604-81c688188d2c-istiod-ca-cert\") pod \"e2e-ipp-b-f0bee1-openshift-default-54d6899946-pjlwb\" (UID: \"7210e159-b3ca-4448-9604-81c688188d2c\") " pod="openshift-ingress/e2e-ipp-b-f0bee1-openshift-default-54d6899946-pjlwb" Jul 29 19:25:30.550919 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:25:30.550898 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7210e159-b3ca-4448-9604-81c688188d2c-istio-envoy\") pod \"e2e-ipp-b-f0bee1-openshift-default-54d6899946-pjlwb\" (UID: \"7210e159-b3ca-4448-9604-81c688188d2c\") " pod="openshift-ingress/e2e-ipp-b-f0bee1-openshift-default-54d6899946-pjlwb" Jul 29 19:25:30.551252 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:25:30.551226 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7210e159-b3ca-4448-9604-81c688188d2c-istio-podinfo\") pod \"e2e-ipp-b-f0bee1-openshift-default-54d6899946-pjlwb\" (UID: \"7210e159-b3ca-4448-9604-81c688188d2c\") " pod="openshift-ingress/e2e-ipp-b-f0bee1-openshift-default-54d6899946-pjlwb" Jul 29 19:25:30.561201 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:25:30.561180 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjqdp\" (UniqueName: \"kubernetes.io/projected/7210e159-b3ca-4448-9604-81c688188d2c-kube-api-access-qjqdp\") pod \"e2e-ipp-b-f0bee1-openshift-default-54d6899946-pjlwb\" (UID: \"7210e159-b3ca-4448-9604-81c688188d2c\") " pod="openshift-ingress/e2e-ipp-b-f0bee1-openshift-default-54d6899946-pjlwb" Jul 29 19:25:30.561284 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:25:30.561255 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7210e159-b3ca-4448-9604-81c688188d2c-istio-token\") pod \"e2e-ipp-b-f0bee1-openshift-default-54d6899946-pjlwb\" (UID: \"7210e159-b3ca-4448-9604-81c688188d2c\") " pod="openshift-ingress/e2e-ipp-b-f0bee1-openshift-default-54d6899946-pjlwb" Jul 29 19:25:30.682569 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:25:30.682500 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-f0bee1-openshift-default-54d6899946-pjlwb" Jul 29 19:25:30.827728 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:25:30.827701 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-f0bee1-openshift-default-54d6899946-pjlwb"] Jul 29 19:25:30.829842 ip-10-0-140-205 kubenswrapper[2560]: W0729 19:25:30.829791 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7210e159_b3ca_4448_9604_81c688188d2c.slice/crio-d6b681e79ae8426d4534019c7e5f91718741813d3af3daada9caf4d51f1ddbe9 WatchSource:0}: Error finding container d6b681e79ae8426d4534019c7e5f91718741813d3af3daada9caf4d51f1ddbe9: Status 404 returned error can't find the container with id d6b681e79ae8426d4534019c7e5f91718741813d3af3daada9caf4d51f1ddbe9 Jul 29 19:25:30.937574 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:25:30.937504 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:25:31.729995 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:25:31.729956 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-f0bee1-openshift-default-54d6899946-pjlwb" event={"ID":"7210e159-b3ca-4448-9604-81c688188d2c","Type":"ContainerStarted","Data":"d6b681e79ae8426d4534019c7e5f91718741813d3af3daada9caf4d51f1ddbe9"} Jul 29 19:25:33.232604 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:25:33.232557 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:25:33.905106 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:25:33.905060 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 29 19:25:33.905252 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:25:33.905172 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 29 19:25:33.905252 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:25:33.905219 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 29 19:25:34.750069 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:25:34.750030 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-f0bee1-openshift-default-54d6899946-pjlwb" event={"ID":"7210e159-b3ca-4448-9604-81c688188d2c","Type":"ContainerStarted","Data":"5e1ead99057e37806c344b40412ce8e02d1eac7fc82917520136b2465c4c3524"} Jul 29 19:25:34.772084 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:25:34.772022 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-f0bee1-openshift-default-54d6899946-pjlwb" podStartSLOduration=1.699105709 podStartE2EDuration="4.772008094s" totalImagesPullingTime="3.072902385s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:25:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:25:34.770837644 +0000 UTC m=+3069.996506516" watchObservedRunningTime="2026-07-29 19:25:34.772008094 +0000 UTC m=+3069.997676965" Jul 29 19:25:35.683075 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:25:35.683044 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-f0bee1-openshift-default-54d6899946-pjlwb" Jul 29 19:25:35.688680 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:25:35.688653 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-f0bee1-openshift-default-54d6899946-pjlwb" Jul 29 19:25:35.756077 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:25:35.756048 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-f0bee1-openshift-default-54d6899946-pjlwb" Jul 29 19:25:35.756994 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:25:35.756973 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-f0bee1-openshift-default-54d6899946-pjlwb" Jul 29 19:25:36.748297 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:25:36.748266 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:25:36.845863 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:25:36.845829 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:25:36.928804 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:25:36.928766 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:25:38.953525 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:25:38.953490 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:25:39.029676 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:25:39.029640 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:25:50.920448 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:25:50.919834 2560 patch_prober.go:28] interesting pod/e2e-ipp-b-f0bee1-openshift-default-54d6899946-pjlwb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": read tcp 10.134.0.2:32832->10.134.0.62:15021: read: connection reset by peer" start-of-body= Jul 29 19:25:50.920448 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:25:50.919905 2560 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-f0bee1-openshift-default-54d6899946-pjlwb" podUID="7210e159-b3ca-4448-9604-81c688188d2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": read tcp 10.134.0.2:32832->10.134.0.62:15021: read: connection reset by peer" Jul 29 19:25:51.852529 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:25:51.852492 2560 generic.go:356] "Generic (PLEG): container finished" podID="7210e159-b3ca-4448-9604-81c688188d2c" containerID="5e1ead99057e37806c344b40412ce8e02d1eac7fc82917520136b2465c4c3524" exitCode=0 Jul 29 19:25:51.852695 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:25:51.852560 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-f0bee1-openshift-default-54d6899946-pjlwb" event={"ID":"7210e159-b3ca-4448-9604-81c688188d2c","Type":"ContainerDied","Data":"5e1ead99057e37806c344b40412ce8e02d1eac7fc82917520136b2465c4c3524"} Jul 29 19:25:51.852937 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:25:51.852922 2560 scope.go:117] "RemoveContainer" containerID="5e1ead99057e37806c344b40412ce8e02d1eac7fc82917520136b2465c4c3524" Jul 29 19:25:51.854980 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:25:51.854949 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 29 19:25:51.855078 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:25:51.855012 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 29 19:25:51.855078 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:25:51.855040 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 29 19:25:52.860508 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:25:52.860470 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-f0bee1-openshift-default-54d6899946-pjlwb" event={"ID":"7210e159-b3ca-4448-9604-81c688188d2c","Type":"ContainerStarted","Data":"15e37a9820c18a74f7e0905b3dd1ee7887790f965ca3d56dd69c91fba6e31ca2"} Jul 29 19:25:53.682837 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:25:53.682808 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-f0bee1-openshift-default-54d6899946-pjlwb" Jul 29 19:25:53.683122 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:25:53.683103 2560 patch_prober.go:28] interesting pod/e2e-ipp-b-f0bee1-openshift-default-54d6899946-pjlwb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Jul 29 19:25:53.683173 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:25:53.683148 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-f0bee1-openshift-default-54d6899946-pjlwb" podUID="7210e159-b3ca-4448-9604-81c688188d2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Jul 29 19:25:54.683652 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:25:54.683613 2560 patch_prober.go:28] interesting pod/e2e-ipp-b-f0bee1-openshift-default-54d6899946-pjlwb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Jul 29 19:25:54.684058 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:25:54.683669 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-f0bee1-openshift-default-54d6899946-pjlwb" podUID="7210e159-b3ca-4448-9604-81c688188d2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Jul 29 19:25:55.682890 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:25:55.682851 2560 patch_prober.go:28] interesting pod/e2e-ipp-b-f0bee1-openshift-default-54d6899946-pjlwb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Jul 29 19:25:55.683070 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:25:55.682910 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-f0bee1-openshift-default-54d6899946-pjlwb" podUID="7210e159-b3ca-4448-9604-81c688188d2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Jul 29 19:25:56.691260 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:25:56.691222 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-f0bee1-openshift-default-54d6899946-pjlwb" Jul 29 19:25:56.691752 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:25:56.691588 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-f0bee1-openshift-default-54d6899946-pjlwb" Jul 29 19:25:56.692326 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:25:56.692309 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-f0bee1-openshift-default-54d6899946-pjlwb" Jul 29 19:25:58.045047 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:25:58.045020 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:25:59.122972 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:25:59.122930 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:26:01.857887 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:26:01.857848 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:26:13.383256 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:26:13.383221 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-cc3e80/ipp-route-cc3e80-kserve-5f76b6665f-dlztz"] Jul 29 19:26:13.386562 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:26:13.386545 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-cc3e80/ipp-route-cc3e80-kserve-5f76b6665f-dlztz" Jul 29 19:26:13.389203 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:26:13.389177 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-cc3e80\"/\"kube-root-ca.crt\"" Jul 29 19:26:13.389940 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:26:13.389923 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-cc3e80\"/\"openshift-service-ca.crt\"" Jul 29 19:26:13.390152 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:26:13.390134 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-cc3e80\"/\"default-dockercfg-jjv2t\"" Jul 29 19:26:13.390197 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:26:13.390134 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-cc3e80\"/\"ipp-route-cc3e80-kserve-self-signed-certs\"" Jul 29 19:26:13.397946 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:26:13.397926 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-cc3e80/ipp-route-cc3e80-kserve-5f76b6665f-dlztz"] Jul 29 19:26:13.410973 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:26:13.410933 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:26:13.435556 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:26:13.435525 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b60018dc-b127-4100-8452-4087fd850a27-home\") pod \"ipp-route-cc3e80-kserve-5f76b6665f-dlztz\" (UID: \"b60018dc-b127-4100-8452-4087fd850a27\") " pod="ai-tenant-e2e-ipp-a-cc3e80/ipp-route-cc3e80-kserve-5f76b6665f-dlztz" Jul 29 19:26:13.435642 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:26:13.435565 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b60018dc-b127-4100-8452-4087fd850a27-dshm\") pod \"ipp-route-cc3e80-kserve-5f76b6665f-dlztz\" (UID: \"b60018dc-b127-4100-8452-4087fd850a27\") " pod="ai-tenant-e2e-ipp-a-cc3e80/ipp-route-cc3e80-kserve-5f76b6665f-dlztz" Jul 29 19:26:13.435642 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:26:13.435603 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b60018dc-b127-4100-8452-4087fd850a27-model-cache\") pod \"ipp-route-cc3e80-kserve-5f76b6665f-dlztz\" (UID: \"b60018dc-b127-4100-8452-4087fd850a27\") " pod="ai-tenant-e2e-ipp-a-cc3e80/ipp-route-cc3e80-kserve-5f76b6665f-dlztz" Jul 29 19:26:13.435719 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:26:13.435666 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b60018dc-b127-4100-8452-4087fd850a27-tls-certs\") pod \"ipp-route-cc3e80-kserve-5f76b6665f-dlztz\" (UID: \"b60018dc-b127-4100-8452-4087fd850a27\") " pod="ai-tenant-e2e-ipp-a-cc3e80/ipp-route-cc3e80-kserve-5f76b6665f-dlztz" Jul 29 19:26:13.435719 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:26:13.435709 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7458m\" (UniqueName: \"kubernetes.io/projected/b60018dc-b127-4100-8452-4087fd850a27-kube-api-access-7458m\") pod \"ipp-route-cc3e80-kserve-5f76b6665f-dlztz\" (UID: \"b60018dc-b127-4100-8452-4087fd850a27\") " pod="ai-tenant-e2e-ipp-a-cc3e80/ipp-route-cc3e80-kserve-5f76b6665f-dlztz" Jul 29 19:26:13.435784 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:26:13.435743 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b60018dc-b127-4100-8452-4087fd850a27-tmp-dir\") pod \"ipp-route-cc3e80-kserve-5f76b6665f-dlztz\" (UID: \"b60018dc-b127-4100-8452-4087fd850a27\") " pod="ai-tenant-e2e-ipp-a-cc3e80/ipp-route-cc3e80-kserve-5f76b6665f-dlztz" Jul 29 19:26:13.536817 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:26:13.536788 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b60018dc-b127-4100-8452-4087fd850a27-tls-certs\") pod \"ipp-route-cc3e80-kserve-5f76b6665f-dlztz\" (UID: \"b60018dc-b127-4100-8452-4087fd850a27\") " pod="ai-tenant-e2e-ipp-a-cc3e80/ipp-route-cc3e80-kserve-5f76b6665f-dlztz" Jul 29 19:26:13.536964 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:26:13.536831 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7458m\" (UniqueName: \"kubernetes.io/projected/b60018dc-b127-4100-8452-4087fd850a27-kube-api-access-7458m\") pod \"ipp-route-cc3e80-kserve-5f76b6665f-dlztz\" (UID: \"b60018dc-b127-4100-8452-4087fd850a27\") " pod="ai-tenant-e2e-ipp-a-cc3e80/ipp-route-cc3e80-kserve-5f76b6665f-dlztz" Jul 29 19:26:13.536964 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:26:13.536865 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b60018dc-b127-4100-8452-4087fd850a27-tmp-dir\") pod \"ipp-route-cc3e80-kserve-5f76b6665f-dlztz\" (UID: \"b60018dc-b127-4100-8452-4087fd850a27\") " pod="ai-tenant-e2e-ipp-a-cc3e80/ipp-route-cc3e80-kserve-5f76b6665f-dlztz" Jul 29 19:26:13.536964 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:26:13.536891 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b60018dc-b127-4100-8452-4087fd850a27-home\") pod \"ipp-route-cc3e80-kserve-5f76b6665f-dlztz\" (UID: \"b60018dc-b127-4100-8452-4087fd850a27\") " pod="ai-tenant-e2e-ipp-a-cc3e80/ipp-route-cc3e80-kserve-5f76b6665f-dlztz" Jul 29 19:26:13.536964 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:26:13.536914 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b60018dc-b127-4100-8452-4087fd850a27-dshm\") pod \"ipp-route-cc3e80-kserve-5f76b6665f-dlztz\" (UID: \"b60018dc-b127-4100-8452-4087fd850a27\") " pod="ai-tenant-e2e-ipp-a-cc3e80/ipp-route-cc3e80-kserve-5f76b6665f-dlztz" Jul 29 19:26:13.536964 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:26:13.536936 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b60018dc-b127-4100-8452-4087fd850a27-model-cache\") pod \"ipp-route-cc3e80-kserve-5f76b6665f-dlztz\" (UID: \"b60018dc-b127-4100-8452-4087fd850a27\") " pod="ai-tenant-e2e-ipp-a-cc3e80/ipp-route-cc3e80-kserve-5f76b6665f-dlztz" Jul 29 19:26:13.537321 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:26:13.537297 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b60018dc-b127-4100-8452-4087fd850a27-home\") pod \"ipp-route-cc3e80-kserve-5f76b6665f-dlztz\" (UID: \"b60018dc-b127-4100-8452-4087fd850a27\") " pod="ai-tenant-e2e-ipp-a-cc3e80/ipp-route-cc3e80-kserve-5f76b6665f-dlztz" Jul 29 19:26:13.537321 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:26:13.537311 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b60018dc-b127-4100-8452-4087fd850a27-model-cache\") pod \"ipp-route-cc3e80-kserve-5f76b6665f-dlztz\" (UID: \"b60018dc-b127-4100-8452-4087fd850a27\") " pod="ai-tenant-e2e-ipp-a-cc3e80/ipp-route-cc3e80-kserve-5f76b6665f-dlztz" Jul 29 19:26:13.537485 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:26:13.537406 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b60018dc-b127-4100-8452-4087fd850a27-tmp-dir\") pod \"ipp-route-cc3e80-kserve-5f76b6665f-dlztz\" (UID: \"b60018dc-b127-4100-8452-4087fd850a27\") " pod="ai-tenant-e2e-ipp-a-cc3e80/ipp-route-cc3e80-kserve-5f76b6665f-dlztz" Jul 29 19:26:13.539445 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:26:13.539426 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b60018dc-b127-4100-8452-4087fd850a27-dshm\") pod \"ipp-route-cc3e80-kserve-5f76b6665f-dlztz\" (UID: \"b60018dc-b127-4100-8452-4087fd850a27\") " pod="ai-tenant-e2e-ipp-a-cc3e80/ipp-route-cc3e80-kserve-5f76b6665f-dlztz" Jul 29 19:26:13.539510 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:26:13.539466 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b60018dc-b127-4100-8452-4087fd850a27-tls-certs\") pod \"ipp-route-cc3e80-kserve-5f76b6665f-dlztz\" (UID: \"b60018dc-b127-4100-8452-4087fd850a27\") " pod="ai-tenant-e2e-ipp-a-cc3e80/ipp-route-cc3e80-kserve-5f76b6665f-dlztz" Jul 29 19:26:13.544818 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:26:13.544798 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7458m\" (UniqueName: \"kubernetes.io/projected/b60018dc-b127-4100-8452-4087fd850a27-kube-api-access-7458m\") pod \"ipp-route-cc3e80-kserve-5f76b6665f-dlztz\" (UID: \"b60018dc-b127-4100-8452-4087fd850a27\") " pod="ai-tenant-e2e-ipp-a-cc3e80/ipp-route-cc3e80-kserve-5f76b6665f-dlztz" Jul 29 19:26:13.697019 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:26:13.696939 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-cc3e80/ipp-route-cc3e80-kserve-5f76b6665f-dlztz" Jul 29 19:26:13.834704 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:26:13.834681 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-cc3e80/ipp-route-cc3e80-kserve-5f76b6665f-dlztz"] Jul 29 19:26:13.836059 ip-10-0-140-205 kubenswrapper[2560]: W0729 19:26:13.836032 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb60018dc_b127_4100_8452_4087fd850a27.slice/crio-35d0effee0f3281eb1f0369d0572ea8cb195132fa7d8e32fa85420e38b2beef0 WatchSource:0}: Error finding container 35d0effee0f3281eb1f0369d0572ea8cb195132fa7d8e32fa85420e38b2beef0: Status 404 returned error can't find the container with id 35d0effee0f3281eb1f0369d0572ea8cb195132fa7d8e32fa85420e38b2beef0 Jul 29 19:26:13.977145 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:26:13.977053 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-cc3e80/ipp-route-cc3e80-kserve-5f76b6665f-dlztz" event={"ID":"b60018dc-b127-4100-8452-4087fd850a27","Type":"ContainerStarted","Data":"626214e209e087ff986c3e61d61fdb2004949470357e57bbe864d5762e37dc31"} Jul 29 19:26:13.977145 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:26:13.977098 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-cc3e80/ipp-route-cc3e80-kserve-5f76b6665f-dlztz" event={"ID":"b60018dc-b127-4100-8452-4087fd850a27","Type":"ContainerStarted","Data":"35d0effee0f3281eb1f0369d0572ea8cb195132fa7d8e32fa85420e38b2beef0"} Jul 29 19:26:13.996223 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:26:13.996175 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-cc3e80/ipp-route-cc3e80-kserve-5f76b6665f-dlztz" podStartSLOduration=0.996162776 podStartE2EDuration="996.162776ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:26:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:26:13.994071292 +0000 UTC m=+3109.219740164" watchObservedRunningTime="2026-07-29 19:26:13.996162776 +0000 UTC m=+3109.221831648" Jul 29 19:26:23.697520 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:26:23.697483 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-cc3e80/ipp-route-cc3e80-kserve-5f76b6665f-dlztz" Jul 29 19:26:23.697520 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:26:23.697530 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-cc3e80/ipp-route-cc3e80-kserve-5f76b6665f-dlztz" Jul 29 19:26:23.710275 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:26:23.710244 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-cc3e80/ipp-route-cc3e80-kserve-5f76b6665f-dlztz" Jul 29 19:26:24.050009 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:26:24.049984 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-cc3e80/ipp-route-cc3e80-kserve-5f76b6665f-dlztz" Jul 29 19:26:38.200279 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:26:38.200243 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-d5kl9"] Jul 29 19:26:38.203380 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:26:38.203365 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-d5kl9" Jul 29 19:26:38.207192 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:26:38.207170 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-spkc9\"" Jul 29 19:26:38.217428 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:26:38.217409 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-d5kl9"] Jul 29 19:26:38.259199 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:26:38.259174 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6p2s\" (UniqueName: \"kubernetes.io/projected/243147ee-d535-4e6a-973f-5c1154b4dff0-kube-api-access-b6p2s\") pod \"kuadrant-operator-catalog-d5kl9\" (UID: \"243147ee-d535-4e6a-973f-5c1154b4dff0\") " pod="kuadrant-system/kuadrant-operator-catalog-d5kl9" Jul 29 19:26:38.359694 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:26:38.359658 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6p2s\" (UniqueName: \"kubernetes.io/projected/243147ee-d535-4e6a-973f-5c1154b4dff0-kube-api-access-b6p2s\") pod \"kuadrant-operator-catalog-d5kl9\" (UID: \"243147ee-d535-4e6a-973f-5c1154b4dff0\") " pod="kuadrant-system/kuadrant-operator-catalog-d5kl9" Jul 29 19:26:38.369210 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:26:38.369180 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6p2s\" (UniqueName: \"kubernetes.io/projected/243147ee-d535-4e6a-973f-5c1154b4dff0-kube-api-access-b6p2s\") pod \"kuadrant-operator-catalog-d5kl9\" (UID: \"243147ee-d535-4e6a-973f-5c1154b4dff0\") " pod="kuadrant-system/kuadrant-operator-catalog-d5kl9" Jul 29 19:26:38.514038 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:26:38.514006 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-d5kl9" Jul 29 19:26:38.648795 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:26:38.648764 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-d5kl9"] Jul 29 19:26:38.650399 ip-10-0-140-205 kubenswrapper[2560]: W0729 19:26:38.650374 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod243147ee_d535_4e6a_973f_5c1154b4dff0.slice/crio-1c8c00f3a81013c7437d468e0c347df78b933ff1fc53dc3e1a2f09a47a676bc6 WatchSource:0}: Error finding container 1c8c00f3a81013c7437d468e0c347df78b933ff1fc53dc3e1a2f09a47a676bc6: Status 404 returned error can't find the container with id 1c8c00f3a81013c7437d468e0c347df78b933ff1fc53dc3e1a2f09a47a676bc6 Jul 29 19:26:39.128766 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:26:39.128729 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-d5kl9" event={"ID":"243147ee-d535-4e6a-973f-5c1154b4dff0","Type":"ContainerStarted","Data":"1c8c00f3a81013c7437d468e0c347df78b933ff1fc53dc3e1a2f09a47a676bc6"} Jul 29 19:26:41.143416 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:26:41.143376 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-d5kl9" event={"ID":"243147ee-d535-4e6a-973f-5c1154b4dff0","Type":"ContainerStarted","Data":"0824bb58fd94e5e9706476fd4a1a87984b127a7dcbe76f0d4fffe7e0f97e6b47"} Jul 29 19:26:41.159997 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:26:41.159940 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-d5kl9" podStartSLOduration=0.859990356 podStartE2EDuration="3.159925084s" totalImagesPullingTime="2.299934728s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:26:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:26:41.158700359 +0000 UTC m=+3136.384369233" watchObservedRunningTime="2026-07-29 19:26:41.159925084 +0000 UTC m=+3136.385593956" Jul 29 19:26:48.514186 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:26:48.514101 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-d5kl9" Jul 29 19:26:48.514186 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:26:48.514176 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-d5kl9" Jul 29 19:26:48.536165 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:26:48.536139 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-d5kl9" Jul 29 19:26:49.210043 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:26:49.210012 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-d5kl9" Jul 29 19:26:49.249454 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:26:49.249416 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-d5kl9"] Jul 29 19:26:51.199539 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:26:51.199482 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-d5kl9" podUID="243147ee-d535-4e6a-973f-5c1154b4dff0" containerName="registry-server" containerID="cri-o://0824bb58fd94e5e9706476fd4a1a87984b127a7dcbe76f0d4fffe7e0f97e6b47" gracePeriod=2 Jul 29 19:26:51.457948 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:26:51.457885 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-d5kl9" Jul 29 19:26:51.578202 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:26:51.578165 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b6p2s\" (UniqueName: \"kubernetes.io/projected/243147ee-d535-4e6a-973f-5c1154b4dff0-kube-api-access-b6p2s\") pod \"243147ee-d535-4e6a-973f-5c1154b4dff0\" (UID: \"243147ee-d535-4e6a-973f-5c1154b4dff0\") " Jul 29 19:26:51.580472 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:26:51.580444 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/243147ee-d535-4e6a-973f-5c1154b4dff0-kube-api-access-b6p2s" (OuterVolumeSpecName: "kube-api-access-b6p2s") pod "243147ee-d535-4e6a-973f-5c1154b4dff0" (UID: "243147ee-d535-4e6a-973f-5c1154b4dff0"). InnerVolumeSpecName "kube-api-access-b6p2s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 19:26:51.679676 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:26:51.679648 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b6p2s\" (UniqueName: \"kubernetes.io/projected/243147ee-d535-4e6a-973f-5c1154b4dff0-kube-api-access-b6p2s\") on node \"ip-10-0-140-205.ec2.internal\" DevicePath \"\"" Jul 29 19:26:52.206395 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:26:52.206335 2560 generic.go:356] "Generic (PLEG): container finished" podID="243147ee-d535-4e6a-973f-5c1154b4dff0" containerID="0824bb58fd94e5e9706476fd4a1a87984b127a7dcbe76f0d4fffe7e0f97e6b47" exitCode=0 Jul 29 19:26:52.206837 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:26:52.206392 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-d5kl9" event={"ID":"243147ee-d535-4e6a-973f-5c1154b4dff0","Type":"ContainerDied","Data":"0824bb58fd94e5e9706476fd4a1a87984b127a7dcbe76f0d4fffe7e0f97e6b47"} Jul 29 19:26:52.206837 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:26:52.206413 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-d5kl9" Jul 29 19:26:52.206837 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:26:52.206459 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-d5kl9" event={"ID":"243147ee-d535-4e6a-973f-5c1154b4dff0","Type":"ContainerDied","Data":"1c8c00f3a81013c7437d468e0c347df78b933ff1fc53dc3e1a2f09a47a676bc6"} Jul 29 19:26:52.206837 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:26:52.206487 2560 scope.go:117] "RemoveContainer" containerID="0824bb58fd94e5e9706476fd4a1a87984b127a7dcbe76f0d4fffe7e0f97e6b47" Jul 29 19:26:52.217452 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:26:52.217425 2560 scope.go:117] "RemoveContainer" containerID="0824bb58fd94e5e9706476fd4a1a87984b127a7dcbe76f0d4fffe7e0f97e6b47" Jul 29 19:26:52.217796 ip-10-0-140-205 kubenswrapper[2560]: E0729 19:26:52.217775 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0824bb58fd94e5e9706476fd4a1a87984b127a7dcbe76f0d4fffe7e0f97e6b47\": container with ID starting with 0824bb58fd94e5e9706476fd4a1a87984b127a7dcbe76f0d4fffe7e0f97e6b47 not found: ID does not exist" containerID="0824bb58fd94e5e9706476fd4a1a87984b127a7dcbe76f0d4fffe7e0f97e6b47" Jul 29 19:26:52.217871 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:26:52.217809 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0824bb58fd94e5e9706476fd4a1a87984b127a7dcbe76f0d4fffe7e0f97e6b47"} err="failed to get container status \"0824bb58fd94e5e9706476fd4a1a87984b127a7dcbe76f0d4fffe7e0f97e6b47\": rpc error: code = NotFound desc = could not find container \"0824bb58fd94e5e9706476fd4a1a87984b127a7dcbe76f0d4fffe7e0f97e6b47\": container with ID starting with 0824bb58fd94e5e9706476fd4a1a87984b127a7dcbe76f0d4fffe7e0f97e6b47 not found: ID does not exist" Jul 29 19:26:52.229170 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:26:52.229141 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-d5kl9"] Jul 29 19:26:52.231500 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:26:52.231475 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-d5kl9"] Jul 29 19:26:53.354062 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:26:53.354029 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="243147ee-d535-4e6a-973f-5c1154b4dff0" path="/var/lib/kubelet/pods/243147ee-d535-4e6a-973f-5c1154b4dff0/volumes" Jul 29 19:26:59.756879 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:26:59.756840 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:27:00.056925 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:27:00.056847 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-21dbec-openshift-default-849976d68b-brcb4"] Jul 29 19:27:00.057270 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:27:00.057256 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="243147ee-d535-4e6a-973f-5c1154b4dff0" containerName="registry-server" Jul 29 19:27:00.057313 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:27:00.057273 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="243147ee-d535-4e6a-973f-5c1154b4dff0" containerName="registry-server" Jul 29 19:27:00.057401 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:27:00.057338 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="243147ee-d535-4e6a-973f-5c1154b4dff0" containerName="registry-server" Jul 29 19:27:00.061961 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:27:00.061944 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-21dbec-openshift-default-849976d68b-brcb4" Jul 29 19:27:00.064406 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:27:00.064389 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-cleanup-21dbec-openshift-default-dockercfg-pwk9w\"" Jul 29 19:27:00.072863 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:27:00.072844 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-21dbec-openshift-default-849976d68b-brcb4"] Jul 29 19:27:00.082090 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:27:00.082066 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:27:00.157330 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:27:00.157296 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0552b317-9b29-4382-bfa6-cd6bc13b519e-istio-podinfo\") pod \"e2e-ipp-cleanup-21dbec-openshift-default-849976d68b-brcb4\" (UID: \"0552b317-9b29-4382-bfa6-cd6bc13b519e\") " pod="openshift-ingress/e2e-ipp-cleanup-21dbec-openshift-default-849976d68b-brcb4" Jul 29 19:27:00.157330 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:27:00.157337 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0552b317-9b29-4382-bfa6-cd6bc13b519e-workload-certs\") pod \"e2e-ipp-cleanup-21dbec-openshift-default-849976d68b-brcb4\" (UID: \"0552b317-9b29-4382-bfa6-cd6bc13b519e\") " pod="openshift-ingress/e2e-ipp-cleanup-21dbec-openshift-default-849976d68b-brcb4" Jul 29 19:27:00.157330 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:27:00.157429 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0552b317-9b29-4382-bfa6-cd6bc13b519e-istio-envoy\") pod \"e2e-ipp-cleanup-21dbec-openshift-default-849976d68b-brcb4\" (UID: \"0552b317-9b29-4382-bfa6-cd6bc13b519e\") " pod="openshift-ingress/e2e-ipp-cleanup-21dbec-openshift-default-849976d68b-brcb4" Jul 29 19:27:00.157710 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:27:00.157466 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0552b317-9b29-4382-bfa6-cd6bc13b519e-istiod-ca-cert\") pod \"e2e-ipp-cleanup-21dbec-openshift-default-849976d68b-brcb4\" (UID: \"0552b317-9b29-4382-bfa6-cd6bc13b519e\") " pod="openshift-ingress/e2e-ipp-cleanup-21dbec-openshift-default-849976d68b-brcb4" Jul 29 19:27:00.157710 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:27:00.157500 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0552b317-9b29-4382-bfa6-cd6bc13b519e-workload-socket\") pod \"e2e-ipp-cleanup-21dbec-openshift-default-849976d68b-brcb4\" (UID: \"0552b317-9b29-4382-bfa6-cd6bc13b519e\") " pod="openshift-ingress/e2e-ipp-cleanup-21dbec-openshift-default-849976d68b-brcb4" Jul 29 19:27:00.157710 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:27:00.157540 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0552b317-9b29-4382-bfa6-cd6bc13b519e-istio-data\") pod \"e2e-ipp-cleanup-21dbec-openshift-default-849976d68b-brcb4\" (UID: \"0552b317-9b29-4382-bfa6-cd6bc13b519e\") " pod="openshift-ingress/e2e-ipp-cleanup-21dbec-openshift-default-849976d68b-brcb4" Jul 29 19:27:00.157710 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:27:00.157561 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhzp5\" (UniqueName: \"kubernetes.io/projected/0552b317-9b29-4382-bfa6-cd6bc13b519e-kube-api-access-qhzp5\") pod \"e2e-ipp-cleanup-21dbec-openshift-default-849976d68b-brcb4\" (UID: \"0552b317-9b29-4382-bfa6-cd6bc13b519e\") " pod="openshift-ingress/e2e-ipp-cleanup-21dbec-openshift-default-849976d68b-brcb4" Jul 29 19:27:00.157710 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:27:00.157579 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0552b317-9b29-4382-bfa6-cd6bc13b519e-istio-token\") pod \"e2e-ipp-cleanup-21dbec-openshift-default-849976d68b-brcb4\" (UID: \"0552b317-9b29-4382-bfa6-cd6bc13b519e\") " pod="openshift-ingress/e2e-ipp-cleanup-21dbec-openshift-default-849976d68b-brcb4" Jul 29 19:27:00.157710 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:27:00.157626 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0552b317-9b29-4382-bfa6-cd6bc13b519e-credential-socket\") pod \"e2e-ipp-cleanup-21dbec-openshift-default-849976d68b-brcb4\" (UID: \"0552b317-9b29-4382-bfa6-cd6bc13b519e\") " pod="openshift-ingress/e2e-ipp-cleanup-21dbec-openshift-default-849976d68b-brcb4" Jul 29 19:27:00.258852 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:27:00.258815 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0552b317-9b29-4382-bfa6-cd6bc13b519e-istio-data\") pod \"e2e-ipp-cleanup-21dbec-openshift-default-849976d68b-brcb4\" (UID: \"0552b317-9b29-4382-bfa6-cd6bc13b519e\") " pod="openshift-ingress/e2e-ipp-cleanup-21dbec-openshift-default-849976d68b-brcb4" Jul 29 19:27:00.258852 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:27:00.258847 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhzp5\" (UniqueName: \"kubernetes.io/projected/0552b317-9b29-4382-bfa6-cd6bc13b519e-kube-api-access-qhzp5\") pod \"e2e-ipp-cleanup-21dbec-openshift-default-849976d68b-brcb4\" (UID: \"0552b317-9b29-4382-bfa6-cd6bc13b519e\") " pod="openshift-ingress/e2e-ipp-cleanup-21dbec-openshift-default-849976d68b-brcb4" Jul 29 19:27:00.259093 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:27:00.258867 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0552b317-9b29-4382-bfa6-cd6bc13b519e-istio-token\") pod \"e2e-ipp-cleanup-21dbec-openshift-default-849976d68b-brcb4\" (UID: \"0552b317-9b29-4382-bfa6-cd6bc13b519e\") " pod="openshift-ingress/e2e-ipp-cleanup-21dbec-openshift-default-849976d68b-brcb4" Jul 29 19:27:00.259093 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:27:00.258910 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0552b317-9b29-4382-bfa6-cd6bc13b519e-credential-socket\") pod \"e2e-ipp-cleanup-21dbec-openshift-default-849976d68b-brcb4\" (UID: \"0552b317-9b29-4382-bfa6-cd6bc13b519e\") " pod="openshift-ingress/e2e-ipp-cleanup-21dbec-openshift-default-849976d68b-brcb4" Jul 29 19:27:00.259093 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:27:00.258976 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0552b317-9b29-4382-bfa6-cd6bc13b519e-istio-podinfo\") pod \"e2e-ipp-cleanup-21dbec-openshift-default-849976d68b-brcb4\" (UID: \"0552b317-9b29-4382-bfa6-cd6bc13b519e\") " pod="openshift-ingress/e2e-ipp-cleanup-21dbec-openshift-default-849976d68b-brcb4" Jul 29 19:27:00.259293 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:27:00.259121 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0552b317-9b29-4382-bfa6-cd6bc13b519e-workload-certs\") pod \"e2e-ipp-cleanup-21dbec-openshift-default-849976d68b-brcb4\" (UID: \"0552b317-9b29-4382-bfa6-cd6bc13b519e\") " pod="openshift-ingress/e2e-ipp-cleanup-21dbec-openshift-default-849976d68b-brcb4" Jul 29 19:27:00.259293 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:27:00.259182 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0552b317-9b29-4382-bfa6-cd6bc13b519e-istio-envoy\") pod \"e2e-ipp-cleanup-21dbec-openshift-default-849976d68b-brcb4\" (UID: \"0552b317-9b29-4382-bfa6-cd6bc13b519e\") " pod="openshift-ingress/e2e-ipp-cleanup-21dbec-openshift-default-849976d68b-brcb4" Jul 29 19:27:00.259293 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:27:00.259210 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0552b317-9b29-4382-bfa6-cd6bc13b519e-istiod-ca-cert\") pod \"e2e-ipp-cleanup-21dbec-openshift-default-849976d68b-brcb4\" (UID: \"0552b317-9b29-4382-bfa6-cd6bc13b519e\") " pod="openshift-ingress/e2e-ipp-cleanup-21dbec-openshift-default-849976d68b-brcb4" Jul 29 19:27:00.259293 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:27:00.259259 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0552b317-9b29-4382-bfa6-cd6bc13b519e-istio-data\") pod \"e2e-ipp-cleanup-21dbec-openshift-default-849976d68b-brcb4\" (UID: \"0552b317-9b29-4382-bfa6-cd6bc13b519e\") " pod="openshift-ingress/e2e-ipp-cleanup-21dbec-openshift-default-849976d68b-brcb4" Jul 29 19:27:00.259293 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:27:00.259268 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0552b317-9b29-4382-bfa6-cd6bc13b519e-workload-socket\") pod \"e2e-ipp-cleanup-21dbec-openshift-default-849976d68b-brcb4\" (UID: \"0552b317-9b29-4382-bfa6-cd6bc13b519e\") " pod="openshift-ingress/e2e-ipp-cleanup-21dbec-openshift-default-849976d68b-brcb4" Jul 29 19:27:00.259553 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:27:00.259418 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0552b317-9b29-4382-bfa6-cd6bc13b519e-credential-socket\") pod \"e2e-ipp-cleanup-21dbec-openshift-default-849976d68b-brcb4\" (UID: \"0552b317-9b29-4382-bfa6-cd6bc13b519e\") " pod="openshift-ingress/e2e-ipp-cleanup-21dbec-openshift-default-849976d68b-brcb4" Jul 29 19:27:00.259553 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:27:00.259472 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0552b317-9b29-4382-bfa6-cd6bc13b519e-workload-certs\") pod \"e2e-ipp-cleanup-21dbec-openshift-default-849976d68b-brcb4\" (UID: \"0552b317-9b29-4382-bfa6-cd6bc13b519e\") " pod="openshift-ingress/e2e-ipp-cleanup-21dbec-openshift-default-849976d68b-brcb4" Jul 29 19:27:00.259640 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:27:00.259623 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0552b317-9b29-4382-bfa6-cd6bc13b519e-workload-socket\") pod \"e2e-ipp-cleanup-21dbec-openshift-default-849976d68b-brcb4\" (UID: \"0552b317-9b29-4382-bfa6-cd6bc13b519e\") " pod="openshift-ingress/e2e-ipp-cleanup-21dbec-openshift-default-849976d68b-brcb4" Jul 29 19:27:00.260190 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:27:00.260168 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0552b317-9b29-4382-bfa6-cd6bc13b519e-istiod-ca-cert\") pod \"e2e-ipp-cleanup-21dbec-openshift-default-849976d68b-brcb4\" (UID: \"0552b317-9b29-4382-bfa6-cd6bc13b519e\") " pod="openshift-ingress/e2e-ipp-cleanup-21dbec-openshift-default-849976d68b-brcb4" Jul 29 19:27:00.261545 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:27:00.261525 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0552b317-9b29-4382-bfa6-cd6bc13b519e-istio-envoy\") pod \"e2e-ipp-cleanup-21dbec-openshift-default-849976d68b-brcb4\" (UID: \"0552b317-9b29-4382-bfa6-cd6bc13b519e\") " pod="openshift-ingress/e2e-ipp-cleanup-21dbec-openshift-default-849976d68b-brcb4" Jul 29 19:27:00.261644 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:27:00.261587 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0552b317-9b29-4382-bfa6-cd6bc13b519e-istio-podinfo\") pod \"e2e-ipp-cleanup-21dbec-openshift-default-849976d68b-brcb4\" (UID: \"0552b317-9b29-4382-bfa6-cd6bc13b519e\") " pod="openshift-ingress/e2e-ipp-cleanup-21dbec-openshift-default-849976d68b-brcb4" Jul 29 19:27:00.272456 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:27:00.272425 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhzp5\" (UniqueName: \"kubernetes.io/projected/0552b317-9b29-4382-bfa6-cd6bc13b519e-kube-api-access-qhzp5\") pod \"e2e-ipp-cleanup-21dbec-openshift-default-849976d68b-brcb4\" (UID: \"0552b317-9b29-4382-bfa6-cd6bc13b519e\") " pod="openshift-ingress/e2e-ipp-cleanup-21dbec-openshift-default-849976d68b-brcb4" Jul 29 19:27:00.274587 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:27:00.274567 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0552b317-9b29-4382-bfa6-cd6bc13b519e-istio-token\") pod \"e2e-ipp-cleanup-21dbec-openshift-default-849976d68b-brcb4\" (UID: \"0552b317-9b29-4382-bfa6-cd6bc13b519e\") " pod="openshift-ingress/e2e-ipp-cleanup-21dbec-openshift-default-849976d68b-brcb4" Jul 29 19:27:00.372690 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:27:00.372614 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-21dbec-openshift-default-849976d68b-brcb4" Jul 29 19:27:00.440176 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:27:00.440139 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:27:00.515091 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:27:00.515069 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-21dbec-openshift-default-849976d68b-brcb4"] Jul 29 19:27:00.516756 ip-10-0-140-205 kubenswrapper[2560]: W0729 19:27:00.516729 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0552b317_9b29_4382_bfa6_cd6bc13b519e.slice/crio-af7d7783812917c06ecd5dae7a96f2a8b788fe3786366e151965233d8ac3bb2b WatchSource:0}: Error finding container af7d7783812917c06ecd5dae7a96f2a8b788fe3786366e151965233d8ac3bb2b: Status 404 returned error can't find the container with id af7d7783812917c06ecd5dae7a96f2a8b788fe3786366e151965233d8ac3bb2b Jul 29 19:27:00.518965 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:27:00.518935 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 29 19:27:00.519031 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:27:00.518999 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 29 19:27:00.519074 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:27:00.519031 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 29 19:27:01.258890 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:27:01.258850 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-21dbec-openshift-default-849976d68b-brcb4" event={"ID":"0552b317-9b29-4382-bfa6-cd6bc13b519e","Type":"ContainerStarted","Data":"85ca53f0bd8a4864c3967678e8ff2858f5d8d7570b7663b6648b11d62c77c5b6"} Jul 29 19:27:01.259265 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:27:01.258914 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-21dbec-openshift-default-849976d68b-brcb4" event={"ID":"0552b317-9b29-4382-bfa6-cd6bc13b519e","Type":"ContainerStarted","Data":"af7d7783812917c06ecd5dae7a96f2a8b788fe3786366e151965233d8ac3bb2b"} Jul 29 19:27:01.283953 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:27:01.283899 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-cleanup-21dbec-openshift-default-849976d68b-brcb4" podStartSLOduration=1.283883224 podStartE2EDuration="1.283883224s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:27:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:27:01.282340502 +0000 UTC m=+3156.508009376" watchObservedRunningTime="2026-07-29 19:27:01.283883224 +0000 UTC m=+3156.509552095" Jul 29 19:27:01.373404 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:27:01.373379 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-21dbec-openshift-default-849976d68b-brcb4" Jul 29 19:27:01.378277 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:27:01.378249 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-21dbec-openshift-default-849976d68b-brcb4" Jul 29 19:27:02.264303 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:27:02.264270 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-21dbec-openshift-default-849976d68b-brcb4" Jul 29 19:27:02.265262 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:27:02.265237 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-21dbec-openshift-default-849976d68b-brcb4" Jul 29 19:27:03.238838 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:27:03.238803 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:27:03.335876 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:27:03.335841 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:27:03.500378 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:27:03.499699 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:27:03.599528 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:27:03.599495 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:27:03.971727 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:27:03.971636 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:27:04.069399 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:27:04.069341 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:27:09.195126 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:27:09.195080 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:27:09.295413 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:27:09.295376 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:27:17.265911 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:27:17.265872 2560 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-21dbec-openshift-default-849976d68b-brcb4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Jul 29 19:27:17.266309 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:27:17.265940 2560 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-cleanup-21dbec-openshift-default-849976d68b-brcb4" podUID="0552b317-9b29-4382-bfa6-cd6bc13b519e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Jul 29 19:27:17.354341 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:27:17.354306 2560 generic.go:356] "Generic (PLEG): container finished" podID="0552b317-9b29-4382-bfa6-cd6bc13b519e" containerID="85ca53f0bd8a4864c3967678e8ff2858f5d8d7570b7663b6648b11d62c77c5b6" exitCode=0 Jul 29 19:27:17.354524 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:27:17.354501 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-21dbec-openshift-default-849976d68b-brcb4" event={"ID":"0552b317-9b29-4382-bfa6-cd6bc13b519e","Type":"ContainerDied","Data":"85ca53f0bd8a4864c3967678e8ff2858f5d8d7570b7663b6648b11d62c77c5b6"} Jul 29 19:27:17.354877 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:27:17.354864 2560 scope.go:117] "RemoveContainer" containerID="85ca53f0bd8a4864c3967678e8ff2858f5d8d7570b7663b6648b11d62c77c5b6" Jul 29 19:27:17.356056 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:27:17.356025 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 29 19:27:17.356113 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:27:17.356096 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 29 19:27:17.356151 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:27:17.356135 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 29 19:27:18.362830 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:27:18.361885 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-21dbec-openshift-default-849976d68b-brcb4" event={"ID":"0552b317-9b29-4382-bfa6-cd6bc13b519e","Type":"ContainerStarted","Data":"ecf11a340720d768dbb80db903dd0e2a729d8a0edad23fd489500a0fc63fd99b"} Jul 29 19:27:18.373303 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:27:18.373258 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-21dbec-openshift-default-849976d68b-brcb4" Jul 29 19:27:18.374100 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:27:18.374071 2560 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-21dbec-openshift-default-849976d68b-brcb4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Jul 29 19:27:18.374610 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:27:18.374281 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-21dbec-openshift-default-849976d68b-brcb4" podUID="0552b317-9b29-4382-bfa6-cd6bc13b519e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Jul 29 19:27:19.373979 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:27:19.373947 2560 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-21dbec-openshift-default-849976d68b-brcb4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Jul 29 19:27:19.374378 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:27:19.373997 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-21dbec-openshift-default-849976d68b-brcb4" podUID="0552b317-9b29-4382-bfa6-cd6bc13b519e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Jul 29 19:27:20.373050 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:27:20.373018 2560 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-21dbec-openshift-default-849976d68b-brcb4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Jul 29 19:27:20.373220 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:27:20.373082 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-21dbec-openshift-default-849976d68b-brcb4" podUID="0552b317-9b29-4382-bfa6-cd6bc13b519e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Jul 29 19:27:21.373537 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:27:21.373499 2560 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-21dbec-openshift-default-849976d68b-brcb4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Jul 29 19:27:21.373946 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:27:21.373576 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-21dbec-openshift-default-849976d68b-brcb4" podUID="0552b317-9b29-4382-bfa6-cd6bc13b519e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Jul 29 19:27:22.373567 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:27:22.373528 2560 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-21dbec-openshift-default-849976d68b-brcb4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Jul 29 19:27:22.373963 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:27:22.373597 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-21dbec-openshift-default-849976d68b-brcb4" podUID="0552b317-9b29-4382-bfa6-cd6bc13b519e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Jul 29 19:27:23.377507 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:27:23.377477 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-21dbec-openshift-default-849976d68b-brcb4" Jul 29 19:27:23.377945 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:27:23.377920 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-21dbec-openshift-default-849976d68b-brcb4" Jul 29 19:27:23.378497 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:27:23.378477 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-21dbec-openshift-default-849976d68b-brcb4" Jul 29 19:27:28.338447 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:27:28.338417 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:27:29.109542 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:27:29.109503 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-21dbec-kvwd9"] Jul 29 19:27:29.114119 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:27:29.114102 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-21dbec-kvwd9" Jul 29 19:27:29.116717 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:27:29.116688 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vgb9z\"" Jul 29 19:27:29.120119 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:27:29.120096 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-21dbec-kvwd9"] Jul 29 19:27:29.130527 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:27:29.130501 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ded1e69-22fa-481e-ac42-d9a12b2d2c0d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-21dbec-kvwd9\" (UID: \"2ded1e69-22fa-481e-ac42-d9a12b2d2c0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-21dbec-kvwd9" Jul 29 19:27:29.231107 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:27:29.231066 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ded1e69-22fa-481e-ac42-d9a12b2d2c0d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-21dbec-kvwd9\" (UID: \"2ded1e69-22fa-481e-ac42-d9a12b2d2c0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-21dbec-kvwd9" Jul 29 19:27:29.231673 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:27:29.231654 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ded1e69-22fa-481e-ac42-d9a12b2d2c0d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-21dbec-kvwd9\" (UID: \"2ded1e69-22fa-481e-ac42-d9a12b2d2c0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-21dbec-kvwd9" Jul 29 19:27:29.427066 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:27:29.426997 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-21dbec-kvwd9" Jul 29 19:27:29.562922 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:27:29.562898 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-21dbec-kvwd9"] Jul 29 19:27:29.564811 ip-10-0-140-205 kubenswrapper[2560]: W0729 19:27:29.564783 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ded1e69_22fa_481e_ac42_d9a12b2d2c0d.slice/crio-f2ef3ac5ed6136a54f8ceec20f74fdd5ee342fa9d1e28ceea51e0ba9ef604571 WatchSource:0}: Error finding container f2ef3ac5ed6136a54f8ceec20f74fdd5ee342fa9d1e28ceea51e0ba9ef604571: Status 404 returned error can't find the container with id f2ef3ac5ed6136a54f8ceec20f74fdd5ee342fa9d1e28ceea51e0ba9ef604571 Jul 29 19:27:30.427949 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:27:30.427859 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-21dbec-kvwd9" event={"ID":"2ded1e69-22fa-481e-ac42-d9a12b2d2c0d","Type":"ContainerDied","Data":"c51eae4a23905c65ee42571d3149168a2417410f7fe62977bffd0d0f8def55c9"} Jul 29 19:27:30.428389 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:27:30.427945 2560 generic.go:356] "Generic (PLEG): container finished" podID="2ded1e69-22fa-481e-ac42-d9a12b2d2c0d" containerID="c51eae4a23905c65ee42571d3149168a2417410f7fe62977bffd0d0f8def55c9" exitCode=0 Jul 29 19:27:30.428546 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:27:30.428524 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-21dbec-kvwd9" event={"ID":"2ded1e69-22fa-481e-ac42-d9a12b2d2c0d","Type":"ContainerStarted","Data":"f2ef3ac5ed6136a54f8ceec20f74fdd5ee342fa9d1e28ceea51e0ba9ef604571"} Jul 29 19:27:31.579732 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:27:31.579706 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-21dbec-kvwd9" Jul 29 19:27:31.654790 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:27:31.654758 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ded1e69-22fa-481e-ac42-d9a12b2d2c0d-maas-api-service-ca\") pod \"2ded1e69-22fa-481e-ac42-d9a12b2d2c0d\" (UID: \"2ded1e69-22fa-481e-ac42-d9a12b2d2c0d\") " Jul 29 19:27:31.655102 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:27:31.655081 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ded1e69-22fa-481e-ac42-d9a12b2d2c0d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "2ded1e69-22fa-481e-ac42-d9a12b2d2c0d" (UID: "2ded1e69-22fa-481e-ac42-d9a12b2d2c0d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 19:27:31.756202 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:27:31.756180 2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ded1e69-22fa-481e-ac42-d9a12b2d2c0d-maas-api-service-ca\") on node \"ip-10-0-140-205.ec2.internal\" DevicePath \"\"" Jul 29 19:27:32.440921 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:27:32.440891 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-21dbec-kvwd9" Jul 29 19:27:32.441105 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:27:32.440896 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-21dbec-kvwd9" event={"ID":"2ded1e69-22fa-481e-ac42-d9a12b2d2c0d","Type":"ContainerDied","Data":"f2ef3ac5ed6136a54f8ceec20f74fdd5ee342fa9d1e28ceea51e0ba9ef604571"} Jul 29 19:27:32.441105 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:27:32.441017 2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f2ef3ac5ed6136a54f8ceec20f74fdd5ee342fa9d1e28ceea51e0ba9ef604571" Jul 29 19:27:33.818479 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:27:33.818447 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:27:33.918645 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:27:33.918611 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:27:34.249236 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:27:34.249206 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:27:34.330914 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:27:34.330875 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:27:34.427489 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:27:34.427452 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:27:34.432931 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:27:34.432904 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-21dbec-kvwd9"] Jul 29 19:27:34.437659 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:27:34.437637 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-21dbec-kvwd9"] Jul 29 19:27:34.514369 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:27:34.514269 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:27:34.831848 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:27:34.831756 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:27:35.353879 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:27:35.353852 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2ded1e69-22fa-481e-ac42-d9a12b2d2c0d" path="/var/lib/kubelet/pods/2ded1e69-22fa-481e-ac42-d9a12b2d2c0d/volumes" Jul 29 19:27:41.076820 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:27:41.076770 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:27:41.120166 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:27:41.120135 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-21dbec-openshift-default-849976d68b-brcb4"] Jul 29 19:27:41.120412 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:27:41.120388 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-cleanup-21dbec-openshift-default-849976d68b-brcb4" podUID="0552b317-9b29-4382-bfa6-cd6bc13b519e" containerName="istio-proxy" containerID="cri-o://ecf11a340720d768dbb80db903dd0e2a729d8a0edad23fd489500a0fc63fd99b" gracePeriod=30 Jul 29 19:27:41.147869 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:27:41.147842 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:27:41.610193 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:27:41.610158 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:27:45.101281 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:27:45.101237 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-cc3e80-rt2p5"] Jul 29 19:27:45.101853 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:27:45.101832 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2ded1e69-22fa-481e-ac42-d9a12b2d2c0d" containerName="revoke-keys" Jul 29 19:27:45.101950 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:27:45.101855 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="2ded1e69-22fa-481e-ac42-d9a12b2d2c0d" containerName="revoke-keys" Jul 29 19:27:45.102006 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:27:45.101982 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="2ded1e69-22fa-481e-ac42-d9a12b2d2c0d" containerName="revoke-keys" Jul 29 19:27:45.105186 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:27:45.105166 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-cc3e80-rt2p5" Jul 29 19:27:45.107551 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:27:45.107523 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vgb9z\"" Jul 29 19:27:45.111661 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:27:45.111637 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-cc3e80-rt2p5"] Jul 29 19:27:45.182535 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:27:45.182495 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0d43bce-5cc1-458c-bea1-8db6e2848f06-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-cc3e80-rt2p5\" (UID: \"d0d43bce-5cc1-458c-bea1-8db6e2848f06\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-cc3e80-rt2p5" Jul 29 19:27:45.201364 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:27:45.201318 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:27:45.283879 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:27:45.283848 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0d43bce-5cc1-458c-bea1-8db6e2848f06-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-cc3e80-rt2p5\" (UID: \"d0d43bce-5cc1-458c-bea1-8db6e2848f06\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-cc3e80-rt2p5" Jul 29 19:27:45.284487 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:27:45.284468 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0d43bce-5cc1-458c-bea1-8db6e2848f06-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-cc3e80-rt2p5\" (UID: \"d0d43bce-5cc1-458c-bea1-8db6e2848f06\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-cc3e80-rt2p5" Jul 29 19:27:45.417291 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:27:45.417206 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-cc3e80-rt2p5" Jul 29 19:27:45.783190 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:27:45.783117 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-cc3e80-rt2p5"] Jul 29 19:27:45.788064 ip-10-0-140-205 kubenswrapper[2560]: W0729 19:27:45.788031 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0d43bce_5cc1_458c_bea1_8db6e2848f06.slice/crio-1c8ec1f859d760259ee9d7880cd33502d6ee94caf0d8b2f1083527cab5cbd333 WatchSource:0}: Error finding container 1c8ec1f859d760259ee9d7880cd33502d6ee94caf0d8b2f1083527cab5cbd333: Status 404 returned error can't find the container with id 1c8ec1f859d760259ee9d7880cd33502d6ee94caf0d8b2f1083527cab5cbd333 Jul 29 19:27:46.364373 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:27:46.364326 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-21dbec-openshift-default-849976d68b-brcb4" Jul 29 19:27:46.496138 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:27:46.496107 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0552b317-9b29-4382-bfa6-cd6bc13b519e-istio-podinfo\") pod \"0552b317-9b29-4382-bfa6-cd6bc13b519e\" (UID: \"0552b317-9b29-4382-bfa6-cd6bc13b519e\") " Jul 29 19:27:46.496290 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:27:46.496147 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0552b317-9b29-4382-bfa6-cd6bc13b519e-workload-socket\") pod \"0552b317-9b29-4382-bfa6-cd6bc13b519e\" (UID: \"0552b317-9b29-4382-bfa6-cd6bc13b519e\") " Jul 29 19:27:46.496290 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:27:46.496168 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qhzp5\" (UniqueName: \"kubernetes.io/projected/0552b317-9b29-4382-bfa6-cd6bc13b519e-kube-api-access-qhzp5\") pod \"0552b317-9b29-4382-bfa6-cd6bc13b519e\" (UID: \"0552b317-9b29-4382-bfa6-cd6bc13b519e\") " Jul 29 19:27:46.496290 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:27:46.496195 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0552b317-9b29-4382-bfa6-cd6bc13b519e-workload-certs\") pod \"0552b317-9b29-4382-bfa6-cd6bc13b519e\" (UID: \"0552b317-9b29-4382-bfa6-cd6bc13b519e\") " Jul 29 19:27:46.496290 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:27:46.496220 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0552b317-9b29-4382-bfa6-cd6bc13b519e-istio-envoy\") pod \"0552b317-9b29-4382-bfa6-cd6bc13b519e\" (UID: \"0552b317-9b29-4382-bfa6-cd6bc13b519e\") " Jul 29 19:27:46.496290 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:27:46.496234 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0552b317-9b29-4382-bfa6-cd6bc13b519e-istio-data\") pod \"0552b317-9b29-4382-bfa6-cd6bc13b519e\" (UID: \"0552b317-9b29-4382-bfa6-cd6bc13b519e\") " Jul 29 19:27:46.496290 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:27:46.496254 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0552b317-9b29-4382-bfa6-cd6bc13b519e-istiod-ca-cert\") pod \"0552b317-9b29-4382-bfa6-cd6bc13b519e\" (UID: \"0552b317-9b29-4382-bfa6-cd6bc13b519e\") " Jul 29 19:27:46.496290 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:27:46.496280 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0552b317-9b29-4382-bfa6-cd6bc13b519e-credential-socket\") pod \"0552b317-9b29-4382-bfa6-cd6bc13b519e\" (UID: \"0552b317-9b29-4382-bfa6-cd6bc13b519e\") " Jul 29 19:27:46.496706 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:27:46.496402 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0552b317-9b29-4382-bfa6-cd6bc13b519e-istio-token\") pod \"0552b317-9b29-4382-bfa6-cd6bc13b519e\" (UID: \"0552b317-9b29-4382-bfa6-cd6bc13b519e\") " Jul 29 19:27:46.496706 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:27:46.496476 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0552b317-9b29-4382-bfa6-cd6bc13b519e-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "0552b317-9b29-4382-bfa6-cd6bc13b519e" (UID: "0552b317-9b29-4382-bfa6-cd6bc13b519e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 19:27:46.496706 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:27:46.496683 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0552b317-9b29-4382-bfa6-cd6bc13b519e-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "0552b317-9b29-4382-bfa6-cd6bc13b519e" (UID: "0552b317-9b29-4382-bfa6-cd6bc13b519e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 19:27:46.496865 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:27:46.496731 2560 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0552b317-9b29-4382-bfa6-cd6bc13b519e-workload-socket\") on node \"ip-10-0-140-205.ec2.internal\" DevicePath \"\"" Jul 29 19:27:46.496865 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:27:46.496740 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0552b317-9b29-4382-bfa6-cd6bc13b519e-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "0552b317-9b29-4382-bfa6-cd6bc13b519e" (UID: "0552b317-9b29-4382-bfa6-cd6bc13b519e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 19:27:46.496974 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:27:46.496907 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0552b317-9b29-4382-bfa6-cd6bc13b519e-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "0552b317-9b29-4382-bfa6-cd6bc13b519e" (UID: "0552b317-9b29-4382-bfa6-cd6bc13b519e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 19:27:46.497310 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:27:46.497285 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0552b317-9b29-4382-bfa6-cd6bc13b519e-istio-data" (OuterVolumeSpecName: "istio-data") pod "0552b317-9b29-4382-bfa6-cd6bc13b519e" (UID: "0552b317-9b29-4382-bfa6-cd6bc13b519e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 19:27:46.498772 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:27:46.498741 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0552b317-9b29-4382-bfa6-cd6bc13b519e-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "0552b317-9b29-4382-bfa6-cd6bc13b519e" (UID: "0552b317-9b29-4382-bfa6-cd6bc13b519e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 19:27:46.498851 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:27:46.498746 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0552b317-9b29-4382-bfa6-cd6bc13b519e-istio-token" (OuterVolumeSpecName: "istio-token") pod "0552b317-9b29-4382-bfa6-cd6bc13b519e" (UID: "0552b317-9b29-4382-bfa6-cd6bc13b519e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 19:27:46.498954 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:27:46.498931 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0552b317-9b29-4382-bfa6-cd6bc13b519e-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "0552b317-9b29-4382-bfa6-cd6bc13b519e" (UID: "0552b317-9b29-4382-bfa6-cd6bc13b519e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 29 19:27:46.498999 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:27:46.498941 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0552b317-9b29-4382-bfa6-cd6bc13b519e-kube-api-access-qhzp5" (OuterVolumeSpecName: "kube-api-access-qhzp5") pod "0552b317-9b29-4382-bfa6-cd6bc13b519e" (UID: "0552b317-9b29-4382-bfa6-cd6bc13b519e"). InnerVolumeSpecName "kube-api-access-qhzp5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 19:27:46.520117 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:27:46.520089 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-cc3e80-rt2p5" event={"ID":"d0d43bce-5cc1-458c-bea1-8db6e2848f06","Type":"ContainerStarted","Data":"29e3bb5a9c4028c4157bc600298a3dc8d9c138a2cb28509710b657bb0fed4042"} Jul 29 19:27:46.520220 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:27:46.520167 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-cc3e80-rt2p5" event={"ID":"d0d43bce-5cc1-458c-bea1-8db6e2848f06","Type":"ContainerStarted","Data":"1c8ec1f859d760259ee9d7880cd33502d6ee94caf0d8b2f1083527cab5cbd333"} Jul 29 19:27:46.521668 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:27:46.521646 2560 generic.go:356] "Generic (PLEG): container finished" podID="0552b317-9b29-4382-bfa6-cd6bc13b519e" containerID="ecf11a340720d768dbb80db903dd0e2a729d8a0edad23fd489500a0fc63fd99b" exitCode=0 Jul 29 19:27:46.521766 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:27:46.521702 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-21dbec-openshift-default-849976d68b-brcb4" Jul 29 19:27:46.521766 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:27:46.521709 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-21dbec-openshift-default-849976d68b-brcb4" event={"ID":"0552b317-9b29-4382-bfa6-cd6bc13b519e","Type":"ContainerDied","Data":"ecf11a340720d768dbb80db903dd0e2a729d8a0edad23fd489500a0fc63fd99b"} Jul 29 19:27:46.521766 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:27:46.521757 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-21dbec-openshift-default-849976d68b-brcb4" event={"ID":"0552b317-9b29-4382-bfa6-cd6bc13b519e","Type":"ContainerDied","Data":"af7d7783812917c06ecd5dae7a96f2a8b788fe3786366e151965233d8ac3bb2b"} Jul 29 19:27:46.521933 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:27:46.521780 2560 scope.go:117] "RemoveContainer" containerID="ecf11a340720d768dbb80db903dd0e2a729d8a0edad23fd489500a0fc63fd99b" Jul 29 19:27:46.531639 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:27:46.531622 2560 scope.go:117] "RemoveContainer" containerID="85ca53f0bd8a4864c3967678e8ff2858f5d8d7570b7663b6648b11d62c77c5b6" Jul 29 19:27:46.538081 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:27:46.538037 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-cc3e80-rt2p5" podStartSLOduration=1.538024383 podStartE2EDuration="1.538024383s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:27:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:27:46.536049561 +0000 UTC m=+3201.761718472" watchObservedRunningTime="2026-07-29 19:27:46.538024383 +0000 UTC m=+3201.763693318" Jul 29 19:27:46.541056 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:27:46.541038 2560 scope.go:117] "RemoveContainer" containerID="ecf11a340720d768dbb80db903dd0e2a729d8a0edad23fd489500a0fc63fd99b" Jul 29 19:27:46.541304 ip-10-0-140-205 kubenswrapper[2560]: E0729 19:27:46.541286 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ecf11a340720d768dbb80db903dd0e2a729d8a0edad23fd489500a0fc63fd99b\": container with ID starting with ecf11a340720d768dbb80db903dd0e2a729d8a0edad23fd489500a0fc63fd99b not found: ID does not exist" containerID="ecf11a340720d768dbb80db903dd0e2a729d8a0edad23fd489500a0fc63fd99b" Jul 29 19:27:46.541378 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:27:46.541312 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ecf11a340720d768dbb80db903dd0e2a729d8a0edad23fd489500a0fc63fd99b"} err="failed to get container status \"ecf11a340720d768dbb80db903dd0e2a729d8a0edad23fd489500a0fc63fd99b\": rpc error: code = NotFound desc = could not find container \"ecf11a340720d768dbb80db903dd0e2a729d8a0edad23fd489500a0fc63fd99b\": container with ID starting with ecf11a340720d768dbb80db903dd0e2a729d8a0edad23fd489500a0fc63fd99b not found: ID does not exist" Jul 29 19:27:46.541378 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:27:46.541328 2560 scope.go:117] "RemoveContainer" containerID="85ca53f0bd8a4864c3967678e8ff2858f5d8d7570b7663b6648b11d62c77c5b6" Jul 29 19:27:46.541593 ip-10-0-140-205 kubenswrapper[2560]: E0729 19:27:46.541577 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"85ca53f0bd8a4864c3967678e8ff2858f5d8d7570b7663b6648b11d62c77c5b6\": container with ID starting with 85ca53f0bd8a4864c3967678e8ff2858f5d8d7570b7663b6648b11d62c77c5b6 not found: ID does not exist" containerID="85ca53f0bd8a4864c3967678e8ff2858f5d8d7570b7663b6648b11d62c77c5b6" Jul 29 19:27:46.541642 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:27:46.541598 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85ca53f0bd8a4864c3967678e8ff2858f5d8d7570b7663b6648b11d62c77c5b6"} err="failed to get container status \"85ca53f0bd8a4864c3967678e8ff2858f5d8d7570b7663b6648b11d62c77c5b6\": rpc error: code = NotFound desc = could not find container \"85ca53f0bd8a4864c3967678e8ff2858f5d8d7570b7663b6648b11d62c77c5b6\": container with ID starting with 85ca53f0bd8a4864c3967678e8ff2858f5d8d7570b7663b6648b11d62c77c5b6 not found: ID does not exist" Jul 29 19:27:46.551019 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:27:46.550997 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-21dbec-openshift-default-849976d68b-brcb4"] Jul 29 19:27:46.555169 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:27:46.555151 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-21dbec-openshift-default-849976d68b-brcb4"] Jul 29 19:27:46.597272 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:27:46.597246 2560 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0552b317-9b29-4382-bfa6-cd6bc13b519e-istio-podinfo\") on node \"ip-10-0-140-205.ec2.internal\" DevicePath \"\"" Jul 29 19:27:46.597272 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:27:46.597269 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qhzp5\" (UniqueName: \"kubernetes.io/projected/0552b317-9b29-4382-bfa6-cd6bc13b519e-kube-api-access-qhzp5\") on node \"ip-10-0-140-205.ec2.internal\" DevicePath \"\"" Jul 29 19:27:46.597406 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:27:46.597280 2560 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0552b317-9b29-4382-bfa6-cd6bc13b519e-workload-certs\") on node \"ip-10-0-140-205.ec2.internal\" DevicePath \"\"" Jul 29 19:27:46.597406 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:27:46.597289 2560 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0552b317-9b29-4382-bfa6-cd6bc13b519e-istio-envoy\") on node \"ip-10-0-140-205.ec2.internal\" DevicePath \"\"" Jul 29 19:27:46.597406 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:27:46.597296 2560 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0552b317-9b29-4382-bfa6-cd6bc13b519e-istio-data\") on node \"ip-10-0-140-205.ec2.internal\" DevicePath \"\"" Jul 29 19:27:46.597406 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:27:46.597304 2560 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0552b317-9b29-4382-bfa6-cd6bc13b519e-istiod-ca-cert\") on node \"ip-10-0-140-205.ec2.internal\" DevicePath \"\"" Jul 29 19:27:46.597406 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:27:46.597312 2560 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0552b317-9b29-4382-bfa6-cd6bc13b519e-credential-socket\") on node \"ip-10-0-140-205.ec2.internal\" DevicePath \"\"" Jul 29 19:27:46.597406 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:27:46.597320 2560 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0552b317-9b29-4382-bfa6-cd6bc13b519e-istio-token\") on node \"ip-10-0-140-205.ec2.internal\" DevicePath \"\"" Jul 29 19:27:47.354439 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:27:47.354406 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0552b317-9b29-4382-bfa6-cd6bc13b519e" path="/var/lib/kubelet/pods/0552b317-9b29-4382-bfa6-cd6bc13b519e/volumes" Jul 29 19:27:47.531641 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:27:47.531607 2560 generic.go:356] "Generic (PLEG): container finished" podID="d0d43bce-5cc1-458c-bea1-8db6e2848f06" containerID="29e3bb5a9c4028c4157bc600298a3dc8d9c138a2cb28509710b657bb0fed4042" exitCode=0 Jul 29 19:27:47.532087 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:27:47.531659 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-cc3e80-rt2p5" event={"ID":"d0d43bce-5cc1-458c-bea1-8db6e2848f06","Type":"ContainerDied","Data":"29e3bb5a9c4028c4157bc600298a3dc8d9c138a2cb28509710b657bb0fed4042"} Jul 29 19:27:48.672164 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:27:48.672143 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-cc3e80-rt2p5" Jul 29 19:27:48.815725 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:27:48.815639 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0d43bce-5cc1-458c-bea1-8db6e2848f06-maas-api-service-ca\") pod \"d0d43bce-5cc1-458c-bea1-8db6e2848f06\" (UID: \"d0d43bce-5cc1-458c-bea1-8db6e2848f06\") " Jul 29 19:27:48.815997 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:27:48.815976 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0d43bce-5cc1-458c-bea1-8db6e2848f06-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d0d43bce-5cc1-458c-bea1-8db6e2848f06" (UID: "d0d43bce-5cc1-458c-bea1-8db6e2848f06"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 19:27:48.916964 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:27:48.916930 2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0d43bce-5cc1-458c-bea1-8db6e2848f06-maas-api-service-ca\") on node \"ip-10-0-140-205.ec2.internal\" DevicePath \"\"" Jul 29 19:27:49.545102 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:27:49.545066 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-cc3e80-rt2p5" Jul 29 19:27:49.545283 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:27:49.545068 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-cc3e80-rt2p5" event={"ID":"d0d43bce-5cc1-458c-bea1-8db6e2848f06","Type":"ContainerDied","Data":"1c8ec1f859d760259ee9d7880cd33502d6ee94caf0d8b2f1083527cab5cbd333"} Jul 29 19:27:49.545283 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:27:49.545211 2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1c8ec1f859d760259ee9d7880cd33502d6ee94caf0d8b2f1083527cab5cbd333" Jul 29 19:28:00.641717 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:28:00.641678 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:28:00.723561 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:28:00.723520 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:28:00.845065 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:28:00.845026 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-cc3e80-rt2p5"] Jul 29 19:28:00.851409 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:28:00.851384 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-cc3e80-rt2p5"] Jul 29 19:28:01.354085 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:28:01.354053 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d0d43bce-5cc1-458c-bea1-8db6e2848f06" path="/var/lib/kubelet/pods/d0d43bce-5cc1-458c-bea1-8db6e2848f06/volumes" Jul 29 19:28:01.841532 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:28:01.841501 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:28:01.945198 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:28:01.945164 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:28:06.256163 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:28:06.256128 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-cc3e80/ipp-route-cc3e80-kserve-5f76b6665f-dlztz"] Jul 29 19:28:07.074536 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:28:07.074496 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:28:21.276239 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:28:21.276128 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-cc3e80/ipp-route-cc3e80-kserve-5f76b6665f-dlztz" podUID="b60018dc-b127-4100-8452-4087fd850a27" containerName="main" containerID="cri-o://626214e209e087ff986c3e61d61fdb2004949470357e57bbe864d5762e37dc31" gracePeriod=45 Jul 29 19:28:21.524718 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:28:21.524692 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-cc3e80/ipp-route-cc3e80-kserve-5f76b6665f-dlztz" Jul 29 19:28:21.606122 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:28:21.606045 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b60018dc-b127-4100-8452-4087fd850a27-tmp-dir\") pod \"b60018dc-b127-4100-8452-4087fd850a27\" (UID: \"b60018dc-b127-4100-8452-4087fd850a27\") " Jul 29 19:28:21.606122 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:28:21.606117 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7458m\" (UniqueName: \"kubernetes.io/projected/b60018dc-b127-4100-8452-4087fd850a27-kube-api-access-7458m\") pod \"b60018dc-b127-4100-8452-4087fd850a27\" (UID: \"b60018dc-b127-4100-8452-4087fd850a27\") " Jul 29 19:28:21.606333 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:28:21.606160 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b60018dc-b127-4100-8452-4087fd850a27-model-cache\") pod \"b60018dc-b127-4100-8452-4087fd850a27\" (UID: \"b60018dc-b127-4100-8452-4087fd850a27\") " Jul 29 19:28:21.606333 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:28:21.606214 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b60018dc-b127-4100-8452-4087fd850a27-home\") pod \"b60018dc-b127-4100-8452-4087fd850a27\" (UID: \"b60018dc-b127-4100-8452-4087fd850a27\") " Jul 29 19:28:21.606333 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:28:21.606241 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b60018dc-b127-4100-8452-4087fd850a27-dshm\") pod \"b60018dc-b127-4100-8452-4087fd850a27\" (UID: \"b60018dc-b127-4100-8452-4087fd850a27\") " Jul 29 19:28:21.606333 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:28:21.606263 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b60018dc-b127-4100-8452-4087fd850a27-tls-certs\") pod \"b60018dc-b127-4100-8452-4087fd850a27\" (UID: \"b60018dc-b127-4100-8452-4087fd850a27\") " Jul 29 19:28:21.606333 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:28:21.606291 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b60018dc-b127-4100-8452-4087fd850a27-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b60018dc-b127-4100-8452-4087fd850a27" (UID: "b60018dc-b127-4100-8452-4087fd850a27"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 19:28:21.606605 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:28:21.606499 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b60018dc-b127-4100-8452-4087fd850a27-home" (OuterVolumeSpecName: "home") pod "b60018dc-b127-4100-8452-4087fd850a27" (UID: "b60018dc-b127-4100-8452-4087fd850a27"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 19:28:21.606605 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:28:21.606513 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b60018dc-b127-4100-8452-4087fd850a27-model-cache" (OuterVolumeSpecName: "model-cache") pod "b60018dc-b127-4100-8452-4087fd850a27" (UID: "b60018dc-b127-4100-8452-4087fd850a27"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 19:28:21.606724 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:28:21.606673 2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b60018dc-b127-4100-8452-4087fd850a27-model-cache\") on node \"ip-10-0-140-205.ec2.internal\" DevicePath \"\"" Jul 29 19:28:21.606724 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:28:21.606687 2560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b60018dc-b127-4100-8452-4087fd850a27-home\") on node \"ip-10-0-140-205.ec2.internal\" DevicePath \"\"" Jul 29 19:28:21.606724 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:28:21.606695 2560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b60018dc-b127-4100-8452-4087fd850a27-tmp-dir\") on node \"ip-10-0-140-205.ec2.internal\" DevicePath \"\"" Jul 29 19:28:21.608514 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:28:21.608481 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b60018dc-b127-4100-8452-4087fd850a27-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b60018dc-b127-4100-8452-4087fd850a27" (UID: "b60018dc-b127-4100-8452-4087fd850a27"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 19:28:21.608625 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:28:21.608540 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b60018dc-b127-4100-8452-4087fd850a27-kube-api-access-7458m" (OuterVolumeSpecName: "kube-api-access-7458m") pod "b60018dc-b127-4100-8452-4087fd850a27" (UID: "b60018dc-b127-4100-8452-4087fd850a27"). InnerVolumeSpecName "kube-api-access-7458m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 19:28:21.608625 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:28:21.608550 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b60018dc-b127-4100-8452-4087fd850a27-dshm" (OuterVolumeSpecName: "dshm") pod "b60018dc-b127-4100-8452-4087fd850a27" (UID: "b60018dc-b127-4100-8452-4087fd850a27"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 19:28:21.707544 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:28:21.707516 2560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b60018dc-b127-4100-8452-4087fd850a27-dshm\") on node \"ip-10-0-140-205.ec2.internal\" DevicePath \"\"" Jul 29 19:28:21.707544 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:28:21.707538 2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b60018dc-b127-4100-8452-4087fd850a27-tls-certs\") on node \"ip-10-0-140-205.ec2.internal\" DevicePath \"\"" Jul 29 19:28:21.707707 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:28:21.707549 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7458m\" (UniqueName: \"kubernetes.io/projected/b60018dc-b127-4100-8452-4087fd850a27-kube-api-access-7458m\") on node \"ip-10-0-140-205.ec2.internal\" DevicePath \"\"" Jul 29 19:28:21.729257 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:28:21.729226 2560 generic.go:356] "Generic (PLEG): container finished" podID="b60018dc-b127-4100-8452-4087fd850a27" containerID="626214e209e087ff986c3e61d61fdb2004949470357e57bbe864d5762e37dc31" exitCode=0 Jul 29 19:28:21.729426 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:28:21.729259 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-cc3e80/ipp-route-cc3e80-kserve-5f76b6665f-dlztz" event={"ID":"b60018dc-b127-4100-8452-4087fd850a27","Type":"ContainerDied","Data":"626214e209e087ff986c3e61d61fdb2004949470357e57bbe864d5762e37dc31"} Jul 29 19:28:21.729426 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:28:21.729278 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-cc3e80/ipp-route-cc3e80-kserve-5f76b6665f-dlztz" Jul 29 19:28:21.729426 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:28:21.729299 2560 scope.go:117] "RemoveContainer" containerID="626214e209e087ff986c3e61d61fdb2004949470357e57bbe864d5762e37dc31" Jul 29 19:28:21.729426 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:28:21.729289 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-cc3e80/ipp-route-cc3e80-kserve-5f76b6665f-dlztz" event={"ID":"b60018dc-b127-4100-8452-4087fd850a27","Type":"ContainerDied","Data":"35d0effee0f3281eb1f0369d0572ea8cb195132fa7d8e32fa85420e38b2beef0"} Jul 29 19:28:21.741179 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:28:21.741169 2560 scope.go:117] "RemoveContainer" containerID="626214e209e087ff986c3e61d61fdb2004949470357e57bbe864d5762e37dc31" Jul 29 19:28:21.741510 ip-10-0-140-205 kubenswrapper[2560]: E0729 19:28:21.741485 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"626214e209e087ff986c3e61d61fdb2004949470357e57bbe864d5762e37dc31\": container with ID starting with 626214e209e087ff986c3e61d61fdb2004949470357e57bbe864d5762e37dc31 not found: ID does not exist" containerID="626214e209e087ff986c3e61d61fdb2004949470357e57bbe864d5762e37dc31" Jul 29 19:28:21.741596 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:28:21.741518 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"626214e209e087ff986c3e61d61fdb2004949470357e57bbe864d5762e37dc31"} err="failed to get container status \"626214e209e087ff986c3e61d61fdb2004949470357e57bbe864d5762e37dc31\": rpc error: code = NotFound desc = could not find container \"626214e209e087ff986c3e61d61fdb2004949470357e57bbe864d5762e37dc31\": container with ID starting with 626214e209e087ff986c3e61d61fdb2004949470357e57bbe864d5762e37dc31 not found: ID does not exist" Jul 29 19:28:21.754725 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:28:21.754700 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-cc3e80/ipp-route-cc3e80-kserve-5f76b6665f-dlztz"] Jul 29 19:28:21.757687 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:28:21.757657 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-cc3e80/ipp-route-cc3e80-kserve-5f76b6665f-dlztz"] Jul 29 19:28:23.353951 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:28:23.353917 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b60018dc-b127-4100-8452-4087fd850a27" path="/var/lib/kubelet/pods/b60018dc-b127-4100-8452-4087fd850a27/volumes" Jul 29 19:28:43.773078 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:28:43.773043 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:28:45.019048 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:28:45.019017 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:28:49.220142 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:28:49.220100 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:28:49.321021 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:28:49.320982 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:28:49.528195 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:28:49.528159 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:28:49.761541 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:28:49.761506 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:28:49.903339 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:28:49.903259 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f0bee1-l4v87"] Jul 29 19:28:49.903657 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:28:49.903644 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0552b317-9b29-4382-bfa6-cd6bc13b519e" containerName="istio-proxy" Jul 29 19:28:49.903705 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:28:49.903659 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="0552b317-9b29-4382-bfa6-cd6bc13b519e" containerName="istio-proxy" Jul 29 19:28:49.903705 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:28:49.903683 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b60018dc-b127-4100-8452-4087fd850a27" containerName="main" Jul 29 19:28:49.903705 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:28:49.903688 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="b60018dc-b127-4100-8452-4087fd850a27" containerName="main" Jul 29 19:28:49.903705 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:28:49.903697 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0552b317-9b29-4382-bfa6-cd6bc13b519e" containerName="istio-proxy" Jul 29 19:28:49.903705 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:28:49.903703 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="0552b317-9b29-4382-bfa6-cd6bc13b519e" containerName="istio-proxy" Jul 29 19:28:49.903857 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:28:49.903711 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d0d43bce-5cc1-458c-bea1-8db6e2848f06" containerName="revoke-keys" Jul 29 19:28:49.903857 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:28:49.903716 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="d0d43bce-5cc1-458c-bea1-8db6e2848f06" containerName="revoke-keys" Jul 29 19:28:49.903857 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:28:49.903772 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="0552b317-9b29-4382-bfa6-cd6bc13b519e" containerName="istio-proxy" Jul 29 19:28:49.903857 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:28:49.903780 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="b60018dc-b127-4100-8452-4087fd850a27" containerName="main" Jul 29 19:28:49.903857 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:28:49.903786 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="0552b317-9b29-4382-bfa6-cd6bc13b519e" containerName="istio-proxy" Jul 29 19:28:49.903857 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:28:49.903794 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="d0d43bce-5cc1-458c-bea1-8db6e2848f06" containerName="revoke-keys" Jul 29 19:28:49.906858 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:28:49.906843 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f0bee1-l4v87" Jul 29 19:28:49.909428 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:28:49.909404 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vgb9z\"" Jul 29 19:28:49.920702 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:28:49.920674 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f0bee1-l4v87"] Jul 29 19:28:50.066378 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:28:50.066330 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/241a7288-375a-4840-bb0d-fce7b02442cd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-f0bee1-l4v87\" (UID: \"241a7288-375a-4840-bb0d-fce7b02442cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f0bee1-l4v87" Jul 29 19:28:50.085086 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:28:50.085055 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:28:50.167571 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:28:50.167470 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/241a7288-375a-4840-bb0d-fce7b02442cd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-f0bee1-l4v87\" (UID: \"241a7288-375a-4840-bb0d-fce7b02442cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f0bee1-l4v87" Jul 29 19:28:50.168159 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:28:50.168130 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/241a7288-375a-4840-bb0d-fce7b02442cd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-f0bee1-l4v87\" (UID: \"241a7288-375a-4840-bb0d-fce7b02442cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f0bee1-l4v87" Jul 29 19:28:50.217246 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:28:50.217208 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f0bee1-l4v87" Jul 29 19:28:50.365886 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:28:50.365862 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f0bee1-l4v87"] Jul 29 19:28:50.367327 ip-10-0-140-205 kubenswrapper[2560]: W0729 19:28:50.367298 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod241a7288_375a_4840_bb0d_fce7b02442cd.slice/crio-d1adef483a69e1aeb2891bce60fa00e7d4a01b015824ab0bff8d16502103d7c7 WatchSource:0}: Error finding container d1adef483a69e1aeb2891bce60fa00e7d4a01b015824ab0bff8d16502103d7c7: Status 404 returned error can't find the container with id d1adef483a69e1aeb2891bce60fa00e7d4a01b015824ab0bff8d16502103d7c7 Jul 29 19:28:50.891842 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:28:50.891738 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f0bee1-l4v87" event={"ID":"241a7288-375a-4840-bb0d-fce7b02442cd","Type":"ContainerStarted","Data":"328e71bb5556500e3d482daf0350fb2c2a654313d81bd15c7de03f2fa1c191c2"} Jul 29 19:28:50.891842 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:28:50.891800 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f0bee1-l4v87" event={"ID":"241a7288-375a-4840-bb0d-fce7b02442cd","Type":"ContainerStarted","Data":"d1adef483a69e1aeb2891bce60fa00e7d4a01b015824ab0bff8d16502103d7c7"} Jul 29 19:28:50.906951 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:28:50.906875 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f0bee1-l4v87" podStartSLOduration=1.9068594939999999 podStartE2EDuration="1.906859494s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:28:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:28:50.905814425 +0000 UTC m=+3266.131483297" watchObservedRunningTime="2026-07-29 19:28:50.906859494 +0000 UTC m=+3266.132528366" Jul 29 19:28:51.898924 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:28:51.898889 2560 generic.go:356] "Generic (PLEG): container finished" podID="241a7288-375a-4840-bb0d-fce7b02442cd" containerID="328e71bb5556500e3d482daf0350fb2c2a654313d81bd15c7de03f2fa1c191c2" exitCode=0 Jul 29 19:28:51.899291 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:28:51.898977 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f0bee1-l4v87" event={"ID":"241a7288-375a-4840-bb0d-fce7b02442cd","Type":"ContainerDied","Data":"328e71bb5556500e3d482daf0350fb2c2a654313d81bd15c7de03f2fa1c191c2"} Jul 29 19:28:53.036324 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:28:53.036301 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f0bee1-l4v87" Jul 29 19:28:53.199254 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:28:53.199177 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/241a7288-375a-4840-bb0d-fce7b02442cd-maas-api-service-ca\") pod \"241a7288-375a-4840-bb0d-fce7b02442cd\" (UID: \"241a7288-375a-4840-bb0d-fce7b02442cd\") " Jul 29 19:28:53.199528 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:28:53.199509 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/241a7288-375a-4840-bb0d-fce7b02442cd-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "241a7288-375a-4840-bb0d-fce7b02442cd" (UID: "241a7288-375a-4840-bb0d-fce7b02442cd"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 19:28:53.300291 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:28:53.300257 2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/241a7288-375a-4840-bb0d-fce7b02442cd-maas-api-service-ca\") on node \"ip-10-0-140-205.ec2.internal\" DevicePath \"\"" Jul 29 19:28:53.912171 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:28:53.912140 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f0bee1-l4v87" Jul 29 19:28:53.912171 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:28:53.912155 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f0bee1-l4v87" event={"ID":"241a7288-375a-4840-bb0d-fce7b02442cd","Type":"ContainerDied","Data":"d1adef483a69e1aeb2891bce60fa00e7d4a01b015824ab0bff8d16502103d7c7"} Jul 29 19:28:53.912406 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:28:53.912208 2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d1adef483a69e1aeb2891bce60fa00e7d4a01b015824ab0bff8d16502103d7c7" Jul 29 19:29:00.581403 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:29:00.581369 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:29:01.186566 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:29:01.186533 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:29:01.355785 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:29:01.355756 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:29:01.456821 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:29:01.456739 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:29:01.514477 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:29:01.514442 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f0bee1-l4v87"] Jul 29 19:29:01.517753 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:29:01.517722 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f0bee1-l4v87"] Jul 29 19:29:01.544157 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:29:01.544133 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:29:01.944991 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:29:01.944963 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:29:03.353887 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:29:03.353849 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="241a7288-375a-4840-bb0d-fce7b02442cd" path="/var/lib/kubelet/pods/241a7288-375a-4840-bb0d-fce7b02442cd/volumes" Jul 29 19:29:07.732938 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:29:07.732900 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:29:07.794437 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:29:07.794392 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:29:07.994544 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:29:07.994524 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-f0bee1-openshift-default-54d6899946-pjlwb"] Jul 29 19:29:07.994764 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:29:07.994745 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-f0bee1-openshift-default-54d6899946-pjlwb" podUID="7210e159-b3ca-4448-9604-81c688188d2c" containerName="istio-proxy" containerID="cri-o://15e37a9820c18a74f7e0905b3dd1ee7887790f965ca3d56dd69c91fba6e31ca2" gracePeriod=30 Jul 29 19:29:08.012173 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:29:08.012143 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:29:08.263202 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:29:08.263107 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:29:08.411756 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:29:08.411717 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-d04442-w55pv"] Jul 29 19:29:08.412101 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:29:08.412089 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="241a7288-375a-4840-bb0d-fce7b02442cd" containerName="revoke-keys" Jul 29 19:29:08.412144 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:29:08.412103 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="241a7288-375a-4840-bb0d-fce7b02442cd" containerName="revoke-keys" Jul 29 19:29:08.412187 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:29:08.412178 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="241a7288-375a-4840-bb0d-fce7b02442cd" containerName="revoke-keys" Jul 29 19:29:08.415356 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:29:08.415327 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-d04442-w55pv" Jul 29 19:29:08.417474 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:29:08.417453 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vgb9z\"" Jul 29 19:29:08.422257 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:29:08.422234 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-d04442-w55pv"] Jul 29 19:29:08.437285 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:29:08.437259 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2f7f1e0-1aa9-474a-b15b-abed94eba79f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-d04442-w55pv\" (UID: \"b2f7f1e0-1aa9-474a-b15b-abed94eba79f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-d04442-w55pv" Jul 29 19:29:08.510024 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:29:08.509984 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:29:08.537983 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:29:08.537909 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2f7f1e0-1aa9-474a-b15b-abed94eba79f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-d04442-w55pv\" (UID: \"b2f7f1e0-1aa9-474a-b15b-abed94eba79f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-d04442-w55pv" Jul 29 19:29:08.538512 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:29:08.538494 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2f7f1e0-1aa9-474a-b15b-abed94eba79f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-d04442-w55pv\" (UID: \"b2f7f1e0-1aa9-474a-b15b-abed94eba79f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-d04442-w55pv" Jul 29 19:29:08.726504 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:29:08.726470 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-d04442-w55pv" Jul 29 19:29:08.867411 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:29:08.867384 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-d04442-w55pv"] Jul 29 19:29:08.869767 ip-10-0-140-205 kubenswrapper[2560]: W0729 19:29:08.869733 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2f7f1e0_1aa9_474a_b15b_abed94eba79f.slice/crio-9c0686e8802260db15bdcb8095a4915d05e2d8e6faed91356cb5367c2c303593 WatchSource:0}: Error finding container 9c0686e8802260db15bdcb8095a4915d05e2d8e6faed91356cb5367c2c303593: Status 404 returned error can't find the container with id 9c0686e8802260db15bdcb8095a4915d05e2d8e6faed91356cb5367c2c303593 Jul 29 19:29:08.999980 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:29:08.999947 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-d04442-w55pv" event={"ID":"b2f7f1e0-1aa9-474a-b15b-abed94eba79f","Type":"ContainerStarted","Data":"9c0686e8802260db15bdcb8095a4915d05e2d8e6faed91356cb5367c2c303593"} Jul 29 19:29:10.006920 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:29:10.006887 2560 generic.go:356] "Generic (PLEG): container finished" podID="b2f7f1e0-1aa9-474a-b15b-abed94eba79f" containerID="1ee4df61377c7892d5fd6eeb9d14721e07faa00e7f0670b1a96a4e97bc20d2e0" exitCode=0 Jul 29 19:29:10.007280 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:29:10.006974 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-d04442-w55pv" event={"ID":"b2f7f1e0-1aa9-474a-b15b-abed94eba79f","Type":"ContainerDied","Data":"1ee4df61377c7892d5fd6eeb9d14721e07faa00e7f0670b1a96a4e97bc20d2e0"} Jul 29 19:29:11.145108 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:29:11.145080 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-d04442-w55pv" Jul 29 19:29:11.266012 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:29:11.265915 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2f7f1e0-1aa9-474a-b15b-abed94eba79f-maas-api-service-ca\") pod \"b2f7f1e0-1aa9-474a-b15b-abed94eba79f\" (UID: \"b2f7f1e0-1aa9-474a-b15b-abed94eba79f\") " Jul 29 19:29:11.266262 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:29:11.266240 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2f7f1e0-1aa9-474a-b15b-abed94eba79f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b2f7f1e0-1aa9-474a-b15b-abed94eba79f" (UID: "b2f7f1e0-1aa9-474a-b15b-abed94eba79f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 19:29:11.367523 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:29:11.367487 2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2f7f1e0-1aa9-474a-b15b-abed94eba79f-maas-api-service-ca\") on node \"ip-10-0-140-205.ec2.internal\" DevicePath \"\"" Jul 29 19:29:11.693785 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:29:11.693687 2560 patch_prober.go:28] interesting pod/e2e-ipp-b-f0bee1-openshift-default-54d6899946-pjlwb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 29 19:29:11.693785 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:29:11.693753 2560 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-f0bee1-openshift-default-54d6899946-pjlwb" podUID="7210e159-b3ca-4448-9604-81c688188d2c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 29 19:29:12.020050 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:29:12.020024 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-d04442-w55pv" Jul 29 19:29:12.020203 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:29:12.020049 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-d04442-w55pv" event={"ID":"b2f7f1e0-1aa9-474a-b15b-abed94eba79f","Type":"ContainerDied","Data":"9c0686e8802260db15bdcb8095a4915d05e2d8e6faed91356cb5367c2c303593"} Jul 29 19:29:12.020203 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:29:12.020094 2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9c0686e8802260db15bdcb8095a4915d05e2d8e6faed91356cb5367c2c303593" Jul 29 19:29:13.243962 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:29:13.243942 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-f0bee1-openshift-default-54d6899946-pjlwb" Jul 29 19:29:13.284855 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:29:13.284779 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7210e159-b3ca-4448-9604-81c688188d2c-istio-data\") pod \"7210e159-b3ca-4448-9604-81c688188d2c\" (UID: \"7210e159-b3ca-4448-9604-81c688188d2c\") " Jul 29 19:29:13.284855 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:29:13.284830 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7210e159-b3ca-4448-9604-81c688188d2c-istio-envoy\") pod \"7210e159-b3ca-4448-9604-81c688188d2c\" (UID: \"7210e159-b3ca-4448-9604-81c688188d2c\") " Jul 29 19:29:13.285049 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:29:13.284868 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7210e159-b3ca-4448-9604-81c688188d2c-istio-token\") pod \"7210e159-b3ca-4448-9604-81c688188d2c\" (UID: \"7210e159-b3ca-4448-9604-81c688188d2c\") " Jul 29 19:29:13.285049 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:29:13.284895 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qjqdp\" (UniqueName: \"kubernetes.io/projected/7210e159-b3ca-4448-9604-81c688188d2c-kube-api-access-qjqdp\") pod \"7210e159-b3ca-4448-9604-81c688188d2c\" (UID: \"7210e159-b3ca-4448-9604-81c688188d2c\") " Jul 29 19:29:13.285049 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:29:13.284929 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7210e159-b3ca-4448-9604-81c688188d2c-istiod-ca-cert\") pod \"7210e159-b3ca-4448-9604-81c688188d2c\" (UID: \"7210e159-b3ca-4448-9604-81c688188d2c\") " Jul 29 19:29:13.285049 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:29:13.284965 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7210e159-b3ca-4448-9604-81c688188d2c-credential-socket\") pod \"7210e159-b3ca-4448-9604-81c688188d2c\" (UID: \"7210e159-b3ca-4448-9604-81c688188d2c\") " Jul 29 19:29:13.285049 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:29:13.285018 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7210e159-b3ca-4448-9604-81c688188d2c-workload-socket\") pod \"7210e159-b3ca-4448-9604-81c688188d2c\" (UID: \"7210e159-b3ca-4448-9604-81c688188d2c\") " Jul 29 19:29:13.285378 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:29:13.285314 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7210e159-b3ca-4448-9604-81c688188d2c-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "7210e159-b3ca-4448-9604-81c688188d2c" (UID: "7210e159-b3ca-4448-9604-81c688188d2c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 19:29:13.285378 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:29:13.285329 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7210e159-b3ca-4448-9604-81c688188d2c-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "7210e159-b3ca-4448-9604-81c688188d2c" (UID: "7210e159-b3ca-4448-9604-81c688188d2c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 19:29:13.285576 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:29:13.285391 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7210e159-b3ca-4448-9604-81c688188d2c-workload-certs\") pod \"7210e159-b3ca-4448-9604-81c688188d2c\" (UID: \"7210e159-b3ca-4448-9604-81c688188d2c\") " Jul 29 19:29:13.285576 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:29:13.285442 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7210e159-b3ca-4448-9604-81c688188d2c-istio-podinfo\") pod \"7210e159-b3ca-4448-9604-81c688188d2c\" (UID: \"7210e159-b3ca-4448-9604-81c688188d2c\") " Jul 29 19:29:13.285807 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:29:13.285783 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7210e159-b3ca-4448-9604-81c688188d2c-istio-data" (OuterVolumeSpecName: "istio-data") pod "7210e159-b3ca-4448-9604-81c688188d2c" (UID: "7210e159-b3ca-4448-9604-81c688188d2c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 19:29:13.285888 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:29:13.285834 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7210e159-b3ca-4448-9604-81c688188d2c-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "7210e159-b3ca-4448-9604-81c688188d2c" (UID: "7210e159-b3ca-4448-9604-81c688188d2c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 19:29:13.286159 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:29:13.286135 2560 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7210e159-b3ca-4448-9604-81c688188d2c-workload-certs\") on node \"ip-10-0-140-205.ec2.internal\" DevicePath \"\"" Jul 29 19:29:13.286257 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:29:13.286162 2560 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7210e159-b3ca-4448-9604-81c688188d2c-istio-data\") on node \"ip-10-0-140-205.ec2.internal\" DevicePath \"\"" Jul 29 19:29:13.286257 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:29:13.286177 2560 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7210e159-b3ca-4448-9604-81c688188d2c-credential-socket\") on node \"ip-10-0-140-205.ec2.internal\" DevicePath \"\"" Jul 29 19:29:13.286257 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:29:13.286190 2560 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7210e159-b3ca-4448-9604-81c688188d2c-workload-socket\") on node \"ip-10-0-140-205.ec2.internal\" DevicePath \"\"" Jul 29 19:29:13.286257 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:29:13.286193 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7210e159-b3ca-4448-9604-81c688188d2c-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "7210e159-b3ca-4448-9604-81c688188d2c" (UID: "7210e159-b3ca-4448-9604-81c688188d2c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 19:29:13.287387 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:29:13.287341 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7210e159-b3ca-4448-9604-81c688188d2c-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "7210e159-b3ca-4448-9604-81c688188d2c" (UID: "7210e159-b3ca-4448-9604-81c688188d2c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 19:29:13.287483 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:29:13.287469 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7210e159-b3ca-4448-9604-81c688188d2c-istio-token" (OuterVolumeSpecName: "istio-token") pod "7210e159-b3ca-4448-9604-81c688188d2c" (UID: "7210e159-b3ca-4448-9604-81c688188d2c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 19:29:13.287545 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:29:13.287521 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7210e159-b3ca-4448-9604-81c688188d2c-kube-api-access-qjqdp" (OuterVolumeSpecName: "kube-api-access-qjqdp") pod "7210e159-b3ca-4448-9604-81c688188d2c" (UID: "7210e159-b3ca-4448-9604-81c688188d2c"). InnerVolumeSpecName "kube-api-access-qjqdp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 19:29:13.288131 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:29:13.288113 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7210e159-b3ca-4448-9604-81c688188d2c-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "7210e159-b3ca-4448-9604-81c688188d2c" (UID: "7210e159-b3ca-4448-9604-81c688188d2c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 29 19:29:13.386832 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:29:13.386799 2560 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7210e159-b3ca-4448-9604-81c688188d2c-istio-podinfo\") on node \"ip-10-0-140-205.ec2.internal\" DevicePath \"\"" Jul 29 19:29:13.386832 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:29:13.386824 2560 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7210e159-b3ca-4448-9604-81c688188d2c-istio-envoy\") on node \"ip-10-0-140-205.ec2.internal\" DevicePath \"\"" Jul 29 19:29:13.386832 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:29:13.386834 2560 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7210e159-b3ca-4448-9604-81c688188d2c-istio-token\") on node \"ip-10-0-140-205.ec2.internal\" DevicePath \"\"" Jul 29 19:29:13.387004 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:29:13.386842 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qjqdp\" (UniqueName: \"kubernetes.io/projected/7210e159-b3ca-4448-9604-81c688188d2c-kube-api-access-qjqdp\") on node \"ip-10-0-140-205.ec2.internal\" DevicePath \"\"" Jul 29 19:29:13.387004 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:29:13.386851 2560 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7210e159-b3ca-4448-9604-81c688188d2c-istiod-ca-cert\") on node \"ip-10-0-140-205.ec2.internal\" DevicePath \"\"" Jul 29 19:29:14.032676 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:29:14.032644 2560 generic.go:356] "Generic (PLEG): container finished" podID="7210e159-b3ca-4448-9604-81c688188d2c" containerID="15e37a9820c18a74f7e0905b3dd1ee7887790f965ca3d56dd69c91fba6e31ca2" exitCode=0 Jul 29 19:29:14.032909 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:29:14.032686 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-f0bee1-openshift-default-54d6899946-pjlwb" event={"ID":"7210e159-b3ca-4448-9604-81c688188d2c","Type":"ContainerDied","Data":"15e37a9820c18a74f7e0905b3dd1ee7887790f965ca3d56dd69c91fba6e31ca2"} Jul 29 19:29:14.032909 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:29:14.032702 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-f0bee1-openshift-default-54d6899946-pjlwb" Jul 29 19:29:14.032909 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:29:14.032729 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-f0bee1-openshift-default-54d6899946-pjlwb" event={"ID":"7210e159-b3ca-4448-9604-81c688188d2c","Type":"ContainerDied","Data":"d6b681e79ae8426d4534019c7e5f91718741813d3af3daada9caf4d51f1ddbe9"} Jul 29 19:29:14.032909 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:29:14.032751 2560 scope.go:117] "RemoveContainer" containerID="15e37a9820c18a74f7e0905b3dd1ee7887790f965ca3d56dd69c91fba6e31ca2" Jul 29 19:29:14.042853 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:29:14.042838 2560 scope.go:117] "RemoveContainer" containerID="5e1ead99057e37806c344b40412ce8e02d1eac7fc82917520136b2465c4c3524" Jul 29 19:29:14.050975 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:29:14.050953 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-f0bee1-openshift-default-54d6899946-pjlwb"] Jul 29 19:29:14.051710 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:29:14.051689 2560 scope.go:117] "RemoveContainer" containerID="15e37a9820c18a74f7e0905b3dd1ee7887790f965ca3d56dd69c91fba6e31ca2" Jul 29 19:29:14.051993 ip-10-0-140-205 kubenswrapper[2560]: E0729 19:29:14.051970 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15e37a9820c18a74f7e0905b3dd1ee7887790f965ca3d56dd69c91fba6e31ca2\": container with ID starting with 15e37a9820c18a74f7e0905b3dd1ee7887790f965ca3d56dd69c91fba6e31ca2 not found: ID does not exist" containerID="15e37a9820c18a74f7e0905b3dd1ee7887790f965ca3d56dd69c91fba6e31ca2" Jul 29 19:29:14.052060 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:29:14.052008 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15e37a9820c18a74f7e0905b3dd1ee7887790f965ca3d56dd69c91fba6e31ca2"} err="failed to get container status \"15e37a9820c18a74f7e0905b3dd1ee7887790f965ca3d56dd69c91fba6e31ca2\": rpc error: code = NotFound desc = could not find container \"15e37a9820c18a74f7e0905b3dd1ee7887790f965ca3d56dd69c91fba6e31ca2\": container with ID starting with 15e37a9820c18a74f7e0905b3dd1ee7887790f965ca3d56dd69c91fba6e31ca2 not found: ID does not exist" Jul 29 19:29:14.052060 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:29:14.052033 2560 scope.go:117] "RemoveContainer" containerID="5e1ead99057e37806c344b40412ce8e02d1eac7fc82917520136b2465c4c3524" Jul 29 19:29:14.052410 ip-10-0-140-205 kubenswrapper[2560]: E0729 19:29:14.052383 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e1ead99057e37806c344b40412ce8e02d1eac7fc82917520136b2465c4c3524\": container with ID starting with 5e1ead99057e37806c344b40412ce8e02d1eac7fc82917520136b2465c4c3524 not found: ID does not exist" containerID="5e1ead99057e37806c344b40412ce8e02d1eac7fc82917520136b2465c4c3524" Jul 29 19:29:14.052503 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:29:14.052417 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e1ead99057e37806c344b40412ce8e02d1eac7fc82917520136b2465c4c3524"} err="failed to get container status \"5e1ead99057e37806c344b40412ce8e02d1eac7fc82917520136b2465c4c3524\": rpc error: code = NotFound desc = could not find container \"5e1ead99057e37806c344b40412ce8e02d1eac7fc82917520136b2465c4c3524\": container with ID starting with 5e1ead99057e37806c344b40412ce8e02d1eac7fc82917520136b2465c4c3524 not found: ID does not exist" Jul 29 19:29:14.055074 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:29:14.055053 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-f0bee1-openshift-default-54d6899946-pjlwb"] Jul 29 19:29:15.353854 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:29:15.353823 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7210e159-b3ca-4448-9604-81c688188d2c" path="/var/lib/kubelet/pods/7210e159-b3ca-4448-9604-81c688188d2c/volumes" Jul 29 19:29:18.894623 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:29:18.894587 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:29:18.962014 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:29:18.961982 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:29:19.035191 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:29:19.035157 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:29:19.066913 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:29:19.066874 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-d04442-w55pv"] Jul 29 19:29:19.069186 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:29:19.069157 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-d04442-w55pv"] Jul 29 19:29:19.096524 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:29:19.096495 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:29:19.354272 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:29:19.354238 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b2f7f1e0-1aa9-474a-b15b-abed94eba79f" path="/var/lib/kubelet/pods/b2f7f1e0-1aa9-474a-b15b-abed94eba79f/volumes" Jul 29 19:29:19.507462 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:29:19.507431 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:29:25.700553 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:29:25.700524 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-v4n5x_1727fb40-fbed-4984-91d4-a5890d0f4103/ovn-acl-logging/0.log" Jul 29 19:29:25.719139 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:29:25.719112 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-v4n5x_1727fb40-fbed-4984-91d4-a5890d0f4103/ovn-acl-logging/0.log" Jul 29 19:29:25.719314 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:29:25.719149 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:29:25.786449 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:29:25.786414 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:29:26.014993 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:29:26.014957 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:29:26.294214 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:29:26.294115 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:29:26.438744 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:29:26.438715 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-422d30-vjrp2"] Jul 29 19:29:26.439118 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:29:26.439106 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7210e159-b3ca-4448-9604-81c688188d2c" containerName="istio-proxy" Jul 29 19:29:26.439163 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:29:26.439120 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="7210e159-b3ca-4448-9604-81c688188d2c" containerName="istio-proxy" Jul 29 19:29:26.439163 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:29:26.439128 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b2f7f1e0-1aa9-474a-b15b-abed94eba79f" containerName="revoke-keys" Jul 29 19:29:26.439163 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:29:26.439133 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="b2f7f1e0-1aa9-474a-b15b-abed94eba79f" containerName="revoke-keys" Jul 29 19:29:26.439261 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:29:26.439205 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="7210e159-b3ca-4448-9604-81c688188d2c" containerName="istio-proxy" Jul 29 19:29:26.439261 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:29:26.439214 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="7210e159-b3ca-4448-9604-81c688188d2c" containerName="istio-proxy" Jul 29 19:29:26.439261 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:29:26.439222 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="b2f7f1e0-1aa9-474a-b15b-abed94eba79f" containerName="revoke-keys" Jul 29 19:29:26.443833 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:29:26.443817 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-422d30-vjrp2" Jul 29 19:29:26.446368 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:29:26.446339 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vgb9z\"" Jul 29 19:29:26.449874 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:29:26.449853 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-422d30-vjrp2"] Jul 29 19:29:26.529492 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:29:26.529409 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:29:26.602157 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:29:26.602079 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e34d1a4-7654-45e4-bd8c-7ed36c18aa73-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-422d30-vjrp2\" (UID: \"6e34d1a4-7654-45e4-bd8c-7ed36c18aa73\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-422d30-vjrp2" Jul 29 19:29:26.703394 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:29:26.703335 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e34d1a4-7654-45e4-bd8c-7ed36c18aa73-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-422d30-vjrp2\" (UID: \"6e34d1a4-7654-45e4-bd8c-7ed36c18aa73\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-422d30-vjrp2" Jul 29 19:29:26.703975 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:29:26.703948 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e34d1a4-7654-45e4-bd8c-7ed36c18aa73-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-422d30-vjrp2\" (UID: \"6e34d1a4-7654-45e4-bd8c-7ed36c18aa73\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-422d30-vjrp2" Jul 29 19:29:26.757057 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:29:26.757023 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-422d30-vjrp2" Jul 29 19:29:26.891958 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:29:26.891879 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-422d30-vjrp2"] Jul 29 19:29:26.895467 ip-10-0-140-205 kubenswrapper[2560]: W0729 19:29:26.895441 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e34d1a4_7654_45e4_bd8c_7ed36c18aa73.slice/crio-cff98ac01a6e22be3b576a89384c051bfcce38d1ae1af0f22bf69cd40aaf7bc6 WatchSource:0}: Error finding container cff98ac01a6e22be3b576a89384c051bfcce38d1ae1af0f22bf69cd40aaf7bc6: Status 404 returned error can't find the container with id cff98ac01a6e22be3b576a89384c051bfcce38d1ae1af0f22bf69cd40aaf7bc6 Jul 29 19:29:27.105629 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:29:27.105597 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-422d30-vjrp2" event={"ID":"6e34d1a4-7654-45e4-bd8c-7ed36c18aa73","Type":"ContainerStarted","Data":"cff98ac01a6e22be3b576a89384c051bfcce38d1ae1af0f22bf69cd40aaf7bc6"} Jul 29 19:29:28.113887 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:29:28.113855 2560 generic.go:356] "Generic (PLEG): container finished" podID="6e34d1a4-7654-45e4-bd8c-7ed36c18aa73" containerID="e514b3feec5edfd2165ddc702d92a8613b2668f39fe74299aa8a894ac55d17b8" exitCode=0 Jul 29 19:29:28.114272 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:29:28.113940 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-422d30-vjrp2" event={"ID":"6e34d1a4-7654-45e4-bd8c-7ed36c18aa73","Type":"ContainerDied","Data":"e514b3feec5edfd2165ddc702d92a8613b2668f39fe74299aa8a894ac55d17b8"} Jul 29 19:29:29.248782 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:29:29.248757 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-422d30-vjrp2" Jul 29 19:29:29.328545 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:29:29.328507 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e34d1a4-7654-45e4-bd8c-7ed36c18aa73-maas-api-service-ca\") pod \"6e34d1a4-7654-45e4-bd8c-7ed36c18aa73\" (UID: \"6e34d1a4-7654-45e4-bd8c-7ed36c18aa73\") " Jul 29 19:29:29.328885 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:29:29.328862 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e34d1a4-7654-45e4-bd8c-7ed36c18aa73-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "6e34d1a4-7654-45e4-bd8c-7ed36c18aa73" (UID: "6e34d1a4-7654-45e4-bd8c-7ed36c18aa73"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 19:29:29.430096 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:29:29.430001 2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e34d1a4-7654-45e4-bd8c-7ed36c18aa73-maas-api-service-ca\") on node \"ip-10-0-140-205.ec2.internal\" DevicePath \"\"" Jul 29 19:29:30.125918 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:29:30.125882 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-422d30-vjrp2" event={"ID":"6e34d1a4-7654-45e4-bd8c-7ed36c18aa73","Type":"ContainerDied","Data":"cff98ac01a6e22be3b576a89384c051bfcce38d1ae1af0f22bf69cd40aaf7bc6"} Jul 29 19:29:30.125918 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:29:30.125912 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-422d30-vjrp2" Jul 29 19:29:30.126119 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:29:30.125941 2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cff98ac01a6e22be3b576a89384c051bfcce38d1ae1af0f22bf69cd40aaf7bc6" Jul 29 19:29:36.897926 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:29:36.897886 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:29:36.998804 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:29:36.998768 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:29:37.092037 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:29:37.092004 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-422d30-vjrp2"] Jul 29 19:29:37.096638 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:29:37.096606 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:29:37.097527 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:29:37.097505 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-422d30-vjrp2"] Jul 29 19:29:37.171560 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:29:37.171488 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:29:37.354259 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:29:37.354226 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6e34d1a4-7654-45e4-bd8c-7ed36c18aa73" path="/var/lib/kubelet/pods/6e34d1a4-7654-45e4-bd8c-7ed36c18aa73/volumes" Jul 29 19:29:37.597455 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:29:37.597415 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:29:43.542871 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:29:43.542842 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:29:43.614792 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:29:43.614750 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:29:43.818594 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:29:43.818505 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:29:44.046842 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:29:44.046804 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:29:44.505807 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:29:44.505775 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-79b6457d9c-8ckk2_58c2b854-b8b9-4217-8fec-a260faf5ab20/authorino/0.log" Jul 29 19:29:50.647237 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:29:50.647157 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-cc9zh_c677dc98-e3ee-497f-ae69-7ce442751820/manager/0.log" Jul 29 19:29:50.901566 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:29:50.901493 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-76c6f8f9b-9dbbf_a07c02c3-7bf1-45e5-b28b-24f29d48494a/copy-manifests/0.log" Jul 29 19:29:50.909120 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:29:50.909089 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-76c6f8f9b-9dbbf_a07c02c3-7bf1-45e5-b28b-24f29d48494a/manager/0.log" Jul 29 19:29:51.561783 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:29:51.561758 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29755845-gfxfs_bd13589c-fe74-4332-8fdd-dec85585f01c/cleanup/0.log" Jul 29 19:29:51.885821 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:29:51.885738 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:29:51.906361 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:29:51.906319 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-kxnkt_7b8c910c-6556-420a-a28d-5fe83264e80c/postgres/0.log" Jul 29 19:29:52.705911 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:29:52.705876 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759s2bcr_e746889e-57fd-4aaf-bcec-4616fc712dc2/util/0.log" Jul 29 19:29:52.712159 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:29:52.712135 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759s2bcr_e746889e-57fd-4aaf-bcec-4616fc712dc2/pull/0.log" Jul 29 19:29:52.721810 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:29:52.721781 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759s2bcr_e746889e-57fd-4aaf-bcec-4616fc712dc2/extract/0.log" Jul 29 19:29:52.837141 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:29:52.837111 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0drl9r_b2a0edd1-d807-4803-b89e-1af7a9d2302d/util/0.log" Jul 29 19:29:52.844284 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:29:52.844261 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0drl9r_b2a0edd1-d807-4803-b89e-1af7a9d2302d/pull/0.log" Jul 29 19:29:52.851171 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:29:52.851151 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0drl9r_b2a0edd1-d807-4803-b89e-1af7a9d2302d/extract/0.log" Jul 29 19:29:52.986402 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:29:52.986315 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73btqth_970cab5f-702a-4e4c-8ea5-b4743aa8718c/util/0.log" Jul 29 19:29:52.992637 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:29:52.992617 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73btqth_970cab5f-702a-4e4c-8ea5-b4743aa8718c/pull/0.log" Jul 29 19:29:52.998808 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:29:52.998786 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73btqth_970cab5f-702a-4e4c-8ea5-b4743aa8718c/extract/0.log" Jul 29 19:29:53.114944 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:29:53.114922 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gvtj8_62597727-1dca-4b09-a51f-ffbd5df32885/util/0.log" Jul 29 19:29:53.121225 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:29:53.121202 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gvtj8_62597727-1dca-4b09-a51f-ffbd5df32885/pull/0.log" Jul 29 19:29:53.129517 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:29:53.129500 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gvtj8_62597727-1dca-4b09-a51f-ffbd5df32885/extract/0.log" Jul 29 19:29:53.257032 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:29:53.257011 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-79b6457d9c-8ckk2_58c2b854-b8b9-4217-8fec-a260faf5ab20/authorino/0.log" Jul 29 19:29:53.490483 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:29:53.490458 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-pdbsk_1f048c37-df77-43f1-94ba-b2c4cdc10458/manager/0.log" Jul 29 19:29:53.608229 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:29:53.608155 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-l4f29_4fb47286-f9fe-449f-9a3d-bda17d693921/kuadrant-console-plugin/0.log" Jul 29 19:29:54.705111 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:29:54.705079 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-54cd75bc79-ns79p_2d9798e3-f217-4a46-ab4d-9f2bd22e6516/kube-auth-proxy/0.log" Jul 29 19:29:55.695178 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:29:55.695147 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-qcvt8_031eec58-a223-4fd8-9faf-5f64c76554b1/main/0.log" Jul 29 19:29:55.945409 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:29:55.945300 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-wbn2l_b5f1c834-97e4-4e1c-9d03-f10adaa9c8b9/main/0.log" Jul 29 19:29:56.196132 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:29:56.196038 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-rbqwc_dc3ba607-5106-4d50-9f19-2d558de8404b/main/0.log" Jul 29 19:29:56.312846 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:29:56.312816 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-bbkz4_bd51f594-611f-44e7-a8e9-6e02aed86875/main/0.log" Jul 29 19:30:02.490230 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:30:02.490195 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:30:02.591267 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:30:02.591240 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:30:03.195678 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:30:03.195625 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29755845-gfxfs"] Jul 29 19:30:03.198189 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:30:03.198165 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29755845-gfxfs"] Jul 29 19:30:03.353403 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:30:03.353376 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bd13589c-fe74-4332-8fdd-dec85585f01c" path="/var/lib/kubelet/pods/bd13589c-fe74-4332-8fdd-dec85585f01c/volumes" Jul 29 19:30:05.813487 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:30:05.813457 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-zbwwp_13fd7656-9a1f-4221-a5ed-6ebecf3664c9/global-pull-secret-syncer/0.log" Jul 29 19:30:05.897810 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:30:05.897777 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" Jul 29 19:30:06.122658 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:30:06.122578 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-140-205.ec2.internal_89287a5239330a896427ad6104c0beff/haproxy/0.log" Jul 29 19:30:10.128489 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:30:10.128458 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759s2bcr_e746889e-57fd-4aaf-bcec-4616fc712dc2/extract/0.log" Jul 29 19:30:10.202114 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:30:10.202084 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759s2bcr_e746889e-57fd-4aaf-bcec-4616fc712dc2/util/0.log" Jul 29 19:30:10.250545 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:30:10.250515 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759s2bcr_e746889e-57fd-4aaf-bcec-4616fc712dc2/pull/0.log" Jul 29 19:30:10.282102 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:30:10.282083 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0drl9r_b2a0edd1-d807-4803-b89e-1af7a9d2302d/extract/0.log" Jul 29 19:30:10.307382 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:30:10.307340 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0drl9r_b2a0edd1-d807-4803-b89e-1af7a9d2302d/util/0.log" Jul 29 19:30:10.332390 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:30:10.332368 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0drl9r_b2a0edd1-d807-4803-b89e-1af7a9d2302d/pull/0.log" Jul 29 19:30:10.367112 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:30:10.367087 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73btqth_970cab5f-702a-4e4c-8ea5-b4743aa8718c/extract/0.log" Jul 29 19:30:10.390884 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:30:10.390825 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73btqth_970cab5f-702a-4e4c-8ea5-b4743aa8718c/util/0.log" Jul 29 19:30:10.418474 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:30:10.418455 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73btqth_970cab5f-702a-4e4c-8ea5-b4743aa8718c/pull/0.log" Jul 29 19:30:10.453831 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:30:10.453803 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gvtj8_62597727-1dca-4b09-a51f-ffbd5df32885/extract/0.log" Jul 29 19:30:10.476560 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:30:10.476539 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gvtj8_62597727-1dca-4b09-a51f-ffbd5df32885/util/0.log" Jul 29 19:30:10.501278 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:30:10.501253 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gvtj8_62597727-1dca-4b09-a51f-ffbd5df32885/pull/0.log" Jul 29 19:30:11.171754 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:30:11.171718 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-79b6457d9c-8ckk2_58c2b854-b8b9-4217-8fec-a260faf5ab20/authorino/0.log" Jul 29 19:30:11.248036 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:30:11.248004 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-pdbsk_1f048c37-df77-43f1-94ba-b2c4cdc10458/manager/0.log" Jul 29 19:30:11.273821 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:30:11.273801 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-l4f29_4fb47286-f9fe-449f-9a3d-bda17d693921/kuadrant-console-plugin/0.log" Jul 29 19:30:13.409325 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:30:13.409230 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_e38c79a4-a26e-4ac4-9e5b-e7060aca254e/alertmanager/0.log" Jul 29 19:30:13.433649 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:30:13.433619 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_e38c79a4-a26e-4ac4-9e5b-e7060aca254e/config-reloader/0.log" Jul 29 19:30:13.456385 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:30:13.456337 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_e38c79a4-a26e-4ac4-9e5b-e7060aca254e/kube-rbac-proxy-web/0.log" Jul 29 19:30:13.479136 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:30:13.479111 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_e38c79a4-a26e-4ac4-9e5b-e7060aca254e/kube-rbac-proxy/0.log" Jul 29 19:30:13.502494 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:30:13.502464 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_e38c79a4-a26e-4ac4-9e5b-e7060aca254e/kube-rbac-proxy-metric/0.log" Jul 29 19:30:13.524750 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:30:13.524722 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_e38c79a4-a26e-4ac4-9e5b-e7060aca254e/prom-label-proxy/0.log" Jul 29 19:30:13.551196 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:30:13.551170 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_e38c79a4-a26e-4ac4-9e5b-e7060aca254e/init-config-reloader/0.log" Jul 29 19:30:13.703765 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:30:13.703686 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-784b5c7fdf-9xt45_5603d54b-3265-4734-8526-72296a9c600e/metrics-server/0.log" Jul 29 19:30:13.732683 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:30:13.732658 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-68b7fc9574-xpcxx_233d2ddf-e1b6-4ee9-8ed3-4706f4fd5260/monitoring-plugin/0.log" Jul 29 19:30:13.860925 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:30:13.860896 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-tb6xb_677a8b8b-b142-4cba-b81a-fefa36ee1f69/node-exporter/0.log" Jul 29 19:30:13.884724 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:30:13.884698 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-tb6xb_677a8b8b-b142-4cba-b81a-fefa36ee1f69/kube-rbac-proxy/0.log" Jul 29 19:30:13.906273 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:30:13.906234 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-tb6xb_677a8b8b-b142-4cba-b81a-fefa36ee1f69/init-textfile/0.log" Jul 29 19:30:14.019221 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:30:14.019200 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-s4rh6_f74a7c59-0701-4b47-bdcd-a314fc2600de/kube-rbac-proxy-main/0.log" Jul 29 19:30:14.044153 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:30:14.044127 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-s4rh6_f74a7c59-0701-4b47-bdcd-a314fc2600de/kube-rbac-proxy-self/0.log" Jul 29 19:30:14.080923 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:30:14.080897 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-s4rh6_f74a7c59-0701-4b47-bdcd-a314fc2600de/openshift-state-metrics/0.log" Jul 29 19:30:14.331160 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:30:14.331082 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-84d69f794d-cxj67_64b54a01-00fb-446a-9def-aaa7127c3fef/prometheus-operator/0.log" Jul 29 19:30:14.351577 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:30:14.351547 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-84d69f794d-cxj67_64b54a01-00fb-446a-9def-aaa7127c3fef/kube-rbac-proxy/0.log" Jul 29 19:30:14.410536 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:30:14.410516 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6c8f44d988-8hrdm_bc69e3d8-7af1-4366-bfe1-531aecbcf766/telemeter-client/0.log" Jul 29 19:30:14.436766 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:30:14.436739 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6c8f44d988-8hrdm_bc69e3d8-7af1-4366-bfe1-531aecbcf766/reload/0.log" Jul 29 19:30:14.460244 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:30:14.460225 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6c8f44d988-8hrdm_bc69e3d8-7af1-4366-bfe1-531aecbcf766/kube-rbac-proxy/0.log" Jul 29 19:30:14.493448 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:30:14.493416 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-76dc89c749-sn6bb_3eb9d5f4-642e-4158-83d6-0d4bc40efd2f/thanos-query/0.log" Jul 29 19:30:14.514976 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:30:14.514953 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-76dc89c749-sn6bb_3eb9d5f4-642e-4158-83d6-0d4bc40efd2f/kube-rbac-proxy-web/0.log" Jul 29 19:30:14.537058 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:30:14.537037 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-76dc89c749-sn6bb_3eb9d5f4-642e-4158-83d6-0d4bc40efd2f/kube-rbac-proxy/0.log" Jul 29 19:30:14.559127 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:30:14.559109 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-76dc89c749-sn6bb_3eb9d5f4-642e-4158-83d6-0d4bc40efd2f/prom-label-proxy/0.log" Jul 29 19:30:14.585507 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:30:14.585420 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-76dc89c749-sn6bb_3eb9d5f4-642e-4158-83d6-0d4bc40efd2f/kube-rbac-proxy-rules/0.log" Jul 29 19:30:14.611226 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:30:14.611203 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-76dc89c749-sn6bb_3eb9d5f4-642e-4158-83d6-0d4bc40efd2f/kube-rbac-proxy-metrics/0.log" Jul 29 19:30:15.575749 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:30:15.575697 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-kmkl2/perf-node-gather-daemonset-q9qrk"] Jul 29 19:30:15.576283 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:30:15.576135 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e34d1a4-7654-45e4-bd8c-7ed36c18aa73" containerName="revoke-keys" Jul 29 19:30:15.576283 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:30:15.576149 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e34d1a4-7654-45e4-bd8c-7ed36c18aa73" containerName="revoke-keys" Jul 29 19:30:15.576283 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:30:15.576165 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7210e159-b3ca-4448-9604-81c688188d2c" containerName="istio-proxy" Jul 29 19:30:15.576283 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:30:15.576170 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="7210e159-b3ca-4448-9604-81c688188d2c" containerName="istio-proxy" Jul 29 19:30:15.576283 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:30:15.576237 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="6e34d1a4-7654-45e4-bd8c-7ed36c18aa73" containerName="revoke-keys" Jul 29 19:30:15.579247 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:30:15.579231 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-kmkl2/perf-node-gather-daemonset-q9qrk" Jul 29 19:30:15.581810 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:30:15.581790 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-kmkl2\"/\"openshift-service-ca.crt\"" Jul 29 19:30:15.582521 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:30:15.582507 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-kmkl2\"/\"default-dockercfg-qtdw2\"" Jul 29 19:30:15.582574 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:30:15.582515 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-kmkl2\"/\"kube-root-ca.crt\"" Jul 29 19:30:15.594435 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:30:15.594397 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-kmkl2/perf-node-gather-daemonset-q9qrk"] Jul 29 19:30:15.747968 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:30:15.747934 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/8e98a3f7-5287-46f4-81b9-04adf23b38df-proc\") pod \"perf-node-gather-daemonset-q9qrk\" (UID: \"8e98a3f7-5287-46f4-81b9-04adf23b38df\") " pod="openshift-must-gather-kmkl2/perf-node-gather-daemonset-q9qrk" Jul 29 19:30:15.747968 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:30:15.747974 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/8e98a3f7-5287-46f4-81b9-04adf23b38df-podres\") pod \"perf-node-gather-daemonset-q9qrk\" (UID: \"8e98a3f7-5287-46f4-81b9-04adf23b38df\") " pod="openshift-must-gather-kmkl2/perf-node-gather-daemonset-q9qrk" Jul 29 19:30:15.748179 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:30:15.748046 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8e98a3f7-5287-46f4-81b9-04adf23b38df-sys\") pod \"perf-node-gather-daemonset-q9qrk\" (UID: \"8e98a3f7-5287-46f4-81b9-04adf23b38df\") " pod="openshift-must-gather-kmkl2/perf-node-gather-daemonset-q9qrk" Jul 29 19:30:15.748179 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:30:15.748117 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8e98a3f7-5287-46f4-81b9-04adf23b38df-lib-modules\") pod \"perf-node-gather-daemonset-q9qrk\" (UID: \"8e98a3f7-5287-46f4-81b9-04adf23b38df\") " pod="openshift-must-gather-kmkl2/perf-node-gather-daemonset-q9qrk" Jul 29 19:30:15.748179 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:30:15.748141 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p88b4\" (UniqueName: \"kubernetes.io/projected/8e98a3f7-5287-46f4-81b9-04adf23b38df-kube-api-access-p88b4\") pod \"perf-node-gather-daemonset-q9qrk\" (UID: \"8e98a3f7-5287-46f4-81b9-04adf23b38df\") " pod="openshift-must-gather-kmkl2/perf-node-gather-daemonset-q9qrk" Jul 29 19:30:15.849596 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:30:15.849518 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8e98a3f7-5287-46f4-81b9-04adf23b38df-lib-modules\") pod \"perf-node-gather-daemonset-q9qrk\" (UID: \"8e98a3f7-5287-46f4-81b9-04adf23b38df\") " pod="openshift-must-gather-kmkl2/perf-node-gather-daemonset-q9qrk" Jul 29 19:30:15.849596 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:30:15.849550 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p88b4\" (UniqueName: \"kubernetes.io/projected/8e98a3f7-5287-46f4-81b9-04adf23b38df-kube-api-access-p88b4\") pod \"perf-node-gather-daemonset-q9qrk\" (UID: \"8e98a3f7-5287-46f4-81b9-04adf23b38df\") " pod="openshift-must-gather-kmkl2/perf-node-gather-daemonset-q9qrk" Jul 29 19:30:15.849596 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:30:15.849577 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/8e98a3f7-5287-46f4-81b9-04adf23b38df-proc\") pod \"perf-node-gather-daemonset-q9qrk\" (UID: \"8e98a3f7-5287-46f4-81b9-04adf23b38df\") " pod="openshift-must-gather-kmkl2/perf-node-gather-daemonset-q9qrk" Jul 29 19:30:15.849860 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:30:15.849606 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/8e98a3f7-5287-46f4-81b9-04adf23b38df-podres\") pod \"perf-node-gather-daemonset-q9qrk\" (UID: \"8e98a3f7-5287-46f4-81b9-04adf23b38df\") " pod="openshift-must-gather-kmkl2/perf-node-gather-daemonset-q9qrk" Jul 29 19:30:15.849860 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:30:15.849665 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8e98a3f7-5287-46f4-81b9-04adf23b38df-sys\") pod \"perf-node-gather-daemonset-q9qrk\" (UID: \"8e98a3f7-5287-46f4-81b9-04adf23b38df\") " pod="openshift-must-gather-kmkl2/perf-node-gather-daemonset-q9qrk" Jul 29 19:30:15.849860 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:30:15.849689 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8e98a3f7-5287-46f4-81b9-04adf23b38df-lib-modules\") pod \"perf-node-gather-daemonset-q9qrk\" (UID: \"8e98a3f7-5287-46f4-81b9-04adf23b38df\") " pod="openshift-must-gather-kmkl2/perf-node-gather-daemonset-q9qrk" Jul 29 19:30:15.849860 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:30:15.849687 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/8e98a3f7-5287-46f4-81b9-04adf23b38df-proc\") pod \"perf-node-gather-daemonset-q9qrk\" (UID: \"8e98a3f7-5287-46f4-81b9-04adf23b38df\") " pod="openshift-must-gather-kmkl2/perf-node-gather-daemonset-q9qrk" Jul 29 19:30:15.849860 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:30:15.849787 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8e98a3f7-5287-46f4-81b9-04adf23b38df-sys\") pod \"perf-node-gather-daemonset-q9qrk\" (UID: \"8e98a3f7-5287-46f4-81b9-04adf23b38df\") " pod="openshift-must-gather-kmkl2/perf-node-gather-daemonset-q9qrk" Jul 29 19:30:15.850026 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:30:15.849889 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/8e98a3f7-5287-46f4-81b9-04adf23b38df-podres\") pod \"perf-node-gather-daemonset-q9qrk\" (UID: \"8e98a3f7-5287-46f4-81b9-04adf23b38df\") " pod="openshift-must-gather-kmkl2/perf-node-gather-daemonset-q9qrk" Jul 29 19:30:15.857487 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:30:15.857467 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p88b4\" (UniqueName: \"kubernetes.io/projected/8e98a3f7-5287-46f4-81b9-04adf23b38df-kube-api-access-p88b4\") pod \"perf-node-gather-daemonset-q9qrk\" (UID: \"8e98a3f7-5287-46f4-81b9-04adf23b38df\") " pod="openshift-must-gather-kmkl2/perf-node-gather-daemonset-q9qrk" Jul 29 19:30:15.891063 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:30:15.891038 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-kmkl2/perf-node-gather-daemonset-q9qrk" Jul 29 19:30:16.029235 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:30:16.029210 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-kmkl2/perf-node-gather-daemonset-q9qrk"] Jul 29 19:30:16.031262 ip-10-0-140-205 kubenswrapper[2560]: W0729 19:30:16.031222 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod8e98a3f7_5287_46f4_81b9_04adf23b38df.slice/crio-74e56a20567ad4b318fb0d24dbafc51fc273f7fe892c6c9de6cd3dba8ad8d9e5 WatchSource:0}: Error finding container 74e56a20567ad4b318fb0d24dbafc51fc273f7fe892c6c9de6cd3dba8ad8d9e5: Status 404 returned error can't find the container with id 74e56a20567ad4b318fb0d24dbafc51fc273f7fe892c6c9de6cd3dba8ad8d9e5 Jul 29 19:30:16.392882 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:30:16.392780 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kmkl2/perf-node-gather-daemonset-q9qrk" event={"ID":"8e98a3f7-5287-46f4-81b9-04adf23b38df","Type":"ContainerStarted","Data":"f628eddf3adb0df8a5583e35f8fcb00710f652372b269f0e01da421acef9ec08"} Jul 29 19:30:16.392882 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:30:16.392832 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kmkl2/perf-node-gather-daemonset-q9qrk" event={"ID":"8e98a3f7-5287-46f4-81b9-04adf23b38df","Type":"ContainerStarted","Data":"74e56a20567ad4b318fb0d24dbafc51fc273f7fe892c6c9de6cd3dba8ad8d9e5"} Jul 29 19:30:16.392882 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:30:16.392858 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-kmkl2/perf-node-gather-daemonset-q9qrk" Jul 29 19:30:16.412250 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:30:16.412197 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-kmkl2/perf-node-gather-daemonset-q9qrk" podStartSLOduration=1.412184182 podStartE2EDuration="1.412184182s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 19:30:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:30:16.409576474 +0000 UTC m=+3351.635245348" watchObservedRunningTime="2026-07-29 19:30:16.412184182 +0000 UTC m=+3351.637853101" Jul 29 19:30:18.792731 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:30:18.792704 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-gls76_4e3cd5b3-9c9f-4282-a3ee-9ebb947273a9/dns/0.log" Jul 29 19:30:18.815483 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:30:18.815462 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-gls76_4e3cd5b3-9c9f-4282-a3ee-9ebb947273a9/kube-rbac-proxy/0.log" Jul 29 19:30:18.840807 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:30:18.840785 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-nnlkl_2fbfc68e-382a-4371-8291-ce71b8eec8ce/dns-node-resolver/0.log" Jul 29 19:30:19.532537 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:30:19.532509 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-nvp4r_96c83b74-1875-4a28-8e0e-1470f61a31c9/node-ca/0.log" Jul 29 19:30:20.819854 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:30:20.819828 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-54cd75bc79-ns79p_2d9798e3-f217-4a46-ab4d-9f2bd22e6516/kube-auth-proxy/0.log" Jul 29 19:30:21.868825 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:30:21.868794 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-f9stl_615e2b70-a2a4-4564-ab30-eb2ca3869414/serve-healthcheck-canary/0.log" Jul 29 19:30:22.409566 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:30:22.409536 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-kmkl2/perf-node-gather-daemonset-q9qrk" Jul 29 19:30:22.536649 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:30:22.536626 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-nr2lw_36f140ba-4691-4e5e-b58f-9305091a175e/kube-rbac-proxy/0.log" Jul 29 19:30:22.563529 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:30:22.563501 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-nr2lw_36f140ba-4691-4e5e-b58f-9305091a175e/exporter/0.log" Jul 29 19:30:22.607365 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:30:22.607318 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-nr2lw_36f140ba-4691-4e5e-b58f-9305091a175e/extractor/0.log" Jul 29 19:30:24.448693 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:30:24.448661 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-69g8w_07b2a3b0-456d-4696-92a3-968f3cf7d956/konnectivity-agent/0.log" limit=1024 Jul 29 19:30:25.850837 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:30:25.850800 2560 scope.go:117] "RemoveContainer" containerID="033263efa9fc9959670587674a24012eb160f24998f0b704ffe46f7fb6f43cdf" Jul 29 19:30:26.110736 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:30:26.110653 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-cc9zh_c677dc98-e3ee-497f-ae69-7ce442751820/manager/0.log" Jul 29 19:30:26.207077 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:30:26.207049 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-76c6f8f9b-9dbbf_a07c02c3-7bf1-45e5-b28b-24f29d48494a/manager/0.log" Jul 29 19:30:26.237309 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:30:26.237282 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-76c6f8f9b-9dbbf_a07c02c3-7bf1-45e5-b28b-24f29d48494a/copy-manifests/0.log" Jul 29 19:30:26.383403 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:30:26.383313 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-54cd75bc79-ns79p_2d9798e3-f217-4a46-ab4d-9f2bd22e6516/kube-auth-proxy/0.log" Jul 29 19:30:27.662211 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:30:27.662186 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-5b7494b68d-72vwg_571420ac-fc46-4194-9a92-515eb6937cec/manager/0.log" Jul 29 19:30:28.214734 ip-10-0-140-205 kubenswrapper[2560]: I0729 19:30:28.214701 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-54cd75bc79-ns79p_2d9798e3-f217-4a46-ab4d-9f2bd22e6516/kube-auth-proxy/0.log"